Language model training method and apparatus, device, storage medium, and product

Abstract

The application discloses a language model training method and device, equipment, a storage medium and a computer program product, relates to the field of natural language processing, and comprises the following steps: acquiring original question data; inputting the original question data into the large language model to obtain similar question data corresponding to the original question data; performing positive and negative example data enhancement on the similar question data to construct a preference data set; and optimizing the large language model based on the preference data set to obtain a trained large language model. In this way, on the basis of obtaining similar question data output by the large language model, the preference data set containing the positive and negative examples of the similar question data is constructed based on the similar question data, so that the large language model is optimized, the model can generate questions with low similarity to existing questions, the diversity and creativity of the questions generated by the large language model are increased, and the quality of the questions generated by the large language model in the question generation system is improved.

Description

Technical Field

[0001] This application relates to the field of natural language processing, and more particularly to a language model training method, apparatus, device, storage medium, and product. Background Technology

[0002] In the fields of natural language processing and artificial intelligence, question generation technology is a key research direction. The main task of question generation is to generate questions with practical value and educational significance based on given text or data.

[0003] Currently, in the industry's question generation technology, the large language models used in question generation systems rely excessively on specific datasets during training, and there is a lack of clear optimization metrics for the diversity of generated content when fine-tuning these models. Even high-performance models tend to generate questions highly similar to existing questions. In summary, the questions generated by question generation systems in related technologies often lack diversity and originality, resulting in low question quality.

[0004] Therefore, improving the quality of questions generated by large language models in question generation systems is an urgent problem to be solved.

[0005] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main objective of this application is to provide a language model training method, apparatus, device, storage medium, and computer program product, aiming to solve the technical problem of how to improve the quality of problems generated by large language models in a problem generation system.

[0007] To achieve the above objectives, this application proposes a language model training method applied to a question generation system, wherein the question generation system deploys a large language model to be trained, and the method includes:

[0008] Obtain the original problem data;

[0009] The original question data is input into the large language model to obtain similar question data corresponding to the original question data;

[0010] The similar problem data is augmented with positive and negative examples to construct a preference dataset;

[0011] The large language model is optimized based on the preference dataset to obtain the trained large language model.

[0012] Optionally, the step of performing positive and negative example data augmentation on the similar problem data to construct a preference dataset includes:

[0013] The similar question data are evaluated for similarity to obtain the evaluation results of the similarity of each of the similar question data.

[0014] Based on the evaluation results, similar problem data with a similarity higher than a preset evaluation threshold are identified as unusable similar problem data, while similar problem data with a similarity lower than or equal to the preset evaluation threshold are identified as usable similar problem data.

[0015] Positive and negative example data augmentation is performed on the unusable similar problem data and the usable similar problem data to obtain the first positive example data and the first negative example data corresponding to the unusable similar problem data, and the second positive example data and the second negative example data corresponding to the usable similar problem data;

[0016] The preference dataset is constructed based on the first positive example data, the first negative example data, the second positive example data, and the second negative example data.

[0017] Optionally, the step of performing positive and negative example data augmentation on the unusable similar problem data and the usable similar problem data to obtain first positive example data and first negative example data corresponding to the unusable similar problem data, and second positive example data and second negative example data corresponding to the usable similar problem data includes:

[0018] The unusable similar problem data is identified as the first negative example data, and problems with similarity higher than a preset screening threshold in the first negative example data are deleted to obtain the first positive example data;

[0019] The available similar problem data is identified as the second positive example data, and the problem is copied from the second positive example data to obtain the second negative example data.

[0020] Optionally, the step of optimizing the large language model based on the preference dataset to obtain the trained large language model includes:

[0021] Based on the large language model, the training model and the comparison model are determined;

[0022] The preference dataset is input into the training model and the comparison model to obtain the first model output score output by the training model and the second model output score output by the comparison model, respectively.

[0023] Based on the first model output score and the second model output score, calculate the model output loss, and update the model parameters of the trained model based on the model output loss;

[0024] Return to the step of inputting the preference dataset into the training model and the comparison model to obtain the first model output score output by the training model and the second model output score output by the comparison model, respectively, until the preset training termination condition is met, and obtain the trained large language model.

[0025] Optionally, the step of inputting the preference dataset into the training model and the comparison model to obtain the first model output score output by the training model and the second model score output by the comparison model, respectively, includes:

[0026] The preference dataset is input into the training model and the comparison model to obtain the first probability and the second probability output by the training model, and the third probability and the fourth probability output by the comparison model, respectively. The first probability is the probability that the question data output by the training model is positive data, the second probability is the probability that the question data output by the training model is negative data, the third probability is the probability that the question data output by the comparison model is positive data, and the fourth probability is the probability that the question data output by the comparison model is negative data.

[0027] Calculate the ratio of the first probability to the second probability to obtain the output score of the first model, and calculate the ratio of the third probability to the fourth probability to obtain the output score of the second model.

[0028] Optionally, the step of determining similar question data with a similarity higher than a preset evaluation threshold as unusable similar question data based on each of the evaluation results, and determining similar question data with a similarity lower than or equal to the preset evaluation threshold as usable similar question data, includes the following afterward:

[0029] Based on the available similar problem data, a vector form conversion is performed to obtain an embedded vector set;

[0030] Obtain the target question data input by the user, and perform vector form conversion based on the target question data to obtain the first embedding vector;

[0031] Using a database retrieval method, the second embedding vector that has the highest matching degree with the first embedding vector is determined from the set of embedding vectors;

[0032] The target answer data corresponding to the target question data is generated based on the second embedding vector.

[0033] Furthermore, to achieve the above objectives, this application also proposes a language model training device, which includes:

[0034] The data acquisition module is used to acquire the original problem data;

[0035] The similar question output module is used to input the original question data into the large language model to obtain similar question data corresponding to the original question data.

[0036] A preference dataset construction module is used to augment the similar problem data with positive and negative examples to construct a preference dataset;

[0037] The model training module is used to optimize the large language model based on the preference dataset to obtain the trained large language model.

[0038] In addition, to achieve the above objectives, this application also proposes a language model training device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the language model training method described above.

[0039] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the language model training method described above.

[0040] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the language model training method described above.

[0041] One or more technical solutions proposed in this application have at least the following technical effects:

[0042] By deploying a large language model in a question generation system and training it, the quality of questions generated by the large language model is improved. Specifically, the system acquires original question data and inputs it into the large language model to obtain similar question data, resulting in a dataset of questions output by the large language model that are similar to those in the original question data. Then, positive and negative example data augmentation is performed on the similar question data to construct a preference dataset, allowing the large language model to be trained based on this dataset. Finally, the large language model is optimized based on the preference dataset to obtain a fully trained large language model. This optimization enables the large language model to generate questions with low similarity to existing questions, thereby increasing the diversity and creativity of the questions generated by the large language model. Therefore, in this embodiment, based on obtaining similar question data that is similar to the questions in the original question data output by the large language model, a preference dataset containing positive and negative examples of similar question data is constructed based on the similar question data, thereby optimizing the large language model so that the model can generate questions with low similarity to existing questions, thereby increasing the diversity and creativity of questions generated by the large language model and improving the quality of questions generated by the large language model in the question generation system. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating the language model training method of this application in Implementation Example 1.

[0046] Figure 2 This is a schematic diagram illustrating the method for training the language model in this application, which involves obtaining similar problem data.

[0047] Figure 3 This is a flowchart illustrating Embodiment 2 of the language model training method of this application;

[0048] Figure 4 This is a flowchart illustrating the language model training method of embodiment three of this application;

[0049] Figure 5This is a schematic diagram of the working logic of the model training module provided in Embodiment 3 of this application;

[0050] Figure 6 This is a schematic diagram of the module structure of the language model training device according to an embodiment of this application;

[0051] Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the language model training method in the embodiments of this application.

[0052] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0053] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0054] The following presents a first embodiment of the language model training method of this application. It should be noted that the executing entity of this embodiment can be a computing service device with language model training, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or problem generation system capable of implementing the above functions. The following description uses a problem generation system as an example to illustrate this embodiment and the subsequent embodiments.

[0055] In the fields of natural language processing and artificial intelligence, question generation technology is a key research direction, widely applied in educational software, automated question-answering systems, intelligent dialogue systems, and many other areas. The main task of question generation is to generate questions with practical value and educational significance based on given text or data. In practical applications, this technology can help build controllable interactive learning environments, provide personalized learning experiences, and automate content creation. Unlike systems that directly use models for real-time dialogue, dialogue systems that match generated questions offer greater control: the system only provides answers already available in the database through question matching retrieval. This is particularly important for maintaining the professionalism and accuracy of the dialogue.

[0056] The language model training method described in this embodiment is applied to a question generation system. This system is a software system based on natural language processing and artificial intelligence technologies. Its main function is to automatically generate questions with practical significance and educational value from given text, data, or knowledge bases. These systems can analyze input information, understand its semantic content, and accordingly create grammatically correct, logically clear questions that stimulate thinking or test knowledge mastery.

[0057] In this embodiment, the question generation system deploys a large language model to be trained. A language model is a model that can be used to perform natural language processing tasks, such as LLaMa2 13B, LLaMa2 7B, OPT 7B, etc. A large language model is an artificial intelligence model trained on a large amount of data, designed to understand and generate natural language text. They are typically based on deep learning techniques, particularly the Transformer architecture, which can capture the complexity and diversity of language. Large language models have a wide range of applications, and in the financial sector, they are commonly used for risk management, transaction analysis, and customer support. For example, banks, insurance companies, and other enterprises need to train language models to perform natural language processing tasks in their business scenarios to improve the effectiveness of their business services.

[0058] The model to be trained refers to the model whose parameters need to be optimized through the training process. Its initial model parameters can be obtained after model pre-training (or, a pre-trained model can be used as the model to be trained), or they can be initialized empirically or randomly. Using a pre-trained model as the model to be trained and training it on a specific natural language processing task is also called "fine-tuning," and the term "training" is used to represent this process in the following embodiments. In a specific implementation, using a pre-trained language model as the language model to be trained and further training it can achieve better training results and enable the trained language model to have stronger natural language processing capabilities. In this embodiment, it is not limited to using a pre-trained language model as the language model to be trained, nor is the specific pre-training method limited.

[0059] Currently, problem generation systems in related technologies often face a significant challenge: the generated problems tend to lack diversity and originality. This is primarily because existing systems rely excessively on specific datasets during training and lack explicit optimization metrics for the diversity of generated content during model fine-tuning. As a result, even high-performance models tend to generate problems highly similar to existing ones, which reduces the system's usability and user experience.

[0060] Furthermore, traditional question generation models often fail to fully utilize the latent semantic connections between input questions and answers, thus failing to effectively control the diversity and originality of the questions. This approach is limited in its ability to generate novel questions and cannot meet diverse application needs.

[0061] Based on the above considerations, this application proposes a language model training method. By constructing a preference dataset and combining it with DPO (Direct Preference Optimization) technology, this method addresses the problem that the questions generated by the aforementioned problem generation system often lack diversity and innovation, resulting in low question quality. This method increases the diversity and creativity of questions generated by large language models and improves the quality of questions generated by large language models in the problem generation system.

[0062] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0063] Reference Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the language model training method of this application. Figure 1 As shown, in this embodiment, the language model training method includes steps S10 to S40:

[0064] Step S10: Obtain the original problem data;

[0065] It should be noted that the original question data serves as the foundation for expanding question diversity and can be understood as reference data. The language model training method proposed in this application is applied to a question-answering generation system to generate other questions with low similarity to those in the original question data. The original question data can consist of user-submitted text or speech. The question-answering generation system can convert the text or speech into a specific data format for input into a large language model to generate and output answers and similar questions.

[0066] Reference Figure 2 , Figure 2 This is a schematic diagram of the question generation system proposed by the language model training method in this embodiment; such as Figure 2As shown, the question generation system can include a model service module, a question-answering module, a preference dataset construction module, and a model training module. The model service module acquires the user's input question text or speech, providing a question-answering service. It generates answers and similar questions corresponding to the question text or speech. The question-answering module converts the generated answers and similar questions into vector form, outputting text or speech data to the user for interaction, thus improving system usability and user experience. The preference dataset is used to incrementally generate preference datasets during user service using engineering methods, adjusting the model's generation preferences and increasing the diversity and uniqueness of generated questions. The model training module trains the large language model based on the DPO algorithm and the preference dataset, optimizing its performance while maintaining model stability.

[0067] Step S20: Input the original question data into the large language model to obtain similar question data corresponding to the original question data;

[0068] Specifically, such as Figure 2 As shown, in this embodiment, the original question data is input into the large language model through the model service module to obtain similar question data corresponding to the original question data. Here, similar question data refers to other questions that are similar to the questions in the original question data.

[0069] For example, if the original question data is about "how to learn programming," this question can be very broad, covering multiple aspects such as which programming language to choose, how to get started, and what good learning resources are available. When this original question data is processed by a large language model (such as the GPT series, BERT, ChatGLM, etc.), the large language model generates a series of similar question data based on the semantics of the question, its context, and similar queries and answers from a large amount of training data. Specifically, questions targeting a particular programming language could be "How to learn Python programming?"; questions about learning methods could be "Which is more suitable for beginners, online programming courses or books?"; and questions about resource recommendations could be "What are some good websites for learning programming resources?".

[0070] Step S30: Perform positive and negative example data augmentation on the similar problem data to construct a preference dataset;

[0071] It should be noted that the language model training method proposed in this application is applied to a question-answering generation system to generate other questions with low similarity to those in the original question data. Therefore, the preference dataset in this embodiment needs to meet the user's preference for generating highly diverse and creative questions. It is understood that constructing the preference dataset is a crucial process and directly relates to whether the question generation system can produce similar questions that are both diverse and creative. The purpose of constructing the preference dataset is to extract user preference features from the collected similar question data. These preference features will be used to guide the subsequent training of the language model, enabling the question generation system to generate questions that meet the user's expectations.

[0072] Additionally, it's important to note that data augmentation refers to increasing the diversity of a dataset by employing a series of transformations (such as rotation, scaling, cropping, and adding noise) while preserving the key features of the original data. Although data augmentation doesn't directly distinguish between the generation of positive and negative examples, it can be applied to already labeled data to generate more data samples with the same labels. For example, in image recognition tasks, rotating an image can generate more positive example images. Thus, this embodiment uses positive and negative example data augmentation to obtain positive and negative example data for the similar question data, thereby constructing a preference dataset. Positive example data represents correct, expected answers, while negative example data represents answers that do not meet expectations.

[0073] Step S40: Optimize the large language model based on the preference dataset to obtain the trained large language model.

[0074] It should be noted that this embodiment optimizes the large language model based on the aforementioned preference dataset using a contrastive optimization method. Specifically, this embodiment employs the DPO algorithm. The DPO algorithm is a technique used to optimize machine learning models, particularly in reinforcement learning and recommender systems. It improves model performance by directly optimizing the model's preference score, avoiding the complexity of separately training a reward model in traditional methods. Its core idea is to transform the optimization of the reward function into direct optimization of the policy, thereby avoiding explicit training of the reward model.

[0075] Specifically, the DPO algorithm directly trains the model using the preference dataset obtained in step S30, aligning its output preference distribution with human preference data. The optimization objective of DPO is to combine maximizing reward and minimizing the KL divergence (information divergence) of the reference policy into a single optimization problem. That is, this embodiment uses the DPO algorithm to find a policy that maximizes the preference score of output y given input x, while keeping the difference from the reference policy (measured by KL divergence) within a controllable range. The DPO loss function is used to train the model. During training, the model adjusts its output distribution based on human preference data, increasing the probability of high-preference outputs and decreasing the probability of low-preference outputs.

[0076] This embodiment improves the quality of questions generated by the large language model by deploying a large language model in the question generation system and training the large language model. Specifically, the system acquires original question data and inputs it into the large language model to obtain similar question data, resulting in a dataset of questions output by the large language model that are similar to those in the original question data. Then, positive and negative example data augmentation is performed on the similar question data to construct a preference dataset, allowing the large language model to be trained based on this dataset. Finally, the large language model is optimized based on the preference dataset to obtain a fully trained large language model. In other words, by optimizing the large language model, it is able to generate questions with low similarity to existing questions, thereby increasing the diversity and creativity of the questions generated by the large language model. Therefore, in this embodiment, based on obtaining similar question data that is similar to the questions in the original question data output by the large language model, a preference dataset containing positive and negative examples of similar question data is constructed based on the similar question data, thereby optimizing the large language model so that the model can generate questions with low similarity to existing questions, thereby increasing the diversity and creativity of questions generated by the large language model and improving the quality of questions generated by the large language model in the question generation system.

[0077] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the speech large model training method of this application; as shown below. Figure 3 As shown, the construction of the preference dataset based on the similar question data, step S30 may include steps S301 to S303:

[0078] Step S301: Perform similarity evaluation on the similar question data to obtain the evaluation results of the similarity of each of the similar question data;

[0079] In this embodiment, the question generation system can evaluate similar question data using any of the following similarity evaluation methods: manual evaluation, Bertscore (best score) evaluation, and training reward model.

[0080] For example, when conducting manual evaluation, first clarify the evaluation criteria for similarity, such as semantic similarity and keyword matching; collect the scores from all evaluators, and summarize and statistically analyze them to obtain the average similarity of each question pair;

[0081] Furthermore, when using the Bertscore evaluation method, a pre-trained model, such as the BERT (Bidirectional Encoder Representations from Transformers) model, is required. First, the similarity data to be evaluated is transformed into high-dimensional vectors, and then cosine similarity or other metrics are used to calculate the similarity between two high-dimensional vectors. The Bertscore evaluation method also assesses similarity by comparing the token (text unit) level alignment of two texts in the BERT model.

[0082] In this embodiment, before performing similarity assessment on similar question data, the collected similar question data can be preprocessed, including removing stop words, stemming, word form restoration, and punctuation removal, in order to reduce noise and unify the data format.

[0083] Step S302: Based on the evaluation results, similar problem data with a similarity higher than a preset evaluation threshold is identified as unusable similar problem data, and similar problem data with a similarity lower than or equal to the preset evaluation threshold is identified as usable similar problem data.

[0084] It should be noted that unusable similar question data refers to data within the similar question data that has a high similarity to questions in the original question data. Directly using this data as positive example input for model training would result in a lack of diversity in the questions generated by the question generation system. Conversely, usable similar question data refers to data within the similar question data that has a low similarity to questions in the original question data. Furthermore, it should be noted that this embodiment does not limit the evaluation threshold for similarity; the threshold can be set according to the specific implementation situation.

[0085] Step S303: Perform positive and negative example data augmentation on the unusable similar problem data and the usable similar problem data to obtain the first positive example data and the first negative example data corresponding to the unusable similar problem data, and the second positive example data and the second negative example data corresponding to the usable similar problem data;

[0086] It should be noted that this embodiment uses positive and negative example data enhancement to obtain positive and negative example data of the similar question data, in order to obtain data for constructing the preference dataset. Positive example data represents correct and expected answers, while negative example data represents answers that do not meet expectations.

[0087] Step S304: Construct the preference dataset based on the first positive example data, the first negative example data, the second positive example data, and the second negative example data.

[0088] It should be noted that the preference dataset is a representative and unique set of questions selected according to specific needs or criteria, used for subsequent analysis, recommendation, or training tasks. This embodiment constructs a dataset that meets specific needs by removing redundant data and retaining unique data, thereby improving the quality and efficiency of the dataset and providing strong support for subsequent applications. The preference dataset constructed in step S304 contains a rich collection of positive and negative examples, comprehensively reflecting users' preferences and expectations for similar questions.

[0089] In one feasible implementation, the step of determining similar problem data with a similarity higher than a preset evaluation threshold as unusable similar problem data based on each of the evaluation results, and determining similar problem data with a similarity lower than or equal to the preset evaluation threshold as usable similar problem data, is followed by:

[0090] Step A1: Based on the available similar problem data, perform vector form conversion to obtain an embedded vector set;

[0091] Specifically, referring to the first embodiment described above... Figure 2 ,like Figure 2 As shown, Figure 2 The question-answering module, after acquiring available similar question data, transforms this data into a corresponding set of embedding vectors using an embedding language model. This embedding language model can be a pre-trained model such as BERT, Word2Vec, or GloVe, which converts the available similar question data into embedding vectors within a high-dimensional vector space.

[0092] Step A2: Obtain the target question data input by the user, and perform vector form conversion based on the target question data to obtain the first embedding vector;

[0093] Specifically, this embodiment can refer to the vector form conversion method provided in step A1 to convert the target question data input by the user into a first embedding vector with the same vector form as the embedding vector set.

[0094] Step A3: Using a database retrieval method, determine the second embedding vector that has the highest matching degree with the first embedding vector from the set of embedding vectors;

[0095] In this implementation, an index can be created for the set of embedded vectors. The similarity between the first embedded vector and each vector in the set is calculated using a similarity metric (such as cosine similarity, Euclidean distance, etc.). Based on the similarity calculation results, the second embedded vector that best matches the first embedded vector is determined. This typically means finding the embedded vector with the highest similarity.

[0096] Step A4: Generate target answer data corresponding to the target question data based on the second embedding vector.

[0097] In this embodiment, since each vector in the embedding vector set corresponds to a similar question and its answer, the target answer data can be directly retrieved from the data associated with the second embedding vector. Then, the question-answering module can adjust the retrieved answers as needed to ensure their accuracy and applicability to the target question data. For example, it can perform steps such as formatting the answer and extracting key information to adjust the target answer data before outputting the adjusted target answer data to the user.

[0098] In another feasible implementation, the step S303, which involves performing positive and negative example data augmentation on the unusable similar problem data and the usable similar problem data to obtain first positive example data and first negative example data corresponding to the unusable similar problem data, and second positive example data and second negative example data corresponding to the usable similar problem data, may include:

[0099] Step B1: Identify the unusable similar problem data as the first negative example data, and delete the problems in the first negative example data whose similarity is higher than the preset screening threshold to obtain the first positive example data;

[0100] Step B2: The available similar problem data is identified as the second positive example data, and the problem is copied from the second positive example data to obtain the second negative example data.

[0101] It's important to note that positive examples refer to samples that match or represent the target or category that the model trainer wants the model to learn and correctly identify. In classification tasks, positive examples are typically labeled "1" or "yes" (or other similar notations) to indicate that they belong to the target category. For example, in a spam detection system, emails labeled "spam" are positive examples because this is the category the model trainer wants the model to recognize and classify.

[0102] Additionally, it's important to clarify that negative examples refer to samples that do not conform to or represent the target or category that the model trainer wants the model to learn and identify. In classification tasks, negative examples are typically labeled "0" or "no" (or other opposite representations) to indicate that they do not belong to the target category. For example, in a spam detection system, emails labeled "non-spam" are negative examples because this is not a category that the model trainer wants the model to specifically focus on or classify.

[0103] Specifically, for unusable similar question data, this embodiment generates or deletes overly similar questions manually. That is, questions with similarity exceeding a preset filtering threshold in the first negative example data are deleted to obtain corresponding first positive example data. The original unusable similar question data is then designated as the first negative example data, and a set of preference data is synthesized and stored in a pre-set preference database. The filtering threshold can be set according to actual implementation conditions; this embodiment does not limit it. For usable similar question data, questions in the usable similar question data are copied to obtain second negative example data. The original usable similar question data is then designated as the second positive example data, and another set of preference data is synthesized and stored in a pre-set preference database. Simultaneously, the usable similar question data can also be directly used in the question-and-answer system to provide online services to users.

[0104] In this embodiment, the first positive example data, the first negative example data, the second positive example data, and the second negative example data can be integrated using database queries, data processing frameworks (such as Pandas in Python), or dedicated data management tools to ensure the integrity and consistency of the data during the integration process. Then, the first positive example data, the first negative example data, the second positive example data, and the second negative example data need to be stored in a pre-set preference database. The preference database needs to be configured to support the storage of different types of data and have efficient query and management capabilities.

[0105] Furthermore, to improve database query efficiency, this embodiment requires creating indexes for key fields (such as query time). Additionally, the database can be optimized based on query patterns and load conditions to ensure it can efficiently support subsequent data analysis and model training.

[0106] This application, through the above-described scheme, specifically evaluates the similarity of the similar question data to obtain the evaluation results for the similarity of each of the similar question data. Based on the evaluation results, similar question data with similarity higher than a preset evaluation threshold are identified as unusable similar question data, while similar question data with similarity lower than or equal to the preset evaluation threshold are identified as usable similar question data. Positive and negative example data augmentation is performed on the unusable and usable similar question data to obtain first positive and first negative example data corresponding to the unusable similar question data, and second positive and second negative example data corresponding to the usable similar question data. Based on the first positive, first negative, second positive, and second negative example data, the preference dataset is constructed. This embodiment uses an engineering system to incrementally generate preference datasets during the service process of a large language model, thereby adjusting the model's question generation preferences and increasing the diversity and uniqueness of the questions generated by the large language model.

[0107] Based on the first embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 , Figure 4 This is a flowchart illustrating the third embodiment of the speech large model training method of this application; as shown below. Figure 4 As shown, the step of optimizing the large language model based on the preference dataset to obtain the trained large language model may include steps S401 to S405:

[0108] Step S401: Based on the large language model, determine the training model and the comparison model;

[0109] Reference Figure 5 , Figure 5 This is a schematic diagram illustrating the working logic of the model training module proposed in the third embodiment of the model training method of this application; as shown below. Figure 5 As shown, the model training module deploys a large language model to be trained, i.e., the training model in this step; and a large language model that does not need to be trained, i.e., the comparison model in this step. In each iteration of the large language model, the model parameters of the training model need to be updated, while the comparison model serves as a reference for iteration updates, and its own model parameters do not need to be updated with the increase of iteration rounds.

[0110] Step S402: Input the preference dataset into the training model and the comparison model to obtain the first model output score output by the training model and the second model output score output by the comparison model, respectively.

[0111] In this embodiment, the preference dataset is input into the training model and the comparison model using the prompt template corresponding to the preference dataset. For each model, the probability of generating each word is output, and the product probability of the generated word being a positive or negative example is calculated, thus obtaining the first model output score and the second model output score. The model output score, as input data, is used to calculate the value of the loss function, thereby evaluating the performance of the current model and guiding the subsequent optimization process.

[0112] Step S403: Calculate the model output loss based on the first model output score and the second model output score, and update the model parameters of the trained model based on the model output loss;

[0113] In this embodiment, firstly, a loss function needs to be defined to quantify the difference between the model's predicted values ​​and the actual values. For classification tasks, commonly used loss functions include cross-entropy loss and hinge loss. In this embodiment, based on the output scores of the first and second models, the loss function may be designed according to the scores, for example, using a metric to measure the degree of difference between the scores as the basis for loss calculation.

[0114] The formula for calculating the model output loss is shown below:

[0115] loss = -log(σ(βlog(R)) policy / R reference )))

[0116] Where loss is the model output loss, σ is the normalization coefficient (which can be set according to actual needs), and β is a hyperparameter in the DPO algorithm used to control the KL divergence between the trained and reference models. By adjusting the value of β, the optimal balance between model performance and stability can be found; R policy Output a score for the first model, R reference Output a score for the second model.

[0117] If the preset training termination conditions are met, then proceed to step S404: obtain the large language model that has been trained.

[0118] If the preset training termination condition is not met, return to step S402.

[0119] In this embodiment, the model parameters of the training model are continuously updated through an iterative update process until the preset conditions are met (such as stable model performance, the number of training rounds reaching the upper limit, etc.), and finally a large language model that has been trained is obtained. Thus, through continuous iteration and feedback, the model performance is gradually optimized.

[0120] In one feasible implementation, the step of inputting the preference dataset into the training model and the comparison model to obtain a first model output score from the training model and a second model score from the comparison model, respectively, may include:

[0121] Step C1: Input the preference dataset into the training model and the comparison model to obtain the first probability and the second probability output by the training model, and the third probability and the fourth probability output by the comparison model, respectively. The first probability is the probability that the problem data output by the training model is positive example data, the second probability is the probability that the problem data output by the training model is negative example data, the third probability is the probability that the problem data output by the comparison model is positive example data, and the fourth probability is the probability that the problem data output by the comparison model is negative example data.

[0122] Specifically, such as Figure 5 As shown, this embodiment first requires ensuring that the preference dataset is prepared and in a format suitable for input into both the training and contrast models. For each input question, the training model outputs two probability values: a first probability (P1) and a second probability (P2). P1 represents the probability that the model considers the generated question to be a positive example, while P2 represents the probability that the model considers the generated question to be a negative example. These two probability values ​​are typically obtained by applying a correlation function (such as the softmax function) to the model's output layer, ensuring that their sum is 1. Similarly, the contrast model also outputs two probability values ​​for the questions it generates: a third probability (P3) and a fourth probability (P4). P3 and P4 have the same meaning as P1 and P2, but they are based on the contrast model's predictions.

[0123] Step C2: Calculate the ratio of the first probability to the second probability to obtain the first model output score, and calculate the ratio of the third probability to the fourth probability to obtain the second model output score.

[0124] Specifically, such as Figure 5 As shown, this embodiment obtains the first model output score by calculating the ratio of the first probability to the second probability, which is also the result of... Figure 5 The R_policy (policy score) is calculated by taking the ratio of the third probability to the fourth probability to obtain the output score of the second model. Figure 5 R_reference (reference score) in the context.

[0125] This embodiment, through the above-described scheme, specifically determines a training model and a comparison model based on the large language model; inputs the preference dataset into the training model and the comparison model, respectively obtaining a first model output score from the training model and a second model output score from the comparison model; calculates the model output loss based on the first and second model output scores, and updates the model parameters of the training model based on the model output loss; returns to the step of inputting the preference dataset into the training model and the comparison model, respectively obtaining the first and second model output scores from the training model, until a preset training termination condition is met, thus obtaining a trained large language model. Based on the obtained preference dataset, this embodiment generates questions with low similarity to existing questions using the preference dataset and the DPO algorithm, thereby increasing the diversity and creativity of questions generated by the large language model and improving the quality of questions generated by the large language model in the question generation system.

[0126] It should be noted that the above embodiments can be combined and implemented in a reasonable manner according to the actual situation, and this embodiment will not elaborate on this further.

[0127] This application also provides a language model training device; please refer to... Figure 6 The language model training device includes:

[0128] Data acquisition module 10 is used to acquire raw problem data;

[0129] Similar question output module 20 is used to input the original question data into the large language model to obtain similar question data corresponding to the original question data;

[0130] The preference dataset construction module 30 is used to perform positive and negative example data augmentation on the similar problem data to construct the preference dataset;

[0131] The model training module 40 is used to optimize the large language model based on the preference dataset to obtain the trained large language model.

[0132] Optionally, the preference dataset construction module 30 is further configured to:

[0133] The similar question data are evaluated for similarity to obtain the evaluation results of the similarity of each of the similar question data.

[0134] Based on the evaluation results, similar problem data with a similarity higher than a preset evaluation threshold are identified as unusable similar problem data, while similar problem data with a similarity lower than or equal to the preset evaluation threshold are identified as usable similar problem data.

[0135] Positive and negative example data augmentation is performed on the unusable similar problem data and the usable similar problem data to obtain the first positive example data and the first negative example data corresponding to the unusable similar problem data, and the second positive example data and the second negative example data corresponding to the usable similar problem data;

[0136] The preference dataset is constructed based on the first positive example data, the first negative example data, the second positive example data, and the second negative example data.

[0137] Optionally, the preference dataset construction module 30 is further configured to:

[0138] The unusable similar problem data is identified as the first negative example data, and problems with similarity higher than a preset screening threshold in the first negative example data are deleted to obtain the first positive example data;

[0139] The available similar problem data is identified as the second positive example data, and the problem is copied from the second positive example data to obtain the second negative example data.

[0140] Optionally, the model training module 40 is further configured to:

[0141] Based on the large language model, the training model and the comparison model are determined;

[0142] The preference dataset is input into the training model and the comparison model to obtain the first model output score output by the training model and the second model output score output by the comparison model, respectively.

[0143] Based on the first model output score and the second model output score, calculate the model output loss, and update the model parameters of the trained model based on the model output loss;

[0144] Return to the step of inputting the preference dataset into the training model and the comparison model to obtain the first model output score output by the training model and the second model output score output by the comparison model, respectively, until the preset training termination condition is met, and obtain the trained large language model.

[0145] Optionally, the model training module 40 is further configured to:

[0146] The preference dataset is input into the training model and the comparison model to obtain the first probability and the second probability output by the training model, and the third probability and the fourth probability output by the comparison model, respectively. The first probability is the probability that the question data output by the training model is positive data, the second probability is the probability that the question data output by the training model is negative data, the third probability is the probability that the question data output by the comparison model is positive data, and the fourth probability is the probability that the question data output by the comparison model is negative data.

[0147] Calculate the ratio of the first probability to the second probability to obtain the output score of the first model, and calculate the ratio of the third probability to the fourth probability to obtain the output score of the second model.

[0148] Optionally, the model training module 40 is further configured to:

[0149] Based on the available similar problem data, a vector form conversion is performed to obtain an embedded vector set;

[0150] Obtain the target question data input by the user, and perform vector form conversion based on the target question data to obtain the first embedding vector;

[0151] Using a database retrieval method, the second embedding vector that has the highest matching degree with the first embedding vector is determined from the set of embedding vectors;

[0152] The target answer data corresponding to the target question data is generated based on the second embedding vector.

[0153] The language model training device provided in this application, employing the language model training method described in the above embodiments, can solve the technical problem of how to improve the quality of questions generated by large language models in a question generation system. Compared with the prior art, the beneficial effects of the language model training device provided in this application are the same as those of the language model training method described in the above embodiments, and other technical features in the language model training device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0154] This application provides a language model training device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the language model training method in Embodiment 1 above.

[0155] The following is for reference. Figure 7The diagram illustrates a structural schematic suitable for implementing the language model training device in the embodiments of this application. The language model training device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The language model training device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0156] like Figure 7 As shown, the language model training device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the language model training device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the language model training device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows language model training devices with various systems, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0157] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0158] The language model training device provided in this application, employing the language model training method described in the above embodiments, can solve the technical problem of how to improve the quality of questions generated by large language models in a question generation system. Compared with the prior art, the beneficial effects of the language model training device provided in this application are the same as those of the language model training method described in the above embodiments, and other technical features of this language model training device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0159] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0160] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0161] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the language model training method described in the above embodiments.

[0162] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0163] The aforementioned computer-readable storage medium may be included in the language model training device; or it may exist independently and not be assembled into the language model training device.

[0164] The aforementioned computer-readable storage medium carries one or more programs. When the one or more programs are executed by a language model training device, the language model training device performs the following actions: acquires original question data; inputs the original question data into the large language model to obtain similar question data corresponding to the original question data; performs positive and negative example data augmentation on the similar question data to construct a preference dataset; and optimizes the large language model based on the preference dataset to obtain a trained large language model.

[0165] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0166] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0167] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0168] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described language model training method. This addresses the technical problem of improving the quality of questions generated by large language models in a question generation system. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the language model training method provided in the above embodiments, and will not be elaborated upon here.

[0169] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the language model training method described above.

[0170] The computer program product provided in this application can solve the technical problem of how to improve the quality of problems generated by large language models in a problem generation system. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the language model training method provided in the above embodiments, and will not be repeated here.

[0171] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A language model training method, characterized in that, Applied to a question generation system, wherein the question generation system deploys a large language model to be trained, the method includes: Obtain the original problem data; The original question data is input into the large language model to obtain similar question data corresponding to the original question data; The similar problem data is augmented with positive and negative examples to construct a preference dataset; The large language model is optimized based on the preference dataset to obtain the trained large language model. The step of augmenting the similar problem data with positive and negative examples to construct a preference dataset includes: The similar question data are evaluated for similarity to obtain the evaluation results of the similarity of each of the similar question data. Based on the evaluation results, similar problem data with a similarity higher than a preset evaluation threshold are identified as unusable similar problem data, while similar problem data with a similarity lower than or equal to the preset evaluation threshold are identified as usable similar problem data. The unusable similar problem data is identified as the first negative example data, and problems with similarity higher than a preset screening threshold in the first negative example data are deleted to obtain the first positive example data; The available similar problem data is identified as the second positive example data, and the problem is copied from the second positive example data to obtain the second negative example data; The preference dataset is constructed based on the first positive example data, the first negative example data, the second positive example data, and the second negative example data. The step of optimizing the large language model based on the preference dataset to obtain the trained large language model includes: Based on the large language model, a training model and a comparison model are determined; the preference dataset is input into the training model and the comparison model to obtain the first model output score output by the training model and the second model output score output by the comparison model, respectively. Based on the first model output score and the second model output score, calculate the model output loss, and update the model parameters of the trained model based on the model output loss; Return to the step of inputting the preference dataset into the training model and the comparison model to obtain the first model output score output by the training model and the second model output score output by the comparison model, respectively, until the preset training termination condition is met, and obtain the trained large language model.

2. The method as described in claim 1, characterized in that, The step of inputting the preference dataset into the training model and the comparison model to obtain the first model output score output by the training model and the second model score output by the comparison model, respectively, includes: The preference dataset is input into the training model and the comparison model to obtain the first probability and the second probability output by the training model, and the third probability and the fourth probability output by the comparison model, respectively. The first probability is the probability that the question data output by the training model is positive data, the second probability is the probability that the question data output by the training model is negative data, the third probability is the probability that the question data output by the comparison model is positive data, and the fourth probability is the probability that the question data output by the comparison model is negative data. Calculate the ratio of the first probability to the second probability to obtain the output score of the first model, and calculate the ratio of the third probability to the fourth probability to obtain the output score of the second model.

3. The method as described in claim 1, characterized in that, After the steps of determining similar question data with a similarity higher than a preset evaluation threshold as unusable similar question data based on the evaluation results, and determining similar question data with a similarity lower than or equal to the preset evaluation threshold as usable similar question data, the method further includes: Based on the available similar problem data, a vector form conversion is performed to obtain an embedded vector set; Obtain the target question data input by the user, and perform vector form conversion based on the target question data to obtain the first embedding vector; Using a database retrieval method, the second embedding vector that has the highest matching degree with the first embedding vector is determined from the set of embedding vectors; The target answer data corresponding to the target question data is generated based on the second embedding vector.

4. A language model training device, characterized in that, The device includes: The data acquisition module is used to acquire the original problem data; The similar question output module is used to input the original question data into the large language model to obtain similar question data corresponding to the original question data. A preference dataset construction module is used to augment the similar problem data with positive and negative examples to construct a preference dataset; The model training module is used to optimize the large language model based on the preference dataset to obtain the trained large language model. The step of augmenting the similar problem data with positive and negative examples to construct a preference dataset includes: The similar question data are evaluated for similarity to obtain the evaluation results of the similarity of each of the similar question data. Based on the evaluation results, similar problem data with a similarity higher than a preset evaluation threshold are identified as unusable similar problem data, while similar problem data with a similarity lower than or equal to the preset evaluation threshold are identified as usable similar problem data. The unusable similar problem data is identified as the first negative example data, and problems with similarity higher than a preset screening threshold in the first negative example data are deleted to obtain the first positive example data; The available similar problem data is identified as the second positive example data, and the problem is copied from the second positive example data to obtain the second negative example data; The preference dataset is constructed based on the first positive example data, the first negative example data, the second positive example data, and the second negative example data. The step of optimizing the large language model based on the preference dataset to obtain the trained large language model includes: Based on the large language model, a training model and a comparison model are determined; the preference dataset is input into the training model and the comparison model to obtain the first model output score output by the training model and the second model output score output by the comparison model, respectively. Based on the first model output score and the second model output score, calculate the model output loss, and update the model parameters of the trained model based on the model output loss; Return to the step of inputting the preference dataset into the training model and the comparison model to obtain the first model output score output by the training model and the second model output score output by the comparison model, respectively, until the preset training termination condition is met, and obtain the trained large language model.

5. A language model training device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the language model training method as described in any one of claims 1 to 3.

6. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the language model training method as described in any one of claims 1 to 3.

7. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the language model training method as described in any one of claims 1 to 3.

Images