Language model training method and device, equipment, storage medium and product
By building preference data sets and DPO technology optimization large language models, the problem of lack of diversity and innovation in the problem generation system is solved, the quality of problem generation is improved, and higher diversity and creativity is achieved.
Patent Information
- Application Number
- CN202411193742.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-28
AI Technical Summary
The problems generated by large language models in existing problem generation systems lack diversity and innovation, resulting in low quality of the problem, mainly due to excessive dependence on specific data sets during training and the lack of clear optimization indicators for the diversity of generated content.
By obtaining the original problem data, generating similar problem data, enhancing the positive and negative example data, building a preference data set, and optimizing the large language model using Direct Preference Optimization (DPO) technology to generate problems with low similarity to existing problems. Thus increasing diversity and creativity.
The quality of problems generated by large language models in the problem generation system has been improved, the diversity and creativity of problems have been increased, and the practicality and user experience of the system have been improved.
Smart Images

Figure CN119066425A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of natural language processing, and in particular to a language model training method, apparatus, device, storage medium and product. Background Art
[0002] In the field of natural language processing and artificial intelligence, question generation technology is a key research direction. The main task of question generation is to generate questions with practical value and educational significance based on given text or data.
[0003] In the current question generation technology in the industry, due to the excessive reliance of the large language model of the question generation system on specific data sets during training, and the lack of clear optimization indicators for the diversity of generated content when fine-tuning the large language model, even high-performance models tend to generate questions that are highly similar to existing questions. In summary, the questions generated by the question generation system in related technologies often lack diversity and innovation, resulting in low question quality.
[0004] Therefore, how to improve the quality of questions generated by large language models in question generation systems is a problem that needs to be solved urgently.
[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention
[0006] The main purpose of this application is to provide a language model training method, apparatus, device, storage medium and computer program product, aiming to solve the technical problem of how to improve the quality of questions generated by a large language model in a question generation system.
[0007] To achieve the above object, the present application proposes a language model training method, which is applied to a question generation system, wherein the question generation system deploys a large language model to be trained, and the method comprises:
[0008] Get the original question data;
[0009] Inputting the original question data into the large language model to obtain similar question data corresponding to the original question data;
[0010] Performing positive and negative example data enhancement on the similar question data to construct a preference data set;
[0011] The large language model is optimized based on the preference data set to obtain a trained large language model.
[0012] Optionally, the step of performing positive and negative example data enhancement on the similar question data to construct a preference data set includes:
[0013] Performing similarity evaluation on the similar question data to obtain evaluation results of the similarities corresponding to the similar question data;
[0014] Based on the evaluation results, similar question data with a similarity higher than a preset evaluation threshold in the similar question data are determined as non-usable similar question data, and similar question data with a similarity lower than or equal to the preset evaluation threshold in the similar question data are determined as usable similar question data;
[0015] Performing positive and negative example data enhancement on the non-available similar question data and the available similar question data to obtain first positive example data and first negative example data corresponding to the non-available similar question data, and second positive example data and second negative example data corresponding to the available similar question data;
[0016] The preference data set is constructed based on the first positive example data, the first negative example data, the second positive example data, and the second negative example data.
[0017] Optionally, the step of performing positive and negative example data enhancement on the non-available similar question data and the available similar question data to obtain first positive example data and first negative example data corresponding to the non-available similar question data, and second positive example data and second negative example data corresponding to the available similar question data includes:
[0018] Determine the non-usable similar question data as the first negative example data, and delete the questions in the first negative example data whose similarity is higher than a preset screening threshold, to obtain the first positive example data;
[0019] The available similar question data is determined as the second positive example data, and the question is replicated on the second positive example data to obtain the second negative example data.
[0020] Optionally, the step of optimizing the large language model based on the preference data set to obtain a trained large language model includes:
[0021] Based on the large language model, determining a training model and a comparison model;
[0022] Inputting the preference data set into the training model and the comparison model to obtain a first model output score output by the training model and a second model output score output by the comparison model, respectively;
[0023] Calculating a model output loss based on the first model output score and the second model output score to update a model parameter of the training model based on the model output loss;
[0024] Return to the step of inputting the preference data set into the training model and the comparison model to obtain a first model output score of the training model and a second model output score of the comparison model, respectively, until a preset training end condition is met, thereby obtaining a trained large language model.
[0025] Optionally, the step of inputting the preference data set into the training model and the comparison model to respectively obtain a first model output score output by the training model and a second model output score output by the comparison model comprises:
[0026] Input the preference data set into the training model and the comparison model, and obtain a first probability and a second probability output by the training model and a third probability and a fourth probability output by the comparison model, respectively, wherein the first probability is the probability that the problem data output by the training model is positive data, the second probability is the probability that the problem data output by the training model is negative data, the third probability is the probability that the problem data output by the comparison model is positive data, and the fourth probability is the probability that the problem data output by the comparison model is negative data;
[0027] The ratio of the first probability to the second probability is calculated to obtain the first model output score, and the ratio of the third probability to the fourth probability is calculated to obtain the second model output score.
[0028] Optionally, the step of determining, based on each of the evaluation results, similar question data having a similarity higher than a preset evaluation threshold as non-usable similar question data, and determining similar question data having a similarity lower than or equal to a preset evaluation threshold as usable similar question data comprises:
[0029] Performing vector form conversion based on the available similar problem data to obtain an embedded vector set;
[0030] Obtain target question data input by a user, and convert the target question data into a vector form based on the target question data to obtain a first embedding vector;
[0031] Determining, by a database retrieval method, from the set of embedding vectors a second embedding vector having the highest matching degree with the first embedding vector;
[0032] Generate target answer data corresponding to the target question data based on the second embedding vector.
[0033] In addition, to achieve the above purpose, the present application also proposes a language model training device, the language model training device comprising:
[0034] Data acquisition module, used to obtain original question data;
[0035] A similar question output module, used for inputting the original question data into the large language model to obtain similar question data corresponding to the original question data;
[0036] A preference data set construction module, used to perform positive and negative example data enhancement on the similar problem data to construct a preference data set;
[0037] The model training module is used to optimize the large language model based on the preference data set to obtain a trained large language model.
[0038] In addition, to achieve the above-mentioned purpose, the present application also proposes a language model training device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the language model training method described above.
[0039] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the language model training method described above are implemented.
[0040] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the language model training method described above.
[0041] One or more technical solutions proposed in this application have at least the following technical effects:
[0042] The invention discloses a method for improving the quality of questions generated by the large language model by deploying a large language model in a question generation system and training the large language model. The method comprises: obtaining original question data through the question generation system, inputting the original question data into the large language model, obtaining similar question data corresponding to the original question data, and obtaining data of questions similar to the questions in the original question data output by the large language model. Then, the positive and negative example data of the similar question data are enhanced to construct a preference data set, so that the large language model can be trained based on the preference data set containing positive and negative examples of similar question data. Finally, the large language model is optimized based on the preference data set to obtain the trained large language model, that is, by optimizing the large language model, the large language model can generate questions with a low degree of similarity to existing questions, thereby increasing the diversity and creativity of the questions generated by the large language model. Therefore, in this embodiment, on the basis of obtaining similar question data similar to the questions in the original question data output by the large language model, a preference data set containing positive and negative examples of the similar question data is constructed based on the similar question data, thereby optimizing the large language model so that the model can generate questions with a low degree of similarity to existing questions, thereby increasing the diversity and creativity of the questions generated by the large language model and improving the quality of the questions generated by the large language model in the question generation system. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0045] Figure 1 A flowchart of the first embodiment of the language model training method of the present application is provided;
[0046] Figure 2 A schematic diagram of obtaining similar question data involved in the language model training method of this application;
[0047] Figure 3 A flow chart of the second embodiment of the language model training method of the present application;
[0048] Figure 4 A flowchart of the third embodiment of the language model training method of the present application is provided;
[0049] Figure 5A schematic diagram of the working logic of the model training module provided in Example 3 of the present application;
[0050] Figure 6 This is a schematic diagram of the module structure of the language model training device according to an embodiment of the present application;
[0051] Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the language model training method in the embodiment of the present application.
[0052] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0053] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.
[0054] The first embodiment of the language model training method of the present application is proposed below. It should be noted that the execution subject of this embodiment can be a computing service device with language model training, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device that can realize the above functions, a question generation system, etc. The following takes the question generation system as an example to illustrate this embodiment and the following embodiments.
[0055] Among the related technologies in the field of natural language processing and artificial intelligence, question generation technology is a key research direction and is widely used in educational software, automatic question-answering systems, intelligent dialogue systems, and many other fields. The main task of question generation is to generate questions with practical value and educational significance based on given text or data. In practical applications, this technology can help build a controllable interactive learning environment, provide personalized learning experience, and automate content creation. Different from systems that directly use models for real-time dialogue, dialogue systems that match based on generated questions have higher controllability: the system will only provide answers that already exist in the database through question matching retrieval. This is particularly important for maintaining the professionalism and accuracy of the dialogue.
[0056] The language model training method described in this embodiment is applied to a question generation system, wherein the question generation system is a software system based on natural language processing and artificial intelligence technology, whose main function is to automatically generate questions with practical significance and educational value from a given text, data or knowledge base. These systems can analyze the input information, understand its semantic content, and create questions that conform to grammatical rules, have clear logic, and can stimulate thinking or test the degree of knowledge mastery.
[0057] In this embodiment, the question generation system deploys a large language model to be trained. A language model is a model that can be used to perform natural language processing tasks, such as LLaMa2 13B, LLaMa2 7B, OPT 7B, etc. A large language model is an artificial intelligence model that is trained with a large amount of data and is designed to understand and generate natural language text. They are usually based on deep learning technology, especially the Transformer architecture, which can capture the complexity and diversity of language. The application field of large language models is very wide, and they are usually used for risk management, transaction analysis, customer support, etc. in the financial field. For example, banks, insurance companies and other companies need to train language models to perform natural language processing tasks in their business scenarios to improve business service effects.
[0058] The model to be trained refers to the model parameters in the model that need to be optimized through the training process, and its initial model parameters can be obtained after model pre-training (or the pre-trained model can be used as the model to be trained), or, its initial model parameters can also be initialized according to experience or randomly initialized. The process of using a pre-trained model as the model to be trained and training the model to be trained on a specific natural language processing task is also called "fine-tuning", and "training" is used to represent this process in the following embodiments. In a specific implementation, a pre-trained language model is used as the language model to be trained, and the language model to be trained is further trained, which can obtain better training results and enable the trained language model to have stronger natural language processing capabilities. In this embodiment, it is not limited to using a pre-trained language model as the language model to be trained, nor is it limited to specific pre-training means.
[0059] Currently, question generation systems in related technologies often face an important challenge: the generated questions often lack diversity and innovation. This is mainly because existing systems rely too much on specific datasets during training and lack clear optimization indicators for the diversity of generated content when fine-tuning the model. As a result, even high-performance models tend to generate questions that are highly similar to existing questions, which reduces the practicality of the system and the user experience.
[0060] In addition, traditional question generation models often fail to fully utilize the potential semantic associations between questions when processing input questions and answers, and thus cannot effectively control the diversity and innovation of questions. The limitation of this method is that it has limited ability to generate novel questions and cannot meet the diverse application needs.
[0061] Based on the above considerations, this application proposes a language model training method, which constructs a preference data set and combines DPO (Direct Preference Optimization) technology to solve the problem that the questions generated by the above-mentioned question generation system often lack diversity and innovation, resulting in low question quality, by precisely adjusting model preferences and enhancing training mechanisms. This method increases the diversity and creativity of the problems generated by the large language model and improves the quality of the problems generated by the large language model in the question generation system.
[0062] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.
[0063] Reference Figure 1 , Figure 1 This is a flow chart of the first embodiment of the language model training method of this application. Figure 1 As shown, in this embodiment, the language model training method includes steps S10 to S40:
[0064] Step S10, obtaining original question data;
[0065] It should be noted that the original question data is the basic data that needs to be expanded for question diversity, and can also be understood as reference data. The language model training method proposed in this application is applied to the question-answering generation system to generate other questions that are less similar to the questions in the original question data. Among them, the content of the original question data can be the question text or voice raised by the user, and the question-answering generation system can convert the question text or voice into a specific data format to input into the large language model for generating and outputting answers and similar questions.
[0066] Reference Figure 2 , Figure 2 This is a schematic diagram of the architecture of the question generation system proposed by the language model training method of this embodiment; Figure 2As shown, the question generation system may include a model service module, a question-answering module, a preference data set construction module, and a model training module. Among them, the model service module is used to obtain the question text or voice input by the user, and provide question-answering services to the user. The model service module can generate answers corresponding to the question text or voice and data of similar questions; the question-answering module is used to convert the above-generated answers and similar question data into vector form, thereby generating text or voice format data output to the user, completing the interaction with the user, thereby improving the practicality of the system and user experience; the preference data set is used to realize the incremental generation of the preference data set by the model in the process of serving the user by using an engineering method, thereby adjusting the model generation preference and increasing the diversity and uniqueness of the generated questions; the model training module trains the large language model based on the DPO algorithm and the preference data set, and optimizes the effect of the large language model while keeping the model itself stable.
[0067] Step S20, inputting the original question data into the large language model to obtain similar question data corresponding to the original question data;
[0068] Specifically, Figure 2 As shown, in this embodiment, the original question data is input into the large language model through the model service module to obtain similar question data corresponding to the original question data. The similar question data refers to other questions similar to the questions in the original question data.
[0069] For example, if the original question data is a query about "how to learn programming", this query may be very broad and involve many aspects, such as which programming language to choose, how to start learning, what are the good learning resources, etc. When this original question data is processed by a large language model (such as the GPT series, BERT, ChatGLM, etc.), the large language model generates a series of similar question data based on the semantics and context of the question and similar queries and answers in a large amount of training data. Among them, questions of the query type for a specific programming language can be "How to learn Python programming?"; inquiries about learning methods can be specifically "Compared with online programming courses and books, which is more suitable for beginners?"; inquiries about resource recommendations can be specifically "What are some good programming learning resource websites?".
[0070] Step S30, performing positive and negative example data enhancement on the similar question data to construct a preference data set;
[0071] It should be noted that the language model training method proposed in this application is applied to the question-answering generation system to generate other questions with low similarity to the questions in the original question data. Therefore, the preference data set in this embodiment needs to meet the user's preference for generating highly diverse and creative questions. It can be understood that building a preference data set is a key process and is directly related to whether the question generation system can produce similar questions that are both diverse and creative. The purpose of building a preference data set is to extract the user's preference features from the collected similar question data. These preference features will be used to guide the subsequent language model training so that the question generation system can generate questions that meet user expectations.
[0072] In addition, it should be noted that data enhancement refers to increasing the diversity of a data set through a series of transformations (such as rotation, scaling, cropping, adding noise, etc.) while retaining the key features of the original data. Although data enhancement does not directly distinguish between the generation of positive and negative examples, it can be applied to data with existing labels to generate more data samples with the same labels. For example, in image recognition tasks, more positive images can be generated by rotating images. In this way, this embodiment obtains positive and negative data of the similar problem data through positive and negative data enhancement to construct a preference data set. Among them, the positive data represents the correct and expected answer, and the negative data represents the answer that does not meet expectations.
[0073] Step S40: optimizing the large language model based on the preference data set to obtain a trained large language model.
[0074] It should be noted that this embodiment uses a comparative optimization method to optimize the large language model based on the preference data set, wherein the comparative optimization method of this embodiment specifically uses the DPO algorithm. The DPO algorithm is a technology used to optimize machine learning models, especially in the fields of reinforcement learning and recommendation systems. It improves the performance of the model by directly optimizing the preference score of the model, avoiding the complexity of the traditional method of separately training the reward model. The core idea is to transform the optimization of the reward function into a direct optimization of the strategy, thereby avoiding explicit training of the reward model.
[0075] Specifically, the DPO algorithm uses the preference data set obtained in step S30 to directly train the model so that the preference distribution of its output is aligned with the human preference data. The optimization goal of DPO is to combine the maximization of rewards and the minimization of the KL divergence (Kullback-Leibler Divergence, information divergence) of the reference strategy into a single optimization problem. That is, this embodiment can find a strategy through the DPO algorithm so that when the input x is given, the preference score of the output y is maximized while keeping the difference with the reference strategy (measured by KL divergence) within a controllable range. The DPO loss function is used to train the model. During the training process, the model adjusts its output distribution according to the human preference data so that the probability of high preference output increases and the probability of low preference output decreases.
[0076] This embodiment uses the above scheme to improve the quality of questions generated by the large language model by deploying a large language model in the question generation system and training the large language model. The original question data is obtained through the question generation system, and the original question data is input into the large language model to obtain similar question data corresponding to the original question data, so as to obtain data of questions similar to the questions in the original question data output by the large language model. Then, the positive and negative example data are enhanced for the similar question data to construct a preference data set, so that the large language model can be trained based on the preference data set containing positive and negative examples of similar question data. Finally, the large language model is optimized based on the preference data set to obtain the trained large language model, that is, by optimizing the large language model, the large language model can generate questions with a low degree of similarity to existing questions, thereby increasing the diversity and creativity of the questions generated by the large language model. Therefore, in this embodiment, on the basis of obtaining similar question data similar to the questions in the original question data output by the large language model, a preference data set containing positive and negative examples of the similar question data is constructed based on the similar question data, thereby optimizing the large language model so that the model can generate questions with a low degree of similarity to existing questions, thereby increasing the diversity and creativity of the questions generated by the large language model and improving the quality of the questions generated by the large language model in the question generation system.
[0077] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 , Figure 3 This is a flow chart of the second embodiment of the speech large model training method of the present application; Figure 3 As shown, the preference data set is constructed based on the similar question data, and step S30 may include steps S301 to S303:
[0078] Step S301, performing similarity evaluation on the similar question data to obtain evaluation results of the similarities corresponding to the similar question data;
[0079] In this embodiment, the question generation system can evaluate similar question data by any similarity evaluation method such as a manual evaluation method, a Bertscore (best score) evaluation method, and a training reward model.
[0080] For example, when conducting manual evaluation, we first clarify the evaluation criteria for similarity, such as semantic similarity, keyword matching, etc.; collect the scores of all evaluators, summarize and statistically analyze them, and obtain the average similarity of each question pair;
[0081] In addition, when using the Bertscore evaluation method, it is necessary to use a pre-trained model, such as the BERT (Bidirectional Encoder Representations from Transformers) model. First, the similar question data to be evaluated is converted into a high-dimensional vector, and the similarity between the two high-dimensional vectors is calculated using cosine similarity or other metrics. The Bertscore evaluation method also evaluates the similarity by comparing the token (text component unit) level alignment of two texts in the BERT model.
[0082] In this embodiment, before similarity evaluation is performed on similar question data, the collected similar question data may also be preprocessed, including removal of stop words, stem extraction, word form restoration, punctuation removal, etc., to reduce noise and unify the data format.
[0083] Step S302, based on the evaluation results, determining the similar question data with a similarity higher than a preset evaluation threshold as unavailable similar question data, and determining the similar question data with a similarity lower than or equal to the preset evaluation threshold as available similar question data;
[0084] It should be noted that non-usable similar question data refers to data in similar question data that is highly similar to the questions in the original question data. If this part of data is directly used as positive example input for model training, the questions generated by the question generation system will lack diversity. In contrast, usable similar question data refers to data in similar question data that is not highly similar to the questions in the original question data. In addition, it should be noted that this embodiment does not limit the evaluation threshold for evaluating similarity. During implementation, the threshold can be set according to the specific implementation situation.
[0085] Step S303, performing positive and negative example data enhancement on the unavailable similar question data and the available similar question data to obtain first positive example data and first negative example data corresponding to the unavailable similar question data, and second positive example data and second negative example data corresponding to the available similar question data;
[0086] It should be noted that this embodiment obtains positive and negative data of the similar question data through positive and negative data enhancement to obtain data for constructing the preference data set, wherein the positive data represents the correct and expected answer, and the negative data represents the undesirable answer.
[0087] Step S304: construct the preference data set based on the first positive example data, the first negative example data, the second positive example data, and the second negative example data.
[0088] It should be noted that the preference data set is a representative and unique set of questions selected according to specific requirements or standards, and is used for subsequent analysis, recommendation or training tasks. This embodiment constructs a data set that meets specific requirements by removing redundant data and retaining unique data, so as to improve the quality and efficiency of the data set and provide strong support for subsequent applications. The preference data set constructed in step S304 contains a wealth of positive and negative samples, which can fully reflect the user's preferences and expectations for similar questions.
[0089] In a feasible implementation manner, the step of determining, based on each of the evaluation results, similar question data having a similarity higher than a preset evaluation threshold as non-usable similar question data, and determining similar question data having a similarity lower than or equal to a preset evaluation threshold as usable similar question data comprises:
[0090] Step A1, performing vector form conversion based on the available similar question data to obtain an embedded vector set;
[0091] Specifically, referring to the first embodiment Figure 2 ,like Figure 2 As shown, Figure 2 The question-answering module in the obtains available similar question data and converts the available similar question data into a corresponding set of embedded vectors through an embedded language model. The embedded language model can be a pre-trained BERT, Word2Vec, GloVe, etc. model, which converts the available similar question data into an embedded vector in a high-dimensional space vector.
[0092] Step A2, obtaining target question data input by the user, and converting the target question data into a vector form to obtain a first embedding vector;
[0093] Specifically, this embodiment may refer to the vector form conversion method provided in step A1 to convert the target question data input by the user into a first embedding vector in the same vector form as the embedding vector set.
[0094] Step A3, determining, from the set of embedding vectors, a second embedding vector having the highest matching degree with the first embedding vector by a database retrieval method;
[0095] In this implementation, the set of embedding vectors may be indexed. A similarity measurement method (such as cosine similarity, Euclidean distance, etc.) is used to calculate the similarity between the first embedding vector and each vector in the set of embedding vectors. Based on the similarity calculation results, a second embedding vector with the highest degree of match to the first embedding vector is determined. This generally means finding the embedding vector with the highest similarity.
[0096] Step A4: Generate target answer data corresponding to the target question data based on the second embedding vector.
[0097] In this embodiment, since each vector in the embedding vector set corresponds to a similar question and its answer, the target answer data can be directly retrieved from the data associated with the second embedding vector. Then, the question-answering module can also adjust the retrieved answer as needed to ensure its accuracy and applicability to the target question data. For example, the target answer data can be adjusted by performing steps such as formatting the answer and extracting key information, and then the adjusted target answer data is output to the user.
[0098] In another feasible implementation manner, the performing positive and negative example data enhancement on the non-available similar problem data and the available similar problem data to obtain first positive example data and first negative example data corresponding to the non-available similar problem data, and second positive example data and second negative example data corresponding to the available similar problem data, step S303 may include:
[0099] Step B1, determining the non-usable similar question data as the first negative example data, and deleting the questions in the first negative example data whose similarity is higher than a preset screening threshold, to obtain the first positive example data;
[0100] Step B2: determine the available similar problem data as the second positive example data, and replicate the problem on the second positive example data to obtain the second negative example data.
[0101] It should be noted that positive data refers to samples that meet or represent the target or category that the model trainer wants the model to learn and correctly identify. In classification tasks, positive data are usually marked as "1" or "yes" (or other similar representations) to indicate that they belong to the target category. For example, in a spam detection system, emails marked as "spam" are positive data because this is the category that the model trainer hopes the model can recognize and classify.
[0102] In addition, it should be noted that negative data refers to samples that do not conform to or represent the target or category that the model trainer wants the model to learn and recognize. In classification tasks, negative data are usually marked as "0" or "no" (or other opposite representations) to indicate that they do not belong to the target category. For example, in a spam detection system, emails marked as "non-spam" are negative data because this is not the category that the model trainer wants the model to pay special attention to or classify.
[0103] Specifically, for unavailable similar question data, this embodiment obtains the corresponding first positive data by artificially constructing or deleting overly similar generation, that is, deleting questions in the first negative data whose similarity is higher than a preset screening threshold, and determines the original unavailable similar question data as the first negative data to synthesize a set of preference data and store it in a preset preference database. Among them, the screening threshold can be set according to the actual implementation situation, and this embodiment does not limit this. For available similar question data, the second negative data is obtained by copying the questions in the available similar question data, and the original available similar question data is determined as the second positive data to synthesize another set of preference data and store it in a preset preference database. At the same time, the available similar question data can also be directly used in the question-and-answer system to provide online services to users.
[0104] In this embodiment, the first positive example data, the first negative example data, the second positive example data, and the second negative example data can be integrated by using database queries, data processing frameworks (such as Pandas in Python), or specialized data management tools to ensure that the integrity and consistency of the data are maintained during the integration process; then, the first positive example data, the first negative example data, the second positive example data, and the second negative example data need to be stored in a pre-set preference database, wherein the preference database needs to be set to a database that supports the storage of different types of data and has the ability to efficiently query and manage.
[0105] In addition, in order to improve the query efficiency of the database, the present embodiment needs to create indexes for key fields (such as question time). In addition, the database can also be optimized according to the query mode and load of the database to ensure that it can efficiently support subsequent data analysis and model training work.
[0106] Through the above scheme, the present application specifically performs similarity evaluation on the similar question data to obtain evaluation results of the similarities corresponding to each of the similar question data; based on each of the evaluation results, the similar question data with a similarity higher than a preset evaluation threshold in the similar question data is determined as non-usable similar question data, and the similar question data with a similarity lower than or equal to a preset evaluation threshold in the similar question data is determined as available similar question data; the non-usable similar question data and the available similar question data are enhanced with positive and negative example data to obtain the first positive example data and the first negative example data corresponding to the non-usable similar question data, as well as the second positive example data and the second negative example data corresponding to the available similar question data; based on the first positive example data, the first negative example data, the second positive example data and the second negative example data, the preference data set is constructed. This embodiment uses an engineering system to implement the incremental generation of preference data sets by a large language model during the service process, thereby adjusting the model question generation preference and increasing the diversity and uniqueness of the questions generated by the large language model.
[0107] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 4 , Figure 4 This is a flow chart of the third embodiment of the speech large model training method of the present application; Figure 4 As shown, the large language model is optimized based on the preference data set to obtain a trained large language model. Step S40 may include steps S401 to S405:
[0108] Step S401, determining a training model and a comparison model based on the large language model;
[0109] Reference Figure 5 , Figure 5 This is a working logic diagram of the model training module proposed in the third embodiment of the model training method of this application; Figure 5 As shown, the model training module deploys a large language model to be trained, i.e., the training model in this step; and a large language model that does not need to be trained, i.e., the comparison model in this step. In each round of iteration of the large language model, the model parameters of the training model need to be updated, while the comparison model serves as a reference for iterative updates, and its own model parameters do not need to be updated as the number of iterations increases.
[0110] Step S402, inputting the preference data set into the training model and the comparison model, and obtaining a first model output score output by the training model and a second model output score output by the comparison model respectively;
[0111] In this embodiment, the prompt template corresponding to the preference data set is used to input the preference data set into the training model and the comparison model. For each model, the probability of generating each word is output, and the product probability of the generated word being a positive example data or a negative example data is calculated, thereby obtaining the first model output score and the second model output score. The model output score is used as input data to calculate the value of the loss function, thereby evaluating the performance of the current model and guiding the subsequent optimization process.
[0112] Step S403, calculating a model output loss based on the first model output score and the second model output score, so as to update a model parameter of the training model based on the model output loss;
[0113] In this embodiment, first, a loss function needs to be defined, which is used to quantify the difference between the model prediction value and the actual value. For classification tasks, commonly used loss functions include cross entropy loss, hinge loss, etc. In this embodiment, based on obtaining the first model output score and the second model output score, the loss function may be designed according to the score, for example, using a measure of the degree of difference between the scores as the basis for loss calculation.
[0114] The formula for calculating the model output loss is as follows:
[0115] loss = -log(σ(βlog(R policy / R reference )))
[0116] Among them, loss is the model output loss, σ is the formula normalization coefficient, which can be set according to actual needs, β is the hyperparameter in the DPO algorithm, which is used to control the KL divergence between the training model and the reference model. By adjusting the value of β, the best balance between model performance and stability can be found; R policy Output score for the first model, R reference Output a score for the second model.
[0117] If the preset training end condition is met, step S404 is executed: obtaining a trained large language model.
[0118] If the preset training end condition is not met, the process returns to step S402.
[0119] In this embodiment, the model parameters of the training model are continuously updated through an iterative update process until the preset conditions are met (such as stable model performance, the number of training rounds reaches an upper limit, etc.), and finally a trained large language model is obtained, thereby gradually optimizing the model performance through continuous iteration and feedback.
[0120] In a feasible implementation manner, the step of inputting the preference data set into the training model and the comparison model to respectively obtain a first model output score output by the training model and a second model output score output by the comparison model may include:
[0121] Step C1, inputting the preference data set into the training model and the comparison model, respectively obtaining a first probability and a second probability output by the training model and a third probability and a fourth probability output by the comparison model, wherein the first probability is the probability that the problem data output by the training model is positive data, the second probability is the probability that the problem data output by the training model is negative data, the third probability is the probability that the problem data output by the comparison model is positive data, and the fourth probability is the probability that the problem data output by the comparison model is negative data;
[0122] Specifically, Figure 5 As shown, this embodiment first needs to ensure that the preference data set has been prepared and the format is suitable for input into the training model and the comparison model; for each input question data, the training model will output two probability values: the first probability (P1) and the second probability (P2). P1 represents the probability that the model believes that the question generated by the model belongs to the positive data, while P2 represents the probability that the model believes that the question generated by the model belongs to the negative data. These two probability values are usually obtained by applying relevant functions (such as the softmax function) to the model output layer to ensure that their sum is 1. Similarly, the comparison model will also output two probability values for the questions generated by the model: the third probability (P3) and the fourth probability (P4). The meanings of P3 and P4 are the same as P1 and P2, but they are predictions based on the comparison model.
[0123] Step C2, calculating the ratio of the first probability to the second probability to obtain the first model output score, and calculating the ratio of the third probability to the fourth probability to obtain the second model output score.
[0124] Specifically, Figure 5 As shown, in this embodiment, the first model output score is obtained by calculating the ratio of the first probability to the second probability, that is, Figure 5 R_policy (policy score) in ; by calculating the ratio of the third probability to the fourth probability, the second model output score is obtained, that is, Figure 5 R_reference (reference score) in.
[0125] This embodiment uses the above scheme to determine a training model and a comparison model based on the large language model; inputs the preference data set into the training model and the comparison model to obtain a first model output score output by the training model and a second model output score output by the comparison model; calculates the model output loss based on the first model output score and the second model output score to update the model parameters of the training model based on the model output loss; returns to execute the step of inputting the preference data set into the training model and the comparison model to obtain a first model output score output by the training model and a second model output score output by the comparison model, until the preset training end condition is met, and a trained large language model is obtained. Based on the preference data set obtained, this embodiment generates questions with a low degree of similarity to existing questions based on the preference data set and the DPO algorithm, thereby increasing the diversity and creativity of the questions generated by the large language model and improving the quality of the questions generated by the large language model in the question generation system.
[0126] It should be noted that the above embodiments can be reasonably combined and implemented according to actual conditions, and this embodiment will not be described in detail.
[0127] This application also provides a language model training device, please refer to Figure 6 , the language model training device comprises:
[0128] A data acquisition module 10 is used to acquire original question data;
[0129] A similar question output module 20 is used to input the original question data into the large language model to obtain similar question data corresponding to the original question data;
[0130] A preference data set construction module 30 is used to enhance the positive and negative example data of the similar problem data to construct a preference data set;
[0131] The model training module 40 is used to optimize the large language model based on the preference data set to obtain a trained large language model.
[0132] Optionally, the preference data set construction module 30 is further used to:
[0133] Performing similarity evaluation on the similar question data to obtain evaluation results of the similarities corresponding to the similar question data;
[0134] Based on the evaluation results, similar question data with a similarity higher than a preset evaluation threshold in the similar question data are determined as non-usable similar question data, and similar question data with a similarity lower than or equal to the preset evaluation threshold in the similar question data are determined as usable similar question data;
[0135] Performing positive and negative example data enhancement on the non-available similar question data and the available similar question data to obtain first positive example data and first negative example data corresponding to the non-available similar question data, and second positive example data and second negative example data corresponding to the available similar question data;
[0136] The preference data set is constructed based on the first positive example data, the first negative example data, the second positive example data, and the second negative example data.
[0137] Optionally, the preference data set construction module 30 is further used to:
[0138] Determine the non-usable similar question data as the first negative example data, and delete the questions in the first negative example data whose similarity is higher than a preset screening threshold, to obtain the first positive example data;
[0139] The available similar question data is determined as the second positive example data, and the question is replicated on the second positive example data to obtain the second negative example data.
[0140] Optionally, the model training module 40 is further used to:
[0141] Based on the large language model, determining a training model and a comparison model;
[0142] Inputting the preference data set into the training model and the comparison model to obtain a first model output score output by the training model and a second model output score output by the comparison model, respectively;
[0143] Calculating a model output loss based on the first model output score and the second model output score to update a model parameter of the training model based on the model output loss;
[0144] Return to the step of inputting the preference data set into the training model and the comparison model to obtain a first model output score of the training model and a second model output score of the comparison model, respectively, until a preset training end condition is met, thereby obtaining a trained large language model.
[0145] Optionally, the model training module 40 is further used to:
[0146] Input the preference data set into the training model and the comparison model, and obtain a first probability and a second probability output by the training model and a third probability and a fourth probability output by the comparison model, respectively, wherein the first probability is the probability that the problem data output by the training model is positive data, the second probability is the probability that the problem data output by the training model is negative data, the third probability is the probability that the problem data output by the comparison model is positive data, and the fourth probability is the probability that the problem data output by the comparison model is negative data;
[0147] The ratio of the first probability to the second probability is calculated to obtain the first model output score, and the ratio of the third probability to the fourth probability is calculated to obtain the second model output score.
[0148] Optionally, the model training module 40 is further used to:
[0149] Performing vector form conversion based on the available similar problem data to obtain an embedded vector set;
[0150] Obtain target question data input by a user, and convert the target question data into a vector form based on the target question data to obtain a first embedding vector;
[0151] Determining, by a database retrieval method, from the set of embedding vectors a second embedding vector having the highest matching degree with the first embedding vector;
[0152] Generate target answer data corresponding to the target question data based on the second embedding vector.
[0153] The language model training device provided by the present application adopts the language model training method in the above embodiment, which can solve the technical problem of how to improve the quality of questions generated by a large language model in a question generation system. Compared with the prior art, the beneficial effects of the language model training device provided by the present application are the same as the beneficial effects of the language model training method provided by the above embodiment, and other technical features in the language model training device are the same as the features disclosed in the above embodiment method, which will not be repeated here.
[0154] The present application provides a language model training device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the language model training method in the above-mentioned embodiment 1.
[0155] Reference below Figure 7, which shows a schematic diagram of the structure of a language model training device suitable for implementing the embodiment of the present application. The language model training device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The language model training device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0156] like Figure 7 As shown, the language model training device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the language model training device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the language model training device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a language model training device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.
[0157] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.
[0158] The language model training device provided by the present application adopts the language model training method in the above embodiment, which can solve the technical problem of how to improve the quality of questions generated by a large language model in a question generation system. Compared with the prior art, the beneficial effects of the language model training device provided by the present application are the same as the beneficial effects of the language model training method provided by the above embodiment, and the other technical features in the language model training device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.
[0159] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.
[0160] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0161] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the language model training method in the above-mentioned embodiment.
[0162] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.
[0163] The computer-readable storage medium may be included in the language model training device; or may exist independently without being assembled into the language model training device.
[0164] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the language model training device, the language model training device: obtains original question data; inputs the original question data into the large language model to obtain similar question data corresponding to the original question data; performs positive and negative example data enhancement on the similar question data to construct a preference data set; optimizes the large language model based on the preference data set to obtain a trained large language model.
[0165] Computer program code for performing the operations of the present application may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0166] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0167] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.
[0168] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned language model training method, and can solve the technical problem of how to improve the quality of questions generated by a large language model in a question generation system. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the language model training method provided in the above-mentioned embodiment, and will not be repeated here.
[0169] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned language model training method when executed by a processor.
[0170] The computer program product provided in this application can solve the technical problem of how to improve the quality of questions generated by a large language model in a question generation system. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the language model training method provided in the above embodiment, and will not be repeated here.
[0171] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.
Claims
1. A language model training method, characterized in that: Applied to a question generation system, the question generation system deploying a large language model to be trained, the method comprising: Get the original question data; Inputting the original question data into the large language model to obtain similar question data corresponding to the original question data; Performing positive and negative example data enhancement on the similar question data to construct a preference data set; The large language model is optimized based on the preference data set to obtain a trained large language model.
2. The method according to claim 1, characterized in that The step of performing positive and negative example data enhancement on the similar question data to construct a preference data set includes: Performing similarity evaluation on the similar question data to obtain evaluation results of the similarities corresponding to the similar question data; Based on the evaluation results, similar question data with a similarity higher than a preset evaluation threshold in the similar question data are determined as non-usable similar question data, and similar question data with a similarity lower than or equal to the preset evaluation threshold in the similar question data are determined as usable similar question data; Performing positive and negative example data enhancement on the non-available similar question data and the available similar question data to obtain first positive example data and first negative example data corresponding to the non-available similar question data, and second positive example data and second negative example data corresponding to the available similar question data; The preference data set is constructed based on the first positive example data, the first negative example data, the second positive example data, and the second negative example data.
3. The method according to claim 2, characterized in that The step of performing positive and negative example data enhancement on the non-available similar problem data and the available similar problem data to obtain first positive example data and first negative example data corresponding to the non-available similar problem data, and second positive example data and second negative example data corresponding to the available similar problem data comprises: Determine the non-usable similar question data as the first negative example data, and delete the questions in the first negative example data whose similarity is higher than a preset screening threshold, to obtain the first positive example data; The available similar question data is determined as the second positive example data, and the question is replicated on the second positive example data to obtain the second negative example data.
4. The method according to claim 1, characterized in that The step of optimizing the large language model based on the preference data set to obtain a trained large language model comprises: Based on the large language model, determining a training model and a comparison model; Inputting the preference data set into the training model and the comparison model to obtain a first model output score output by the training model and a second model output score output by the comparison model, respectively; Calculating a model output loss based on the first model output score and the second model output score to update a model parameter of the training model based on the model output loss; Return to the step of inputting the preference data set into the training model and the comparison model to obtain a first model output score of the training model and a second model output score of the comparison model, respectively, until a preset training end condition is met, thereby obtaining a trained large language model.
5. The method according to claim 4, characterized in that The step of inputting the preference data set into the training model and the comparison model to obtain a first model output score output by the training model and a second model output score output by the comparison model respectively comprises: Input the preference data set into the training model and the comparison model, and obtain a first probability and a second probability output by the training model and a third probability and a fourth probability output by the comparison model, respectively, wherein the first probability is the probability that the problem data output by the training model is positive data, the second probability is the probability that the problem data output by the training model is negative data, the third probability is the probability that the problem data output by the comparison model is positive data, and the fourth probability is the probability that the problem data output by the comparison model is negative data; The ratio of the first probability to the second probability is calculated to obtain the first model output score, and the ratio of the third probability to the fourth probability is calculated to obtain the second model output score.
6. The method according to claim 2, characterized in that After the step of determining, based on each of the evaluation results, similar question data with a similarity higher than a preset evaluation threshold in the similar question data as non-usable similar question data, and determining similar question data with a similarity lower than or equal to the preset evaluation threshold in the similar question data as usable similar question data, the method further includes: Performing vector form conversion based on the available similar problem data to obtain an embedded vector set; Obtain target question data input by a user, and convert the target question data into a vector form based on the target question data to obtain a first embedding vector; Determining, by a database retrieval method, from the set of embedding vectors a second embedding vector having the highest matching degree with the first embedding vector; Generate target answer data corresponding to the target question data based on the second embedding vector.
7. A language model training device, characterized in that: The device comprises: Data acquisition module, used to obtain original question data; A similar question output module, used for inputting the original question data into the large language model to obtain similar question data corresponding to the original question data; A preference data set construction module, used to enhance the positive and negative example data of the similar problem data to construct a preference data set; The model training module is used to optimize the large language model based on the preference data set to obtain a trained large language model.
8. A language model training device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the language model training method according to any one of claims 1 to 6.
9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the language model training method according to any one of claims 1 to 6 are implemented.
10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the language model training method according to any one of claims 1 to 6 are implemented.
Citation Information
Patent Citations
Natural resource industry knowledge base construction method and system based on large language model
CN117172319A
Extraction type question and answer data enhancement method based on parrot retelling generator
CN117216231A
Problem generation method and device based on context information, equipment and medium
CN118152529A
Knowledge question-answering method and system based on large language model
CN118210890A
GQL corpus generation model training method and device and storage medium
CN118410060A
