Reading model training method, reviewing method and device

Abstract

The invention provides a review model training method, review method and device, and the method comprises the steps: carrying out the unsupervised training of a large-scale language model according to the first sample data in a first sample data set, and obtaining a first language model; performing supervised training on the first language model according to second sample data in the second sample data set and score tags and comment tags corresponding to the second sample data to obtain a second language model; performing reinforcement learning training on the second language model according to third sample data in the third sample data set and the score labels corresponding to the third sample data to obtain a review model; the number of the sample data in the first sample data set and the number of the sample data in the third sample data set are both larger than the number of the sample data in the second sample data set. According to the method, the review model with relatively high review quality can be effectively trained through a small number of review labels, the labeling cost is effectively reduced, and meanwhile, the review quality is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a review model training method, a review method and a device. Background Art

[0002] Intelligent review of students' answers based on large language models can assist students in learning, help teachers correct and diagnose students' homework, etc., thereby reducing teachers' teaching pressure, improving review efficiency, and alleviating the uneven distribution of educational resources. Therefore, how to conduct intelligent review efficiently and conveniently is an important topic that needs to be studied urgently.

[0003] In related technologies, it is usually necessary to use a large number of high-quality comment labels and high-quality rating labels to fine-tune and train a large language model in order to obtain a review model with high review quality, thereby ensuring the quality of the review.

[0004] However, due to the different professional capabilities of labelers, the high difficulty of labeling, and the time and economic costs of labeling, there is a shortage of high-quality comment labels. Therefore, it is difficult to train a review model with high review quality, which limits the quality of the review. Summary of the invention

[0005] The present invention provides a review model training method, a review method and a device, which are used to solve the defect in the prior art that the review quality is difficult to guarantee when high-quality review labels are scarce, and to improve the review quality when high-quality review labels are scarce.

[0006] The present invention provides a review model training method, comprising: Performing unsupervised training on a large language model according to first sample data in a first sample data set to obtain a first language model; According to the second sample data in the second sample data set and the score labels and comment labels corresponding to each of the second sample data, the first language model is subjected to supervised training to obtain a second language model; According to the third sample data in the third sample data set and the scoring labels corresponding to the third sample data, the second language model is subjected to reinforcement learning training to obtain a review model; The number of sample data in the first sample data set and the number of sample data in the third sample data set are both greater than the number of sample data in the second sample data set.

[0007] According to a review model training method provided by the present invention, the second language model is subjected to reinforcement learning training according to the third sample data in the third sample data set and the scoring labels corresponding to the third sample data to obtain the review model, including: Input each of the third sample data into the second language model to obtain a review data set corresponding to each of the third sample data output by the second language model; Calculate the difference between the score prediction data of each review data in the review data set corresponding to each of the third sample data and the score label corresponding to each of the third sample data, and determine the positive review data and the negative review data corresponding to each of the third sample data; According to each of the third sample data, and the positive review data and the negative review data corresponding to each of the third sample data, reinforcement learning training is performed on the second language model to obtain the review model.

[0008] According to a review model training method provided by the present invention, the second language model is subjected to reinforcement learning training according to each of the third sample data and the positive review data and the negative review data corresponding to each of the third sample data to obtain the review model, including: For the current round of training, each of the third sample data is input into the language model to be trained after the previous round of training, and a first probability that the predicted review data corresponding to each of the third sample data belongs to the positive review data and a second probability that the predicted review data belongs to the negative review data are obtained; Input each of the third sample data into the second language model to obtain a third probability that the predicted review data belongs to the positive review data and a fourth probability that the predicted review data belongs to the negative review data; Constructing a target loss function according to the first probability, the second probability, the third probability and the fourth probability; Based on the target loss function, iteratively train the language model to be trained after the previous round of training to obtain the language model to be trained after the current round of training; The review model is constructed according to the language model to be trained after each round of training.

[0009] According to a review model training method provided by the present invention, constructing a target loss function according to the first probability, the second probability, the third probability and the fourth probability includes: Calculating a first difference value according to the first probability and the third probability; Calculating a second difference value according to the second probability and the fourth probability; Constructing the target loss function according to the first difference value and the second difference value; The first difference value is the predicted difference value between the language model to be trained after the previous round of training and the second language model on the positive review data, and the second difference value is the predicted difference value between the language model to be trained after the previous round of training and the second language model on the negative review data.

[0010] According to a review model training method provided by the present invention, unsupervised training is performed on a large language model according to first sample data in a first sample data set to obtain a first language model, including: Divide each of the first sample data into a plurality of sample data unit sequences, and obtain, in the first sample data set, a next sample data unit corresponding to each of the sample data unit sequences; Inputting each of the sample data unit sequences into the large language model to obtain a first log-likelihood probability that the next predicted data unit corresponding to each of the sample data unit sequences belongs to the next sample data unit; The first language model is obtained by performing unsupervised training on the large language model with the goal of maximizing the sum of first log-likelihood probabilities corresponding to all the sample data unit sequences in the first sample data set.

[0011] According to a review model training method provided by the present invention, the first language model is supervisedly trained according to the second sample data in the second sample data set and the score labels and comment labels corresponding to each of the second sample data to obtain the second language model, including: Input each of the second sample data into the first language model to obtain a second log-likelihood probability that the score prediction data corresponding to each of the second sample data belongs to the score label corresponding to each of the second sample data and a third log-likelihood probability that the comment prediction data corresponding to each of the second sample data belongs to the comment label corresponding to each of the second sample data; With the goal of maximizing the sum of the second log-likelihood probability and the third log-likelihood probability corresponding to all second sample data in the second sample data set, supervised training is performed on the first language model to obtain the second language model.

[0012] According to a review model training method provided by the present invention, each of the first sample data, the second sample data and the third sample data includes sample question data, sample answer data and sample answer data.

[0013] The present invention also provides a review method, comprising: Acquire the data to be reviewed, wherein the data to be reviewed includes target question data, target answer data and target answer data; Inputting the data to be reviewed into the review model to obtain the score prediction data and comment prediction data corresponding to the data to be reviewed; Wherein, the review model is trained based on any of the review model training methods described above.

[0014] The present invention also provides a review model training device, comprising: A first training unit is used to perform unsupervised training on the large language model according to first sample data in the first sample data set to obtain a first language model; A second training unit is used to perform supervised training on the first language model according to the second sample data in the second sample data set and the score labels and comment labels corresponding to each of the second sample data, so as to obtain a second language model; A third training unit is used to perform reinforcement learning training on the second language model according to the third sample data in the third sample data set and the scoring labels corresponding to the third sample data to obtain a review model; The number of sample data in the first sample data set and the number of sample data in the third sample data set are both greater than the number of sample data in the second sample data set.

[0015] The present invention also provides a review device, comprising: An acquisition unit, used for acquiring data to be reviewed, wherein the data to be reviewed includes target question data, target answer data and target answer data; A review unit, used for inputting the data to be reviewed into a review model to obtain score prediction data and comment prediction data corresponding to the data to be reviewed; Wherein, the review model is trained based on any of the review model training methods described above.

[0016] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the review model training method or review method described above is implemented.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the review model training methods or review methods described above.

[0018] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the review model training method or review method described above is implemented.

[0019] The review model training method, review method and device provided by the present invention first use a large amount of sample data in a first sample data set to perform unsupervised training on a large language model, so as to obtain a first language model with certain basic subject knowledge and problem-solving comprehension ability through unlabeled training, and use a small amount of sample data in a second sample data set and its corresponding scoring labels and comment labels to perform supervised training on the first language model, so as to fine-tune the model through a small amount of high-quality scoring comment label data, so as to obtain a second language model with instruction following ability and learning and mastering review scoring and correction tasks, and use a large amount of sample data with scoring labels in a third sample data set to perform reinforcement learning training of scoring feedback on the second language model, so as to constrain the large language model to generate high-quality comments through accurate scoring labels, so as to obtain a review model with accurate comment generation ability and scoring and correction ability, thereby reducing the dependence on high-quality comment labels and being able to effectively train a review model with high review quality, effectively reducing the time cost and economic cost of annotation, and at the same time improving the quality of the review model and the accuracy of the review, thereby improving the overall quality of the review. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is one of the flow charts of the review model training method provided by the present invention.

[0022] Figure 2 This is the second flow chart of the review model training method provided by the present invention.

[0023] Figure 3 It is a schematic diagram of the flow of the review method provided by the present invention.

[0024] Figure 4 It is a structural schematic diagram of the review model training device provided by the present invention.

[0025] Figure 5 It is a structural schematic diagram of the review device provided by the present invention.

[0026] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] The field of natural language processing is an important branch of artificial intelligence. The research goal of this field is to enable computers to understand and generate natural language, that is, the language used in daily life, such as Chinese and English. Common tasks involved in the field of natural language processing include machine translation, speech recognition, sentiment analysis, text summarization, named entity recognition, dialogue systems and chatbots. In recent years, large language models (LLMs), referred to as large language models or large models, have broken the barriers between different tasks in the field of natural language processing, and have shown high performance in many natural language processing tasks. Among them, the field of education is one of the landing scenarios of natural language processing technology. Common applications in this field include intelligent review of student answer information (hereinafter referred to as comment scoring), which uses large language models to understand the questions, standard answers and student answers, diagnose student answers, and give a correction comment in the form of a chain of thinking to point out the student's error location and error cause, and then give the whole question correction process score to assist students in learning and help teachers correct and diagnose students' homework, thereby reducing teachers' teaching pressure, improving review efficiency, and alleviating the uneven distribution of educational resources.

[0029] Both pre-training and fine-tuning of large language models require massive amounts of labeled data, and the quality of the labeled data has a crucial impact on the performance of large language models. Due to the small amount of labeled data in vertical fields, the training data of large language models in vertical fields requires hiring a large number of labelers for large-scale labeling. Therefore, how to design effective technical solutions and achieve results when it is difficult to obtain a large amount of high-quality labeled data in vertical fields is a technical difficulty.

[0030] Common intelligent education review products on the market, such as learning machines, usually have certain correction and scoring capabilities, but do not have the ability to correct students' specific mistakes. With the development of large language models and natural language processing technology, intelligent correction through the thinking chain of large language models has become a possible solution to this pain point.

[0031] In practical applications, if the review model to be trained needs to have the ability to perform a high-performance two-step thinking chain of comments and ratings, it is necessary to use a large amount of high-quality comment annotation data and high-quality rating annotation data to fine-tune the large language model.

[0032] High-quality rating and annotation data can usually be collected in many educational grading scenarios, but high-quality comment annotation data is often scarce and very expensive, requiring a lot of manpower for data annotation. In addition to the high price of data annotation, the professional capabilities of annotators vary, the annotation difficulty is high, and the time and economic costs of annotation cannot guarantee the effective acquisition of high-quality comment annotation data, resulting in a shortage of high-quality comment labels.

[0033] However, since there is little comment annotation data in the training data, but a lot of score annotation data, the large language model does not learn the comments sufficiently. In the stage of chain reasoning, it is easy to generate incorrect comments, which leads to inaccurate scores. This makes it impossible to train a review model with high review quality. For example, when a large language model generates comments, it mistakenly believes that the student's answer is correct, which can easily induce the model to give a full score. In addition, when only using the same amount of score annotation data as the comment score annotation data for training, although this problem can be alleviated, a large amount of score annotation data cannot be learned by the model, which will result in low data utilization efficiency, and it is also impossible to train a review model with high review quality by relying only on a small amount of annotation data.

[0034] In addition, the scoring task is a regression task, and the difficulty of model learning is relatively low when the amount of data is sufficient. The review task is a generation task, and the semantic space of the review is very large. The difficulty difference between the two tasks is large. When the amount of training data for the two tasks is unbalanced, the model is likely to fall into a local optimum. In this case, the model is likely to produce many incomprehensible hallucination behaviors. For example, the model is likely to generate comments that appear to be correct but are actually wrong to mislead users, which will also make it impossible to train a review model with high review quality.

[0035] In this regard, the present embodiment provides a review model training method, which first performs domain pre-training on a large language model so that the large language model acquires domain knowledge such as grading and problem solving, and is familiar with the data format of questions and standard answers. A small number of high-quality comment labels and scoring labels that have been manually labeled are then used to perform supervised fine-tuning on the large language model after the first stage of training, so as to activate the instruction following ability of the trained large language model and learn to master the scoring and commenting tasks. Finally, a large number of manually verified scoring labels are used to further strengthen the learning and training of the large language model after the second stage of training, so as to constrain the comments generated by the large language model through reinforcement learning of scoring feedback, allowing all training data to be efficiently used in the case of imbalanced training data, so as to achieve the effect of taking into account both the comment generation ability and the scoring ability, thereby improving the comment generation ability and the scoring and grading ability of the trained large model and improving the quality of the review.

[0036] It should be noted that the execution subject of the method may be an electronic device. The electronic device may be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device may be an intelligent educational product, such as a learning device, or may be a mobile phone, a tablet computer, a laptop computer, a PDA, etc. The non-mobile electronic device may be a server, a personal computer, etc., which is not specifically limited in this embodiment.

[0037] This method aims to obtain a review model that can accurately generate high-quality comments and scores through training of a large language model, thereby having excellent grading capabilities to ensure the quality of the review.

[0038] Figure 1 FIG. 1 is one of the flow charts of the review model training method provided by the present invention. Figure 1 As shown, the method includes: step 110, step 120 and step 130.

[0039] Step 110: Perform unsupervised training on a large language model according to first sample data in the first sample data set to obtain a first language model.

[0040] Optionally, during the training process of the review model, the large language model can be first trained in the first stage, that is, the question bank pre-training stage, so as to perform domain pre-training on the large language model, so that the pre-trained large language model, that is, the first language model, can learn the common knowledge and rules in various review fields, so as to accurately obtain domain knowledge such as grading and problem solving, and at the same time be familiar with the question data and answer data format, so as to have certain basic subject knowledge and problem-solving comprehension ability.

[0041] Among them, the specific implementation steps of the training in the question bank pre-training stage are as follows: First, a large amount of unlabeled question bank data exceeding the first amount is collected from the Internet or a domain corpus to obtain first sample data, and the first sample data set is formed by arranging the data. , thus obtaining the first stage training sample set; here The first sample data is sample data. In some embodiments, each first sample data is sample data that needs to be marked for review in the historical review process, including sample question data, sample answer data, and sample answer data.

[0042] These first sample data are stored in text form and can be further processed into a format suitable for training large language models.

[0043] For example, the first sample data may be specifically represented as follows: Question: Xiao Ming has 10 apples and Xiao Hong has 5 apples. How many more apples does Xiao Ming have than Xiao Hong? Standard answer: 10-5=5. Answer: Xiao Ming has 5 more apples than Xiao Hong.

[0044] Student’s answer: 10+5=15.

[0045] Then, based on the first sample data in the first sample data set, the large language model is iteratively trained for unsupervised learning, so as to update the general large model that only has general domain knowledge to the education grading field with professional knowledge of the question bank and the logical reasoning ability in the certain education grading field. In this way, the first language model is obtained, which can accurately learn the domain knowledge of grading and problem solving, and is familiar with the format of question data and answer data, as well as the potential relationship between question data, answer data and answer data, so as to lay the foundation for subsequent model training.

[0046] Here, unsupervised pre-training may be achieved through autoregressive prediction training or self-supervised learning, etc., and this embodiment does not specifically limit this.

[0047] Here, a large language model refers to a natural language processing (NLP) model with a large parameter scale. The number of model parameters and / or the complexity of the model structure exceeds a preset threshold. The model is pre-trained on large-scale general-domain text data and has a high degree of semantic understanding and the ability to generate natural language.

[0048] Step 120 , performing supervised training on the first language model according to the second sample data in the second sample data set and the rating labels and comment labels corresponding to each of the second sample data, to obtain a second language model.

[0049] Optionally, after the first language model is obtained through the first stage of training, the second stage of training, that is, the supervised fine-tuning stage of comment scoring training, can be further performed to fine-tune the first language model obtained through the first stage of training with a small number of high-quality comment labels and score labels that have been manually labeled, thereby activating the instruction-following ability of the large language model and learning to master the comment scoring and grading tasks, so that the first language model fine-tuned and trained in this way, that is, the second language model, has the instruction-following ability to effectively learn to master the comment scoring tasks.

[0050] Among them, the specific implementation steps of the second stage of training are as follows: First, collect a small amount of high-quality scoring data that has been manually verified, which is larger than the third amount, to form the training sample set for the second stage. ; The training sample set of the second stage includes each second sample data in the second sample data set, and the high-quality scoring label and high-quality comment label corresponding to each second sample data. is the sample data, is the comment label corresponding to the sample data; is the score label corresponding to the sample data.

[0051] In some embodiments, each second sample data is also sample data that needs to be reviewed and marked in the historical review process, and the type of data content included is the same as the first sample data, that is, each second sample data also includes sample question data, sample answer data and sample answer data. The second number is much smaller than the first number.

[0052] Among them, the comment label can be a comment label formed by pointing out the wrong position and type of the student's answer in a fine-grained manner during the marking process, and generating a corresponding error cause analysis text, that is, the comment label at least includes error position information, error type and error analysis information; the scoring label can be the result of correcting the process points of the whole question during the marking process, and it is necessary not only to consider whether the student's answer to the question is correct, but also to refer to the comments and the scoring points of the problem-solving steps to give appropriate process points when the student's answer is not completely correct, and mark the output score rate value between 0 and 1. Therefore, the scoring label can be a correction score rate in [0,1] expressed with two decimal places.

[0053] For example, the examples of the second sample data and its corresponding labels can be specifically represented as follows: Question: Xiao Ming has 10 apples and Xiao Hong has 5 apples. How many more apples does Xiao Ming have than Xiao Hong? Standard answer: 10-5=5. Answer: Xiao Ming has 5 more apples than Xiao Hong.

[0054] Student’s answer: 10+5=15.

[0055] Error location information: first line.

[0056] Error Analysis Message: You have confused the concepts of addition and subtraction.

[0057] Correction result: 0.00.

[0058] Then, based on the second sample data in the second sample data set, and the scoring labels and comment labels corresponding to each second sample data, supervised fine-tuning training is performed on the first language model obtained in the first stage of training, so as to customize the training in the vertical field of test question review and optimize the parameters of the language model, so as to further activate the instruction following ability of the language model, learn and master the grading tasks of comments and scores, and obtain the second language model, so that the second language model has good performance in the test question review task.

[0059] The supervised fine-tuning training here can be obtained by iterative training based on a loss function constructed based on the error between the score prediction data corresponding to each second sample data output by the first language model and the score label, and the error between the comment prediction data corresponding to each second sample data output by the first language model and the comment label; it can also be obtained by iterative training based on a loss function constructed based on the log-likelihood probability between the score prediction data corresponding to each second sample data output by the first language model and the score label, and the log-likelihood probability between the comment prediction data corresponding to each second sample data output by the first language model and the comment label, which is not specifically limited in this embodiment.

[0060] Step 130, performing reinforcement learning training on the second language model according to the third sample data in the third sample data set and the scoring labels corresponding to the third sample data to obtain a review model; The number of sample data in the first sample data set and the number of sample data in the third sample data set are both greater than the number of sample data in the second sample data set.

[0061] In some embodiments, each of the first sample data, the second sample data, and the third sample data includes sample question data, sample response data, and sample answer data.

[0062] It should be noted that after the second stage of fine-tuning training, the trained second language model has the ability to complete the review and rating correction tasks, but due to the small amount of training data in this stage, the trained second language model has not achieved the optimal effect in the two sub-tasks of comment generation and rating generation. Therefore, based on the second language model, the third stage of training can be further performed, that is, the reinforcement learning stage based on rating feedback, to perform reinforcement learning training on the second language model, constrain the large language model to generate high-quality comments, and further improve the model's comment generation ability and rating correction ability, so as to finally obtain a review model that can accurately generate high-quality comments and ratings and has excellent thinking chain ability and correction ability.

[0063] Among them, the specific implementation steps of the third stage of training are as follows: First, collect a large amount of high-quality scoring data that has been manually verified, which is larger than the third amount, to form the training sample set for the third stage. ; The training sample set of the third stage includes each third sample data in the third sample data set, and the high-quality scoring label corresponding to each third sample data. is the sample data, is the score label corresponding to the sample data.

[0064] In some embodiments, each third sample data is also sample data that needs to be reviewed and marked in the historical review process, and the type of data content included therein is the same as that of the first sample data and the second sample data, that is, each third sample data also includes sample question data, sample response data and sample answer data; wherein, the third quantity is much larger than the second quantity; the third quantity may be the same as or different from the first quantity, and this embodiment does not specifically limit this.

[0065] Then, based on each third sample data in the third sample data set and the scoring label corresponding to each third sample data, the second language model obtained in the second stage of training is subjected to reinforcement learning training with scoring feedback, so as to reversely improve the comment generation ability with the help of the accurate scoring and correction ability of the second language model, thereby achieving a self-evolutionary training effect, further improving the comment generation ability and scoring correction ability of the large model, and ultimately achieving a large language model with thinking chain ability and high-performance comment scoring ability by fine-tuning training using only a small number of comment labels and a large number of scoring labels.

[0066] The reinforcement learning training of rating feedback here can be achieved through reinforcement learning training of human feedback, or can be achieved by combining reinforcement learning of human feedback and direct preference optimization, which is not specifically limited in this embodiment.

[0067] The method provided in this embodiment first uses a large amount of sample data in the first sample data set to perform unsupervised training on a large language model, so as to obtain a first language model with certain basic subject knowledge and problem-solving comprehension ability through unlabeled training, and uses a small amount of sample data in the second sample data set and its corresponding rating labels and comment labels to perform supervised training on the first language model, so as to fine-tune the model through a small amount of high-quality rating comment label data, so as to obtain a second language model with instruction following ability and learning and mastering comment rating and correction tasks, and uses a large amount of sample data with rating labels in the third sample data set to perform reinforcement learning training of rating feedback on the second language model, so as to constrain the large language model to generate high-quality comments through accurate rating labels, and obtain a review model that can accurately perform comment generation and rating and correction capabilities, thereby reducing the dependence on high-quality comment labels while effectively training a review model with high review quality, effectively reducing the time cost and economic cost of annotation, and at the same time improving the quality of the review model and the accuracy of the review, thereby improving the overall quality of the review.

[0068] In some embodiments, step 130 specifically includes: Step 131, inputting each of the third sample data into the second language model to obtain a review data set corresponding to each of the third sample data output by the second language model; each review data in the review data set includes score prediction data and comment prediction data; Step 132, calculating the difference between the score prediction data of each review data in the review data set corresponding to each of the third sample data and the score label corresponding to each of the third sample data, and determining the positive review data and the negative review data corresponding to each of the third sample data; Step 133, performing reinforcement learning training on the second language model according to each of the third sample data and the positive review data and the negative review data corresponding to each of the third sample data, to obtain the review model; The review error corresponding to the positive review data is smaller than the review error corresponding to the negative review data.

[0069] Optionally, for the third stage of reinforcement learning based on rating feedback, the second language model with the ability to rate and correct comments obtained through fine-tuning training in the second stage can be used first, and then the third sample data can be used to train the second language model. The test questions are corrected on the test paper to obtain a large amount of review data of each third sample data, forming a review data set corresponding to each third sample data. Among them, the review data set corresponding to each third sample data It contains a large amount of review generation results (i.e. review prediction data) and rating correction results (i.e. rating prediction data).

[0070] Then, the score prediction data of each review data corresponding to each third sample data is Compare with the score label z corresponding to each third sample data to calculate the difference between the score prediction data of each review data corresponding to each third sample data and the score label corresponding to each third sample data, and obtain the review error corresponding to each review data. According to the review error corresponding to each review data, screen out the review data with large consistency difference between the two score prediction data, and construct the correct positive review data and the incorrectly corrected negative review data corresponding to each third sample data to obtain high-quality artificial preference data. The positive review data is the score prediction data of the second language model. The review data that is the same as or has a small difference with the corresponding manually marked score label z. Negative review data is the score prediction data of the second language model. The review data with the corresponding manually labeled rating label z is larger.

[0071] The screening of the review data here can be to take the review data with a review error greater than a certain error threshold as negative review data, and the review data with a review error less than a certain error threshold as positive review data; or, to sort the review errors corresponding to each review data under each third sample data according to the quality of the correction results, so as to take the review data with the largest review error as negative review data, and the review data with the smallest review error as positive review data, etc. This embodiment does not make specific limitations on this.

[0072] The positive review data and the negative review data are described in detail below using specific examples.

[0073] For the sample data of the same student answering the same question, the second language model predicts and outputs two different correction results. The score prediction data formed by the first correction is 0.50, and the score prediction data formed by the second correction is 0.00. The score label of the manual score corresponding to this sample data is 0.50. Since the score prediction data formed by the first correction is consistent with the score label of the manual score, the review data formed by the first correction is positive review data. Since the score prediction data formed by the second correction is significantly different from the score label of the manual score, the review data formed by the first correction is negative review data. Therefore, based on these two trial correction samples, the following positive and negative data sample pairs can be constructed: The positive data samples can be specifically expressed as follows: "[Question] Xiao Ming originally had 10 apples and ate 2 apples. Xiao Hong had 5 apples. How many more apples does Xiao Ming have than Xiao Hong now? [Standard answer] 10-2=8 (pieces), 8-5=3 (pieces). Answer: Xiao Ming has 3 more apples than Xiao Hong.

Student answer

Error location

Correction result

[0074] The negative data samples can be specifically expressed as follows: "[Question] Xiao Ming originally had 10 apples and ate 2 apples. Xiao Hong had 5 apples. How many more apples does Xiao Ming have than Xiao Hong now? [Standard answer] 10-2=8 (pieces) 8-5=3 (pieces) Answer: Xiao Ming has 3 more apples than Xiao Hong

Student answer

Error location

Correction result

[0075] After obtaining the positive review data and the negative review data, the second language model can be combined with each third sample data, as well as the positive review data and the negative review data corresponding to each third sample data, to perform reinforcement learning training, so as to constrain the comments generated by the large language model through reinforcement learning of scoring feedback, improve the correction accuracy of the large language model, alleviate the large language model hallucination phenomenon, and thus achieve efficient use of all training data in the case of unbalanced training data, and thus fine-tune the training to obtain a large language model with thinking chain ability and high-performance comment scoring ability.

[0076] The reinforcement learning based on scoring feedback here can be implemented by a reinforcement learning algorithm of the synchronous policy gradient method and the value function method in reinforcement learning based on human feedback (abbreviated as the A2C method of RLHF, or the Actor-Critic method of RLHF), or it can be implemented by a direct preference optimization algorithm in reinforcement learning based on human feedback, etc. This embodiment does not specifically limit this.

[0077] Figure 2 This is the second flow chart of the review model training method provided by the present invention. Figure 2 As shown, the following four models can be included in the A2C method of RLHF: The strategy model (also called the Actor model) is initially obtained by fine-tuning the second language model. As a strategy model, it is used to receive context and take corresponding actions.

[0078] The reference model (also called the Reference model) is also initially obtained by fine-tuning the second language model. The parameters are frozen during the training process to interact with the policy model to ensure that the trained policy model does not deviate from the fine-tuned second language model.

[0079] The reward model (also called the Reward model or RW model) freezes the parameters during the training process and gives a corresponding reward value for each state.

[0080] The evaluation network (also called the Critic model) is initialized by the reward model and is used to approximate the value function. The input is the state and the output is the value of the state.

[0081] Among them, the training steps of RLHF's A2C method specifically include: In the first stage, based on the human-annotated data, that is, each second sample data and the rating label and comment label corresponding to each second sample data, the pre-trained model, that is, the first language model, is fine-tuned in a supervised manner to generate a fine-tuned model (also called an SFT model), that is, the second language model.

[0082] In the second stage, positive and negative sample data are collected, that is, the positive review data and negative review data corresponding to each third sample data, and the Reward model is trained to evaluate the review quality and provide reward signals for subsequent reinforcement learning.

[0083] In the third stage, the SFT model is used as the initialization model of the Actor model, and the model parameters are smoothed by an optional exponential moving average (EMA) to improve training stability; the Reward model is used as the Critic model to evaluate the behavior of the Actor model; the frozen SFT model is used as the Reference model to provide a stable comparison benchmark; the frozen Reward model is used to output the reward signal. The Proximal Policy Optimization (PPO) algorithm is used to perform the following reinforcement learning training on the model: Generate data: Actor model generates data; Calculate rewards: Use the Reward model to calculate the rewards for generating data; Optimization: Use the reinforcement learning algorithm of the policy gradient method and the value function method (referred to as the Actor-Critic algorithm) for optimization.

[0084] The implementation process of the Actor-Critic algorithm includes: Initialize the policy parameters and value function (also known as baseline function) parameters; In each interaction cycle, for each time step t, the following steps are performed: According to the current strategy Select Action ; Calculate the discounted reward corresponding to time step t. Among them, the discounted reward corresponding to time step t The calculation formula is: ; is the discount factor; is the time step Rewards received when Calculate the baseline function value corresponding to time step t; the calculation formula of the baseline function value is: ; in, is the baseline function for the state estimates; The baseline is refitted by minimizing the difference between the baseline function and the discounted reward, which is calculated as follows: ; Update policy parameters using gradient ascent , in order to increase the probability of producing high-advantage actions, the specific calculation formula is as follows: ; in, is the learning rate; is the strategy parameter The gradient operator.

[0085] Through the above training method, RLHF's A2C model can learn behaviors that are more in line with human preferences, while taking advantage of reinforcement learning to optimize the model's decision-making process, so that the trained review model can accurately handle review tasks.

[0086] The method provided in this embodiment uses rating labels to perform reinforcement learning training based on rating feedback on a second language model with review rating capabilities. This can improve the review performance of the model through reinforcement learning training under the condition of limited high-quality review labels and rating labels, so that the model can perform rating and comment generation more accurately, while reducing the dependence on a large amount of manually annotated data and improving the efficiency and quality of the review.

[0087] In some embodiments, step 134 specifically includes: For the current round of training, each of the third sample data is input into the language model to be trained after the previous round of training, and a first probability that the predicted review data corresponding to each of the third sample data belongs to the positive review data and a second probability that the predicted review data belongs to the negative review data are obtained; Input each of the third sample data into the second language model to obtain a third probability that the predicted review data belongs to the positive review data and a fourth probability that the predicted review data belongs to the negative review data; Constructing a target loss function according to the first probability, the second probability, the third probability and the fourth probability; Based on the target loss function, iteratively train the language model to be trained after the previous round of training to obtain the language model to be trained after the current round of training; Constructing the review model according to the language model to be trained after each round of training; Among them, the language model to be trained is constructed based on the second language model; the first probability is the probability value of the review data output by the language model to be trained after the previous round of training belonging to the positive review data, the second probability is the probability value of the review data output by the language model to be trained after the previous round of training belonging to the negative review data, the third probability is the probability value of the review data output by the second language model belonging to the positive review data, and the fourth probability is the probability value of the review data output by the second language model belonging to the negative review data.

[0088] Optionally, in the reinforcement learning stage, the language model to be trained is constructed by initializing the second language model obtained by the fine-tuning training in the second stage.

[0089] For each round of training, the following steps can be performed: Each third sample data is input into the language model to be trained after the previous round of training, so that the language model to be trained after the previous round of training predicts the probability value of the review data corresponding to each third sample data belonging to the positive review data based on the given context information corresponding to each third sample data, so as to obtain the first probability corresponding to each third sample data, and the language model to be trained after the previous round of training predicts the probability value of the review data corresponding to each third sample data belonging to the negative review data based on the given context information corresponding to each third sample data, so as to obtain the second probability corresponding to each third sample data.

[0090] Furthermore, each third sample data is input into the second language model, so that the second language model predicts, based on the given context information corresponding to each third sample data, the probability value of the review data corresponding to each third sample data being positive review data, and obtains the third probability corresponding to each third sample data; and the second language model predicts, based on the given context information corresponding to each third sample data, the probability value of the review data corresponding to each third sample data being negative review data, and obtains the fourth probability corresponding to each third sample data.

[0091] Then, the first probability, the second probability, the third probability and the fourth probability corresponding to each third sample data are combined, and the target loss function is constructed with the goal of maximizing the probability value that the review data output by the language model to be trained after the previous round of training belongs to the positive review data, and minimizing the probability value that the review data output by the language model to be trained after the previous round of training belongs to the negative review data.

[0092] Therefore, based on the target loss function, the language model to be trained after the previous round of training is iteratively reinforced learning training to obtain the language model to be trained after the current round of training.

[0093] According to the above steps, each round of model training is iteratively performed to obtain a language model to be trained after each round of training, and the language model to be trained with the best model performance is selected from the language models to be trained after multiple rounds of training to construct a review model.

[0094] The method provided in this embodiment uses positive and negative sample data corresponding to the scoring labels to perform reinforcement learning training based on scoring feedback on a second language model with review scoring capability. This can improve the review performance of the model through reinforcement learning training under the condition of limited high-quality review labels and scoring labels, so that the model can perform scoring and comment generation more accurately, while reducing the dependence on a large amount of manually annotated data and improving the efficiency and quality of the review.

[0095] In some embodiments, constructing a target loss function according to the first probability, the second probability, the third probability and the fourth probability includes: Calculating a first difference value according to the first probability and the third probability; Calculating a second difference value according to the second probability and the fourth probability; Constructing the target loss function according to the first difference value and the second difference value; The first difference value is the predicted difference value between the language model to be trained after the previous round of training and the second language model on the positive review data, and the second difference value is the predicted difference value between the language model to be trained after the previous round of training and the second language model on the negative review data.

[0096] Optionally, during the reinforcement learning phase, you can use Represents the large language model in training, that is, the language model to be trained after the last round of training. It represents the large language model obtained in the supervised fine-tuning stage, that is, the second language model. It should be noted that the two are equal in the initial stage of the reinforcement learning stage before training. That is, the language model to be trained in the initial stage is constructed based on the second language model.

[0097] in, The state at time t It can be defined as the state of the first t tokens generated by the model, which can be expressed as At time t+1, the large model uses the sample data And the generated token predicts the next token, which corresponds to the action in reinforcement learning, denoted as .

[0098] The training goal of the reinforcement learning phase is to maximize the reward , while minimizing and The KL divergence between is calculated as follows: ; in, is the expected function; is the data distribution of the sample data; is a hyperparameter used to balance the weight between maximizing reward and minimizing KL divergence.

[0099] In the optimization process, the direct preference optimization algorithm can be used to solve the problem. The algorithm is based on Bradley-Terry preference modeling and implicitly defines a reward function related to the large language model so that the model The generated output score data and comment data are relative to the model While being more preferred, the characteristics of the dataset are taken into consideration to train a review model that is more in line with human preferences.

[0100] The specific calculation formula of the reward function is as follows: ; in, The partition function has nothing to do with the model and is only related to the dataset itself, so it does not need to be explicitly solved during training. It is a hyperparameter and can be set according to actual needs, such as setting it to 0.1.

[0101] Based on the definition of the training objectives and reward functions in the above reinforcement learning phase, it can be seen that the key to model training is and The review difference on the positive and negative review data, therefore, the preference optimization algorithm can be directly used to optimize the model to convert the above training objectives into the prediction difference value between the language model to be trained and the second language model after the previous round of training on the positive review data according to the first probability and the third probability, and obtain the first difference value, and calculate the prediction difference value between the language model to be trained and the second language model after the previous round of training on the negative review data according to the second probability and the fourth probability, and obtain the second difference value, and construct a direct preference optimization loss function with the goal of maximizing the probability value that the review data output by the language model to be trained after the previous round of training belongs to the positive review data and minimizing the probability value that the review data output by the language model to be trained after the previous round of training belongs to the negative review data, that is, the target loss function.

[0102] Among them, the specific calculation formula of the loss function of direct preference optimization can be: ; in, For sample data The corresponding first probability; For sample data The corresponding third probability; For sample data The corresponding second probability; For sample data The corresponding fourth probability; is the first difference value; is the second difference value. , The sample data are The corresponding comment data and review data in the positive review data. , The sample data are The corresponding negative review data and comment data.

[0103] The role of this loss function is to increase the model when given the question data, answer data and student answer data The probability of correct correction is increased while the probability of incorrect correction is reduced, so that based on this objective loss function, the model training of the reinforcement learning training strategy with direct preference optimization is carried out, which can enable the trained review model to better learn human preferences and accurately handle scoring tasks and comment tasks.

[0104] In some embodiments, step 110 specifically includes: Divide each of the first sample data into a plurality of sample data unit sequences, and obtain, in the first sample data set, a next sample data unit corresponding to each of the sample data unit sequences; Inputting each of the sample data unit sequences into the large language model to obtain a first log-likelihood probability that the next predicted data unit corresponding to each of the sample data unit sequences belongs to the next sample data unit; Performing unsupervised training on the large language model with the goal of maximizing the sum of first log-likelihood probabilities corresponding to all the sample data unit sequences in the first sample data set to obtain the first language model; The first log-likelihood probability is the log-likelihood probability that the next predicted data unit output by the large language model belongs to the next sample data unit.

[0105] Optionally, for the first stage of question bank pre-training, each first sample data in a large number of unlabeled first sample data sets may be divided into more detailed data units to obtain multiple sample data unit sequences; wherein each sample data unit sequence contains a series of multiple interrelated but relatively independent data units (also called tokens), and each data unit is arranged in sequence. The data unit may include at least one of a word, a phrase, a sentence, and a paragraph.

[0106] In addition, for each divided sample data unit sequence, it is also necessary to determine the corresponding next sample data unit in the first sample data set. For example, the current sample data unit sequence is { }, then the corresponding next sample data unit is ;in, Represents token, { } is the index corresponding to token.

[0107] During the pre-training process, each sample data unit sequence of each first sample data can be input into a large language model, so that the large language model can predict the log-likelihood probability that the next predicted data unit corresponding to each sample data unit sequence belongs to the next sample data unit corresponding to each sample data unit sequence, based on the given context information corresponding to each sample data unit sequence, and obtain the first log-likelihood probability corresponding to each sample data unit sequence. Then, with the goal of maximizing the sum of the first log-likelihood probabilities corresponding to all sample data unit sequences in the first sample data set, a pre-training loss function is constructed; wherein the pre-training loss function The specific calculation formula is as follows: ; in, is the log-likelihood probability function; are the model parameters of the large language model.

[0108] Then, based on the pre-training loss function, the large language model is unsupervisedly trained to optimize the model parameters of the large language model until the maximum number of iterations is reached or the model converges. The pre-training process is stopped, and the pre-trained large language model with the best model performance during the training process is used as the first language model. The first language model is a language model with knowledge in the fields of accurate learning and correction, problem solving, etc., and is familiar with the formats of question data and answer data, as well as the potential relationship between question data, answer data and answer data.

[0109] The method provided in this embodiment can optimize the parameters of a large language model during the pre-training process by carefully dividing a large amount of unlabeled sample data into multiple interrelated and independent unit sequences, and performing unsupervised training with the goal of maximizing the log-likelihood probability of the corresponding next sample data unit, so that the first language model that has been pre-trained can accurately learn domain knowledge such as grading and problem solving, and be familiar with data formats and potential relationships, so that a review model with high review quality can be quickly and accurately trained based on this in the future, thereby improving the quality of the review.

[0110] In some embodiments, step 120 specifically includes: Input each of the second sample data into the first language model to obtain a second log-likelihood probability that the score prediction data corresponding to each of the second sample data belongs to the score label corresponding to each of the second sample data and a third log-likelihood probability that the comment prediction data corresponding to each of the second sample data belongs to the comment label corresponding to each of the second sample data; Performing supervised training on the first language model with the goal of maximizing the sum of the second log-likelihood probability and the third log-likelihood probability corresponding to all second sample data in the second sample data set to obtain the second language model; Among them, the second log-likelihood probability is the log-likelihood probability that the score prediction data output by the first language model belongs to the score label corresponding to each second sample data; the third log-likelihood probability is the log-likelihood probability that the comment prediction data output by the first language model belongs to the comment label corresponding to each second sample data.

[0111] Optionally, for the second stage of supervised fine-tuning of review and scoring, high-quality review and scoring thinking chain training data, including sample question data, sample answer data and sample answer data, as well as review labels and score labels, can be used for supervised fine-tuning to enable the large language model to understand the grading task and learn how to generate grading comments and grading scores, thereby obtaining a language model capable of grading tasks of comments and scores. The specific implementation steps are as follows: Each second sample data in the second sample data set is input into the first language model obtained by the first stage training, so that the first language model performs score prediction and comment prediction based on each second sample data, so as to calculate the log-likelihood probability that the score prediction data corresponding to each second sample data belongs to the score label corresponding to each second sample data on the basis of the context information corresponding to each second sample data, and obtain the second log-likelihood probability corresponding to each second sample data, and calculate the log-likelihood probability that the comment prediction data corresponding to each second sample data belongs to the comment label corresponding to each second sample data on the basis of the context information corresponding to each second sample data, and obtain the third log-likelihood probability corresponding to each second sample data. Then, with the goal of maximizing the sum of the second log-likelihood probability and the third log-likelihood probability corresponding to all the second sample data in the second sample data set, that is, with the goal of maximizing the log-likelihood probability of comment generation and correction scores when given question data, standard answer data and student answer data, a supervised training loss function is constructed; wherein the supervised training loss function The calculation formula is as follows: ; in, is the log-likelihood probability function; are the model parameters of the first language model.

[0112] Then, based on the supervised training loss function, the first language model is unsupervisedly trained to optimize the model parameters of the first language model until the maximum number of iterations is reached or the model converges. The pre-training process is stopped, and the fine-tuned first language model with the best model performance in the training process is used as the second language model. The second language model is a language model capable of performing review and scoring tasks.

[0113] The method provided in this embodiment, during the fine-tuning training process, first inputs the second sample data set into the first language model to calculate the log-likelihood probability, and then performs supervised training with the goal of maximizing the sum of these probabilities, ultimately obtaining a second language model that achieves performance optimization in the task of correcting comments and scores, so that subsequently, a review model with high review quality can be trained quickly and accurately, thereby improving the quality of the review.

[0114] In some embodiments, the present application also provides a review method.

[0115] Figure 3 It is a schematic diagram of the process of the review method provided by the present invention; Figure 3 As shown, the method includes: step 310 and step 320.

[0116] Step 310, obtaining the data to be reviewed, wherein the data to be reviewed includes target question data, target answer data and target answer data; Step 320, inputting the data to be reviewed into the review model to obtain the score prediction data and comment prediction data corresponding to the data to be reviewed; The review model is obtained by training based on the review model training method provided in the above embodiments. Figure 1 The training process shown will not be repeated here.

[0117] Optionally, after obtaining the target question data, target response data and target answer data, the target question data, target response data and target answer data can be input into the review model, so that the review model can perform scoring task processing and comment task processing on the review data, thereby obtaining scoring prediction data and comment prediction data corresponding to the data to be reviewed.

[0118] The method provided in this embodiment uses a review model with high review quality trained with a small amount of high-quality comment labels to review test questions, which effectively reduces the time and economic costs of labeling, while improving the quality of the review model and the accuracy of the review, thereby improving the overall quality of the review.

[0119] The review model training device provided by the present invention is described below. The review model training device described below and the review model training method described above can be referenced to each other.

[0120] Figure 4 Schematic diagram of the structure of the review model training device provided by the present invention; Figure 4 As shown, the device comprises: The first training unit 410 is used to perform unsupervised training on the large language model according to the first sample data in the first sample data set to obtain a first language model; The second training unit 420 is used to perform supervised training on the first language model according to the second sample data in the second sample data set and the score labels and comment labels corresponding to each of the second sample data to obtain a second language model; The third training unit 430 is used to perform reinforcement learning training on the second language model according to the third sample data in the third sample data set and the scoring labels corresponding to the third sample data to obtain a review model; The number of sample data in the first sample data set and the number of sample data in the third sample data set are both greater than the number of sample data in the second sample data set.

[0121] The device provided in this embodiment first uses a large amount of sample data in the first sample data set to perform unsupervised training on a large language model, so as to obtain a first language model with certain basic subject knowledge and problem-solving comprehension ability through unlabeled training, and uses a small amount of sample data in the second sample data set and its corresponding scoring labels and comment labels to perform supervised training on the first language model, so as to fine-tune the model through a small amount of high-quality scoring comment label data, so as to obtain a second language model with instruction following ability and learning and mastering comment scoring and correction tasks, and uses a large amount of sample data with scoring labels in the third sample data set to perform reinforcement learning training of scoring feedback on the second language model, so as to constrain the large language model to generate high-quality comments through accurate scoring labels, and obtain a review model that can accurately perform comment generation and scoring correction capabilities, thereby reducing the dependence on high-quality comment labels while effectively training a review model with high review quality, effectively reducing the time cost and economic cost of annotation, and at the same time improving the quality of the review model and the accuracy of the review, thereby improving the overall quality of the review.

[0122] In some embodiments, the third training unit is specifically configured to: Input each of the third sample data into the second language model to obtain a review data set corresponding to each of the third sample data output by the second language model; Calculate the difference between the score prediction data of each review data in the review data set corresponding to each of the third sample data and the score label corresponding to each of the third sample data, and determine the positive review data and the negative review data corresponding to each of the third sample data; According to each of the third sample data, and the positive review data and the negative review data corresponding to each of the third sample data, reinforcement learning training is performed on the second language model to obtain the review model.

[0123] In some embodiments, the third training unit is further configured to: For the current round of training, each of the third sample data is input into the language model to be trained after the previous round of training, and a first probability that the predicted review data corresponding to each of the third sample data belongs to the positive review data and a second probability that the predicted review data belongs to the negative review data are obtained; Input each of the third sample data into the second language model to obtain a third probability that the predicted review data belongs to the positive review data and a fourth probability that the predicted review data belongs to the negative review data; Constructing a target loss function according to the first probability, the second probability, the third probability and the fourth probability; Based on the target loss function, iteratively train the language model to be trained after the previous round of training to obtain the language model to be trained after the current round of training; The review model is constructed according to the language model to be trained after each round of training.

[0124] In some embodiments, the third training unit is further configured to: Calculating a first difference value according to the first probability and the third probability; Calculating a second difference value according to the second probability and the fourth probability; Constructing the target loss function according to the first difference value and the second difference value; The first difference value is the predicted difference value between the language model to be trained after the previous round of training and the second language model on the positive review data, and the second difference value is the predicted difference value between the language model to be trained after the previous round of training and the second language model on the negative review data.

[0125] In some embodiments, the first training unit is specifically configured to: Divide each of the first sample data into a plurality of sample data unit sequences, and obtain, in the first sample data set, a next sample data unit corresponding to each of the sample data unit sequences; Inputting each of the sample data unit sequences into the large language model to obtain a first log-likelihood probability that the next predicted data unit corresponding to each of the sample data unit sequences belongs to the next sample data unit; The first language model is obtained by performing unsupervised training on the large language model with the goal of maximizing the sum of first log-likelihood probabilities corresponding to all the sample data unit sequences in the first sample data set.

[0126] In some embodiments, the second training unit is specifically configured to: Input each of the second sample data into the first language model to obtain a second log-likelihood probability that the score prediction data corresponding to each of the second sample data belongs to the score label corresponding to each of the second sample data and a third log-likelihood probability that the comment prediction data corresponding to each of the second sample data belongs to the comment label corresponding to each of the second sample data; With the goal of maximizing the sum of the second log-likelihood probability and the third log-likelihood probability corresponding to all second sample data in the second sample data set, supervised training is performed on the first language model to obtain the second language model.

[0127] The device provided by the present invention is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for the specific processes and detailed contents, which will not be repeated here.

[0128] In some embodiments, the present application also provides a review device.

[0129] Figure 5 Schematic diagram of the structure of the review device provided by the present invention; Figure 5 As shown, the device comprises: The acquisition unit 510 is used to acquire the data to be reviewed, wherein the data to be reviewed includes target question data, target answer data and target answer data; The review unit 520 is used to input the data to be reviewed into the review model to obtain the score prediction data and comment prediction data corresponding to the data to be reviewed; The review model is obtained by training based on the review model training method provided in the above embodiments. Figure 1 The training process shown will not be repeated here.

[0130] The device provided in this embodiment reviews test questions by using a review model with high review quality trained with a small amount of high-quality comment labels, which effectively reduces the time and economic costs of labeling, while improving the quality of the review model and the accuracy of the review, thereby improving the overall quality of the review.

[0131] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6As shown, the electronic device may include: a processor (processor) 610 , a communication interface (Communications Interface) 620 , a memory (memory) 630 and a communication bus 640 , wherein the processor 610 , the communication interface 620 , and the memory 630 communicate with each other through the communication bus 640 . The processor 610 may call the logic instructions in the memory 630 to execute the review model training method, which includes: performing unsupervised training on a large language model according to the first sample data in the first sample data set to obtain a first language model; performing supervised training on the first language model according to the second sample data in the second sample data set and the score labels and comment labels corresponding to each of the second sample data to obtain a second language model; performing reinforcement learning training on the second language model according to the third sample data in the third sample data set and the score labels corresponding to each of the third sample data to obtain a review model; wherein the number of sample data in the first sample data set and the number of sample data in the third sample data set are both greater than the number of sample data in the second sample data set; or, executing a review method, which includes: obtaining data to be reviewed, wherein the data to be reviewed includes target question data, target answer data and target answer data; inputting the data to be reviewed into the review model to obtain score prediction data and comment prediction data corresponding to the data to be reviewed; wherein the review model is trained based on any one of the review model training methods provided in the above embodiments.

[0132] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0133] On the other hand, the present invention also provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the review model training method provided by the above methods, the method comprising: according to the first sample data in the first sample data set, performing unsupervised training on a large language model to obtain a first language model; according to the second sample data in the second sample data set and the scoring labels and comment labels corresponding to each of the second sample data, performing supervised training on the first language model to obtain a second language model; according to the third sample data in the third sample data set and the scoring labels and comment labels corresponding to each of the second sample data, performing supervised training on the first language model to obtain a second language model. The scoring label corresponding to the third sample data is used to perform reinforcement learning training on the second language model to obtain a review model; wherein the number of sample data in the first sample data set and the number of sample data in the third sample data set are both greater than the number of sample data in the second sample data set; or, a review method is executed, which includes: obtaining data to be reviewed, wherein the data to be reviewed includes target question data, target response data and target answer data; inputting the data to be reviewed into the review model to obtain score prediction data and comment prediction data corresponding to the data to be reviewed; wherein the review model is trained based on any one of the review model training methods provided in the above embodiments.

[0134] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the review model training method provided by the above methods, the method comprising: performing unsupervised training on a large language model according to first sample data in a first sample data set to obtain a first language model; performing supervised training on the first language model according to second sample data in a second sample data set and the scoring labels and comment labels corresponding to each of the second sample data to obtain a second language model; performing supervised training on the first language model according to third sample data in a third sample data set and the scoring labels corresponding to each of the third sample data to obtain a second language model; , performing reinforcement learning training on the second language model to obtain a review model; wherein the number of sample data in the first sample data set and the number of sample data in the third sample data set are both greater than the number of sample data in the second sample data set; or, executing a review method, the method comprising: obtaining data to be reviewed, the data to be reviewed comprising target question data, target response data and target answer data; inputting the data to be reviewed into the review model to obtain score prediction data and comment prediction data corresponding to the data to be reviewed; wherein the review model is trained based on any one of the review model training methods provided in the above embodiments.

[0135] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0136] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A review model training method, characterized in that: include: Performing unsupervised training on a large language model according to first sample data in a first sample data set to obtain a first language model; According to the second sample data in the second sample data set and the score labels and comment labels corresponding to each of the second sample data, the first language model is subjected to supervised training to obtain a second language model; According to the third sample data in the third sample data set and the scoring labels corresponding to the third sample data, the second language model is subjected to reinforcement learning training to obtain a review model; The number of sample data in the first sample data set and the number of sample data in the third sample data set are both greater than the number of sample data in the second sample data set.

2. The review model training method according to claim 1, characterized in that: The step of performing reinforcement learning training on the second language model according to the third sample data in the third sample data set and the scoring labels corresponding to the third sample data to obtain the review model includes: Input each of the third sample data into the second language model to obtain a review data set corresponding to each of the third sample data output by the second language model; Calculate the difference between the score prediction data of each review data in the review data set corresponding to each of the third sample data and the score label corresponding to each of the third sample data, and determine the positive review data and the negative review data corresponding to each of the third sample data; According to each of the third sample data, and the positive review data and the negative review data corresponding to each of the third sample data, reinforcement learning training is performed on the second language model to obtain the review model.

3. The review model training method according to claim 2, characterized in that: The step of performing reinforcement learning training on the second language model according to each of the third sample data and the positive review data and the negative review data corresponding to each of the third sample data to obtain the review model includes: For the current round of training, each of the third sample data is input into the language model to be trained after the previous round of training, and a first probability that the predicted review data corresponding to each of the third sample data belongs to the positive review data and a second probability that the predicted review data belongs to the negative review data are obtained; Input each of the third sample data into the second language model to obtain a third probability that the predicted review data belongs to the positive review data and a fourth probability that the predicted review data belongs to the negative review data; Constructing a target loss function according to the first probability, the second probability, the third probability and the fourth probability; Based on the target loss function, iteratively train the language model to be trained after the previous round of training to obtain the language model to be trained after the current round of training; The review model is constructed according to the language model to be trained after each round of training.

4. The review model training method according to claim 3, characterized in that: The constructing a target loss function according to the first probability, the second probability, the third probability and the fourth probability includes: Calculating a first difference value according to the first probability and the third probability; Calculating a second difference value according to the second probability and the fourth probability; Constructing the target loss function according to the first difference value and the second difference value; The first difference value is the predicted difference value between the language model to be trained after the previous round of training and the second language model on the positive review data, and the second difference value is the predicted difference value between the language model to be trained after the previous round of training and the second language model on the negative review data.

5. The review model training method according to any one of claims 1 to 4, characterized in that: The step of performing unsupervised training on a large language model according to first sample data in the first sample data set to obtain a first language model includes: Divide each of the first sample data into a plurality of sample data unit sequences, and obtain, in the first sample data set, a next sample data unit corresponding to each of the sample data unit sequences; Inputting each of the sample data unit sequences into the large language model to obtain a first log-likelihood probability that the next predicted data unit corresponding to each of the sample data unit sequences belongs to the next sample data unit; The first language model is obtained by performing unsupervised training on the large language model with the goal of maximizing the sum of first log-likelihood probabilities corresponding to all the sample data unit sequences in the first sample data set.

6. The review model training method according to any one of claims 1 to 4, characterized in that: The step of performing supervised training on the first language model according to the second sample data in the second sample data set and the score labels and comment labels corresponding to each of the second sample data to obtain the second language model comprises: Input each of the second sample data into the first language model to obtain a second log-likelihood probability that the score prediction data corresponding to each of the second sample data belongs to the score label corresponding to each of the second sample data and a third log-likelihood probability that the comment prediction data corresponding to each of the second sample data belongs to the comment label corresponding to each of the second sample data; With the goal of maximizing the sum of the second log-likelihood probability and the third log-likelihood probability corresponding to all second sample data in the second sample data set, supervised training is performed on the first language model to obtain the second language model.

7. The review model training method according to any one of claims 1 to 4, characterized in that: Each of the first sample data, the second sample data and the third sample data includes sample question data, sample answer data and sample answer data.

8. A review method, characterized in that: include: Acquire the data to be reviewed, wherein the data to be reviewed includes target question data, target answer data and target answer data; Inputting the data to be reviewed into the review model to obtain the score prediction data and comment prediction data corresponding to the data to be reviewed; Wherein, the review model is trained based on the review model training method according to any one of claims 1 to 7.

9. A review model training device, characterized in that: include: A first training unit is used to perform unsupervised training on the large language model according to first sample data in the first sample data set to obtain a first language model; A second training unit is used to perform supervised training on the first language model according to the second sample data in the second sample data set and the score labels and comment labels corresponding to each of the second sample data, so as to obtain a second language model; A third training unit is used to perform reinforcement learning training on the second language model according to the third sample data in the third sample data set and the scoring labels corresponding to the third sample data to obtain a review model; The number of sample data in the first sample data set and the number of sample data in the third sample data set are both greater than the number of sample data in the second sample data set.

10. A review device, characterized in that: include: An acquisition unit, used for acquiring data to be reviewed, wherein the data to be reviewed includes target question data, target answer data and target answer data; A review unit, used for inputting the data to be reviewed into a review model to obtain score prediction data and comment prediction data corresponding to the data to be reviewed; Wherein, the review model is trained based on the review model training method according to any one of claims 1 to 7.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the review model training method as described in any one of claims 1 to 7, or the review method as described in claim 8 is implemented.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the review model training method according to any one of claims 1 to 7 or the review method according to claim 8 is implemented.

Images