Training and use method, device, storage medium and equipment for comment generation model
By clustering and deduplication of real comments, combining the sorting learning of the reward model and the dual threshold setting, the problem of diversity and labeling difficulty of comment generation models when the data volume is large, and the diversity and model accuracy of comment generation are improved. sex.
Patent Information
- Application Number
- CN202410507369.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-25
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2044-04-25
AI Technical Summary
When the training sample data is large, the diversity of the generated comments is affected, and the labeling is difficult, which affects the accuracy of the model.
By clustering and deduplication operations on real comments, a training data set is generated, and the reward model is used for sorting and binary classification learning, and a double threshold is set to expand the comment score gap and reduce the difficulty of labeling.
It improves the diversity of generated comments, reduces the difficulty of labeling training data, enhances the accuracy of the reward model, and realizes detailed cutting of comments.
Smart Images
Figure CN118350481B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a method, device, storage medium and equipment for training and using a review generation model. Background Art
[0002] Existing text generation technologies and natural language processing algorithms have made great progress, and these technologies can be used to automatically generate relevant comments based on text and prompt words.
[0003] In the related art, LLM (Large Language Models) models can be used to generate comments. Before using the LLM model, a large number of training samples are needed to automatically learn how to predict the probability distribution of the next word, so that corresponding comments can be automatically generated based on the text and the prompt word.
[0004] However, comments under the same text usually contain a large number of identical or similar comments. If all these comments are used as training samples, it will not only increase the amount of training sample data, but also cause the generated comments to be biased towards comments with a large number, thus affecting the diversity of the comments. Summary of the invention
[0005] The present application provides a method, apparatus, storage medium and device for training and using a review generation model, which is used to solve the problem that the amount of training sample data is large and affects the diversity of generated reviews. The technical solution is as follows:
[0006] According to a first aspect of the present application, a method for training a review generation model is provided, wherein the review generation model includes a generation model and a reward model, wherein the generation model is used to generate a plurality of predicted reviews, and the reward model is used to score the plurality of predicted reviews, so as to output the plurality of predicted reviews after being sorted according to the scores, and the method includes:
[0007] Acquire a training data set, and train the generation model according to the training data set, wherein each set of training data in the training data set includes a prompt word and a plurality of real comments obtained after clustering and deduplication operations, and the prompt word is generated according to a preset prompt word template and text corresponding to the real comment;
[0008] Using the generation model to generate multiple predicted comments for the acquired prompt words, marking each predicted comment as a superior comment or an inferior comment, and generating multiple groups of comparison data based on the multiple predicted comments, each group of comparison data including one superior comment and one inferior comment;
[0009] The reward model is trained according to each group of comparison data. The loss function of the reward model is used to rank the superior comments and inferior comments in each group of comparison data, and the scores of the superior comments and the inferior comments are binary classified based on a double threshold to increase the gap between the superior comments and the inferior comments. The loss is calculated according to the binary classification result, and the reward model is updated according to the loss calculation.
[0010] In a possible implementation, the training of the reward model according to each set of comparison data includes:
[0011] For each set of comparison data, using the reward model to generate a first score for the superior comments in the comparison data, and a second score for the inferior comments in the comparison data;
[0012] Calculating the difference between the first score and a first threshold using the loss function to obtain a first difference, where the first threshold is positive;
[0013] Calculating the difference between the second score and a second threshold using the loss function to obtain a second difference, where the second threshold is negative;
[0014] The loss function is used to perform loss calculation according to the first difference and the second difference, and the reward model is updated according to the loss calculation.
[0015] In a possible implementation, obtaining a training data set includes:
[0016] Obtain multiple texts and multiple real comments corresponding to each text, and cluster and remove duplicates of the multiple real comments corresponding to each text;
[0017] Acquire a preset prompt word template library, wherein the prompt word template library includes at least one prompt word template;
[0018] The text and a prompt word template selected from the prompt word template library are combined into a prompt word, and the prompt word and a plurality of screened real comments are combined into a set of training data.
[0019] In a possible implementation, the text and a prompt word template selected from the prompt word template library are combined into a prompt word, and the prompt word and a plurality of screened real comments are combined into a set of training data, including:
[0020] For each text, splitting multiple real comments corresponding to the text into multiple comment groups;
[0021] For each comment group, a prompt word template is selected from the prompt word template library, the prompt word template and the text are combined into a prompt word, and the prompt word and the comment group are combined into a set of training data.
[0022] According to a second aspect of the present application, a method for using a review generation model is provided, the method comprising:
[0023] Acquire a target text, and generate a prompt word according to the target text and a preset prompt word template;
[0024] Using a generation model in a comment generation model to generate a plurality of predicted comments for the prompt word, and performing clustering and deduplication operations on the plurality of predicted comments, wherein the comment generation model is obtained by training using the training method described above;
[0025] Scoring the screened multiple predicted reviews using the reward model in the review generation model;
[0026] The plurality of predicted comments are sorted according to the scores and then outputted.
[0027] In a possible implementation, outputting the plurality of predicted comments after sorting them according to their scores includes:
[0028] Deleting predicted comments whose scores are less than a preset threshold from the plurality of predicted comments;
[0029] Sort the remaining multiple predicted reviews in descending order of ratings;
[0030] Output the sorted predicted comments.
[0031] According to a third aspect of the present application, a training device for a review generation model is provided, wherein the review generation model includes a generation model and a reward model, wherein the generation model is used to generate a plurality of predicted reviews, and the reward model is used to score the plurality of predicted reviews, so as to output the plurality of predicted reviews after being sorted according to the scores, and the device includes:
[0032] A first training module is used to obtain a training data set, and train the generation model according to the training data set, wherein each group of training data in the training data set includes a prompt word and a plurality of real comments obtained after clustering and deduplication operations, and the prompt word is generated according to a preset prompt word template and the text corresponding to the real comment;
[0033] A generation module, used to generate multiple predicted comments for the acquired prompt words using the generation model, mark each predicted comment as a superior comment or an inferior comment, and generate multiple groups of comparison data based on the multiple predicted comments, each group of comparison data including one superior comment and one inferior comment;
[0034] The second training module is used to train the reward model according to each group of comparison data. The loss function of the reward model is used to rank the superior comments and inferior comments in each group of comparison data, and to classify the scores of the superior comments and the inferior comments based on a double threshold value to increase the gap between the superior comments and the inferior comments. The loss is calculated according to the binary classification result, and the reward model is updated according to the loss calculation.
[0035] According to a fourth aspect of the present application, a method for using a review generation model is provided, the method comprising:
[0036] An acquisition module, used for acquiring a target text, and generating a prompt word according to the target text and a preset prompt word template;
[0037] A generation module, used to generate multiple predicted comments for the prompt word using a generation model in a comment generation model, and to perform clustering and deduplication operations on the multiple predicted comments, wherein the comment generation model is trained using the training method described above;
[0038] A scoring module, used to score the screened multiple predicted reviews using the reward model in the review generation model;
[0039] The output module is used to output the plurality of predicted comments after sorting them according to the scores.
[0040] According to a fifth aspect of the present application, a computer-readable storage medium is provided, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement the training method of the comment generation model as described above; or, the at least one instruction is loaded and executed by a processor to implement the method of using the comment generation model as described above.
[0041] According to a sixth aspect of the present application, a computer device is provided, which includes a training device for the above-mentioned review generation model or a method for using the above-mentioned review generation model.
[0042] The beneficial effects of the technical solution provided by this application include at least:
[0043] By clustering and deduplicating real reviews, a training data set is generated based on the prompt word template, text and filtered real reviews. The training data set is used to train the generative model. In this way, when the generative model generates predicted reviews in small quantities and multiple times, the content similarity of the generated predicted reviews will not be too high, thereby improving the diversity of the reviews.
[0044] By adjusting the loss function of the reward model so that it can perform ranking learning and binary classification learning, we only need to label each predicted comment as an excellent comment or a poor comment (binary classification) to generate comparison data containing one excellent comment and one poor comment. Then, we use the reward model to perform ranking learning and binary classification learning on the two good and poor comments. There is no need to perform difficult labeling and then sorting of multiple predicted comments. Learning based on the ranked data reduces the difficulty of labeling the training data and improves the accuracy of the reward model.
[0045] By setting double thresholds for the loss function, with the first threshold being positive and the second threshold being negative, the upper and lower ranges of the score are significantly expanded, which can increase the gap between excellent reviews and inferior reviews. When using the reward model, the threshold can be selected according to actual conditions to achieve a more detailed segmentation of the reviews. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0047] Figure 1 is a flowchart of a method for training a review generation model provided by an embodiment of the present application;
[0048] Figure 2 is a flowchart of a method for training a review generation model provided by an embodiment of the present application;
[0049] Figure 3 is a flow chart of a method for using a review generation model provided by an embodiment of the present application;
[0050] Figure 4 It is a structural block diagram of a training device for a review generation model provided by an embodiment of the present application;
[0051] Figure 5 It is a structural block diagram of a device for using a review generation model provided by an embodiment of the present application. DETAILED DESCRIPTION
[0052] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.
[0053] This embodiment involves a comment generation model, which includes a generation model and a reward model. The generation model is used to generate multiple predicted comments, and the reward model is used to score the multiple predicted comments, so that the multiple predicted comments are sorted according to the scores and then output.
[0054] In this embodiment, the generation model can be a model trained based on SFT (Supervised Fine-Tuning), and the reward model can be a model trained based on RLHF (Reinforcement Learning from Human Feedback). Of course, the above two models can also be trained based on other methods, which is not limited in this embodiment.
[0055] like Figure 1 As shown, it shows a method flow chart of a method for training a comment generation model provided by an embodiment of the present application, and the training method for the comment generation model can be applied to a computer device. The training method for the comment generation model may include:
[0056] Step 101, obtain a training data set, and train a generation model based on the training data set, wherein each group of training data in the training data set includes prompt words and multiple real comments obtained after clustering and deduplication operations, and the prompt words are generated based on a preset prompt word template and the text corresponding to the real comments.
[0057] In order to ensure the diversity of prompt word styles, we can preset a prompt word template library containing multiple prompt word templates according to human language habits. For example, one prompt word template is: write a comment based on the background information XXXX; another prompt word template is: please write a comment based on the XXXX article.
[0058] For the sake of distinction, in this embodiment, the comments actually obtained are referred to as real comments, and the comments generated by the generation model are referred to as predicted comments.
[0059] Computer equipment can obtain a large amount of text and all the real comments corresponding to each text. Real comments with similar attitudes under the same or the same type of text have high similarity. For example, there may be a large number of real comments expressing "too beautiful", "so beautiful", "really pretty" and so on under the text introducing the scenic spot. If all real comments are directly used to train the generative model, the generative model will be particularly biased towards generating comments on the common parts of the text, while we hope that the generative model can comment on the different parts of different content. To solve this problem, we introduced a similarity monitoring mechanism, that is, clustering and deduplication operations on real comments, removing overly similar real comments, to ensure that the trained generative model can generate diverse predicted comments.
[0060] The computer device can generate a prompt word according to the prompt word template and each text, and then form training data with each prompt word and the real comments filtered by clustering and deduplication operations to obtain a training data set. Then, the computer device can train the generation model based on the SFT and the training data set.
[0061] Step 102, using the generative model to generate multiple predicted comments for the acquired prompt words, marking each predicted comment as a superior comment or an inferior comment, and generating multiple groups of comparison data based on the multiple predicted comments, each group of comparison data including one superior comment and one inferior comment.
[0062] The computer device can obtain multiple texts, generate a prompt word from each text and a prompt word template, input each prompt word into the generation model, and the generation model can generate multiple predicted comments for each prompt word obtained. The text mentioned here can be text in the training data or other text, which is not limited in this embodiment.
[0063] For multiple predicted comments generated for each text, each predicted comment can be manually labeled as a good comment or a bad comment, and then each good comment and a bad comment are combined into a group of comparison data to obtain multiple groups of comparison data.
[0064] Step 103, training the reward model according to each set of comparison data, the loss function of the reward model is used to rank the superior comments and inferior comments in each set of comparison data, and to classify the scores of the superior comments and inferior comments based on the double threshold to increase the gap between the superior comments and inferior comments, calculate the loss according to the binary classification results, and update the reward model according to the loss calculation.
[0065] In this embodiment, we adjust the loss function of the reward model to enable it to perform ranking learning and binary classification learning. The computer device can train the reward model based on the RLHF and the comparison data. In short, we add a double threshold to the loss function to make the loss function more like a binary classification task, so that the reward model can calculate the loss of two good and bad reviews at the same time, and help the reward model learn the ranking of different reviews based on the binary classification.
[0066] In summary, the training method of the comment generation model provided in the embodiment of the present application performs clustering and deduplication operations on real comments, generates a training data set based on prompt word templates, texts and screened real comments, and uses the training data set to train the generation model. In this way, when the generation model generates predicted comments in small quantities and multiple times, the content similarity of the generated predicted comments will not be too high, thereby improving the diversity of the comments.
[0067] By adjusting the loss function of the reward model so that it can perform ranking learning and binary classification learning, we only need to label each predicted comment as an excellent comment or a poor comment (binary classification) to generate comparison data containing one excellent comment and one poor comment. Then, we use the reward model to perform ranking learning and binary classification learning on the two good and poor comments. There is no need to perform difficult labeling and then sorting of multiple predicted comments. Learning based on the ranked data reduces the difficulty of labeling the training data and improves the accuracy of the reward model.
[0068] By setting double thresholds for the loss function, with the first threshold being positive and the second threshold being negative, the upper and lower ranges of the score are significantly expanded, which can increase the gap between excellent reviews and inferior reviews. When using the reward model, the threshold can be selected according to actual conditions to achieve a more detailed segmentation of the reviews.
[0069] like Figure 2 As shown, it shows a method flow chart of a method for training a comment generation model provided by an embodiment of the present application, and the training method for the comment generation model can be applied to a computer device. The training method for the comment generation model may include:
[0070] Step 201, obtain multiple texts and multiple real comments corresponding to each text, and cluster and remove duplicates for the multiple real comments corresponding to each text.
[0071] For the sake of distinction, in this embodiment, the comments actually obtained are referred to as real comments, and the comments generated by the generation model are referred to as predicted comments.
[0072] Computer equipment can obtain a large amount of text and all the real comments corresponding to each text. Real comments with similar attitudes under the same or the same type of text have high similarity. For example, there may be a large number of real comments expressing "too beautiful", "so beautiful", "really pretty" and so on under the text introducing the scenic spot. If all real comments are directly used to train the generative model, the generative model will be particularly biased towards generating comments on the common parts of the text, while we hope that the generative model can comment on the different parts of different content. To solve this problem, we introduced a similarity monitoring mechanism, that is, clustering and deduplication operations on real comments, removing overly similar real comments, to ensure that the trained generative model can generate diverse predicted comments.
[0073] Step 202: Obtain a preset prompt word template library, wherein the prompt word template library includes at least one prompt word template.
[0074] In order to ensure the diversity of prompt word styles, we can preset a prompt word template library containing multiple prompt word templates according to human language habits. For example, one prompt word template is: write a comment based on the background information XXXX; another prompt word template is: please write a comment based on the XXXX article.
[0075] Step 203: Combine the text and a prompt word template selected from a prompt word template library to form a prompt word, and combine the prompt word and the screened multiple real comments to form a set of training data.
[0076] The computer device can generate a prompt word according to the prompt word template and each text, and then form training data with each prompt word and real comments that have been screened by clustering and deduplication operations, and finally obtain a training data set.
[0077] When a text corresponds to a large number of real comments, the computer device can divide the real comments into at least two comment groups, and generate a set of training data for each comment group together with the text and the prompt words to reduce the data volume of a set of training data.
[0078] Specifically, the text and a prompt word template selected from a prompt word template library are combined into a prompt word, and the prompt word and a plurality of screened real comments are combined into a set of training data, which may include: for each text, splitting the plurality of real comments corresponding to the text into a plurality of comment groups; for each comment group, selecting a prompt word template from the prompt word template library, combining the prompt word template and the text into a prompt word, and combining the prompt word and the comment group into a set of training data.
[0079] For example, a text corresponds to 20 real comments, and the prompt word template library includes 4 prompt word templates. Then, prompt word 1 can be generated according to prompt word template 1 and the text, and prompt word 1 and the first 10 real comments form a group of training data; prompt word 2 can be generated according to prompt word template 2 and the text, and prompt word 2 and the last 10 real comments form another group of training data.
[0080] Step 204: train and generate a model based on the training data set.
[0081] The computer device can train the generative model based on the SFT and the training data set.
[0082] Step 205, using the generation model to generate multiple predicted comments for the acquired prompt words, marking each predicted comment as a superior comment or an inferior comment, and generating multiple groups of comparison data based on the multiple predicted comments, each group of comparison data including one superior comment and one inferior comment.
[0083] The computer device can obtain multiple texts, generate a prompt word from each text and a prompt word template, input each prompt word into the generation model, and the generation model can generate multiple predicted comments for each prompt word obtained. The text mentioned here can be text in the training data or other text, which is not limited in this embodiment.
[0084] For multiple predicted comments generated for each text, each predicted comment can be manually labeled as a good comment or a bad comment, and then each good comment and a bad comment are combined into a group of comparison data to obtain multiple groups of comparison data.
[0085] Step 206, for each set of comparison data, use the reward model to generate a first score for the superior comments in the comparison data, and generate a second score for the inferior comments in the comparison data; use the loss function to calculate the difference between the first score and the first threshold to obtain a first difference, and the first threshold is positive; use the loss function to calculate the difference between the second score and the second threshold to obtain a second difference, and the second threshold is negative; use the loss function to calculate the loss based on the first difference and the second difference, and update the reward model based on the loss calculation.
[0086] Among them, the loss function of the reward model is used to rank the superior and inferior comments in each group of comparison data, and to classify the scores of the superior and inferior comments based on the double threshold to increase the gap between the superior and inferior comments. The loss is calculated according to the binary classification results, and the reward model is updated according to the loss calculation.
[0087] In this embodiment, we adjust the loss function of the reward model to enable it to perform ranking learning and binary classification learning. The computer device can train the reward model based on the RLHF and the comparison data. In short, we add a double threshold to the loss function to make the loss function more like a binary classification task, so that the reward model can calculate the loss of two good and bad reviews at the same time, and help the reward model learn the ranking of different reviews based on the binary classification.
[0088] The first threshold and the second threshold can be set according to actual needs and are not limited in this embodiment. In one example, assuming that the first threshold is 3, the second threshold is -3, the first score is a, and the second score is b, then the first difference = a-3, and the second difference = -3-b.
[0089] After obtaining the first difference and the second difference, the computer device calculates the cross entropy of the first difference and the second difference, and updates the reward model according to the cross entropy.
[0090] In summary, the training method of the comment generation model provided in the embodiment of the present application performs clustering and deduplication operations on real comments, generates a training data set based on prompt word templates, texts and screened real comments, and uses the training data set to train the generation model. In this way, when the generation model generates predicted comments in small quantities and multiple times, the content similarity of the generated predicted comments will not be too high, thereby improving the diversity of the comments.
[0091] By adjusting the loss function of the reward model so that it can perform ranking learning and binary classification learning, we only need to label each predicted comment as an excellent comment or a poor comment (binary classification) to generate comparison data containing one excellent comment and one poor comment. Then, we use the reward model to perform ranking learning and binary classification learning on the two good and poor comments. There is no need to perform difficult labeling and then sorting of multiple predicted comments. Learning based on the ranked data reduces the difficulty of labeling the training data and improves the accuracy of the reward model.
[0092] By setting double thresholds for the loss function, with the first threshold being positive and the second threshold being negative, the upper and lower ranges of the score are significantly expanded, which can increase the gap between excellent reviews and inferior reviews. When using the reward model, the threshold can be selected according to actual conditions to achieve a more detailed segmentation of the reviews.
[0093] like Figure 3 As shown, it shows a flowchart of a method for using a review generation model provided by an embodiment of the present application, and the method for using the review generation model can be applied to a computer device. The method for using the review generation model may include:
[0094] Step 301, obtaining a target text, and generating prompt words according to the target text and a preset prompt word template.
[0095] The computer device can obtain the text for which a comment needs to be generated as the target text, and then select a prompt word template from the prompt word template library, and combine the prompt word template and the target text into a prompt word.
[0096] Step 302: Generate multiple predicted comments for the prompt word using the generation model in the comment generation model, and perform clustering and deduplication operations on the multiple predicted comments.
[0097] Among them, the comment generation model is trained using the above-mentioned training method.
[0098] Since the number of predicted comments generated by the generative model at a time is limited, if you need to obtain a lot of predicted comments, you need to use the generative model multiple times to generate predicted comments. Suppose you need to generate 300 predicted comments for a text, and the generative model can only generate 10 predicted comments each time, then you need to use the generative model 30 times to generate 300 predicted comments.
[0099] The process of generating multiple predicted comments is independent of each other, so the predicted comments generated multiple times may have the same or similar problems. To solve this problem, we introduced a similarity monitoring mechanism, that is, clustering and deduplication operations on the generated predicted comments, removing overly similar predicted comments, and ensuring the diversity of the output predicted comments.
[0100] Step 303: score the screened multiple predicted reviews using the reward model in the review generation model.
[0101] The computer device inputs the generated predicted comments into the reward model so that the reward model generates a score for each predicted comment.
[0102] Step 304, sorting the plurality of predicted comments according to the scores and outputting them.
[0103] The computer device may sort all the predicted reviews in descending order of scores and output the sorted predicted reviews. Or,
[0104] The computer device may first delete irrelevant predicted comments with low scores, and then sort the remaining predicted comments in descending order of scores, and output the sorted predicted comments. Specifically, outputting the plurality of predicted comments after sorting them in order of scores may include: deleting predicted comments with scores less than a preset threshold from the plurality of predicted comments; sorting the remaining plurality of predicted comments in descending order of scores; and outputting the sorted plurality of predicted comments. The preset threshold may be set according to actual needs and is not limited in this embodiment.
[0105] In summary, the method for using the comment generation model provided in the embodiment of the present application performs clustering and deduplication operations on predicted comments, so that the content similarity of the generated predicted comments is not too high, thereby improving the diversity of the comments.
[0106] like Figure 4 As shown, it shows a structural block diagram of a training device for a review generation model provided by an embodiment of the present application, and the training device for the review generation model can be applied to a computer device. The training device for the review generation model may include:
[0107] A first training module 410 is used to obtain a training data set and train a model based on the training data set, wherein each set of training data in the training data set includes a prompt word and a plurality of real comments obtained after clustering and deduplication operations, and the prompt word is generated based on a preset prompt word template and text corresponding to the real comment;
[0108] A generation module 420 is used to generate multiple predicted comments for the acquired prompt words using a generation model, mark each predicted comment as a good comment or a bad comment, and generate multiple groups of comparison data based on the multiple predicted comments, each group of comparison data includes a good comment and a bad comment;
[0109] The second training module 430 is used to train the reward model according to each group of comparison data. The loss function of the reward model is used to rank the superior comments and inferior comments in each group of comparison data, and to classify the scores of the superior comments and inferior comments based on a double threshold to increase the gap between the superior comments and the inferior comments. The loss is calculated based on the binary classification results, and the reward model is updated based on the loss calculation.
[0110] In an optional embodiment, the second training module 430 is further configured to:
[0111] For each set of comparison data, the reward model is used to generate a first score for the superior comments in the comparison data, and a second score for the inferior comments in the comparison data;
[0112] The loss function is used to calculate the difference between the first score and the first threshold value to obtain a first difference value, where the first threshold value is positive;
[0113] The loss function is used to calculate the difference between the second score and the second threshold value to obtain a second difference value, where the second threshold value is negative;
[0114] The loss function is used to calculate the loss according to the first difference and the second difference, and the reward model is updated according to the loss calculation.
[0115] In an optional embodiment, the first training module 410 is further configured to:
[0116] Obtain multiple texts and multiple real comments corresponding to each text, and cluster and remove duplicates of the multiple real comments corresponding to each text;
[0117] Acquire a preset prompt word template library, wherein the prompt word template library includes at least one prompt word template;
[0118] The text and the prompt word template selected from the prompt word template library are combined into prompt words, and the prompt words and the screened multiple real comments are combined into a set of training data.
[0119] In an optional embodiment, the first training module 410 is further configured to:
[0120] For each text, split the multiple real comments corresponding to the text into multiple comment groups;
[0121] For each comment group, a prompt word template is selected from the prompt word template library, the prompt word template and the text are combined into a prompt word, and the prompt word and the comment group are combined into a set of training data.
[0122] In summary, the training device for the comment generation model provided in the embodiment of the present application performs clustering and deduplication operations on real comments, generates a training data set based on prompt word templates, texts and screened real comments, and uses the training data set to train the generation model. In this way, when the generation model generates predicted comments in small quantities and multiple times, the content similarity of the generated predicted comments will not be too high, thereby improving the diversity of the comments.
[0123] By adjusting the loss function of the reward model so that it can perform ranking learning and binary classification learning, we only need to label each predicted comment as an excellent comment or a poor comment (binary classification) to generate comparison data containing one excellent comment and one poor comment. Then, we use the reward model to perform ranking learning and binary classification learning on the two good and poor comments. There is no need to perform difficult labeling and then sorting of multiple predicted comments. Learning based on the ranked data reduces the difficulty of labeling the training data and improves the accuracy of the reward model.
[0124] By setting double thresholds for the loss function, with the first threshold being positive and the second threshold being negative, the upper and lower ranges of the score are significantly expanded, which can increase the gap between excellent reviews and inferior reviews. When using the reward model, the threshold can be selected according to actual conditions to achieve a more detailed segmentation of the reviews.
[0125] like Figure 5 As shown, it shows a structural block diagram of a device for using a review generation model provided by an embodiment of the present application, and the device for using the review generation model can be applied to a computer device. The device for using the review generation model may include:
[0126] An acquisition module 510 is used to acquire a target text and generate a prompt word according to the target text and a preset prompt word template;
[0127] A generation module 520 is used to generate multiple predicted comments for the prompt word using a generation model in the comment generation model, and to perform clustering and deduplication operations on the multiple predicted comments. The comment generation model is trained using the above training method;
[0128] A scoring module 530 is used to score the filtered multiple predicted reviews using a reward model in the review generation model;
[0129] The output module 540 is used to output the plurality of predicted comments after sorting them according to the scores.
[0130] In an optional embodiment, the output module 540 is further configured to:
[0131] Deleting predicted reviews whose ratings are less than a preset threshold from the plurality of predicted reviews;
[0132] Sort the remaining multiple predicted reviews in descending order of ratings;
[0133] Output the sorted predicted comments.
[0134] In summary, the device for using the comment generation model provided in the embodiment of the present application performs clustering and deduplication operations on predicted comments, so that the content similarity of the generated predicted comments is not too high, thereby improving the diversity of the comments.
[0135] An embodiment of the present application provides a computer-readable storage medium, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the training method of the comment generation model as described above; or, the at least one instruction is loaded and executed by a processor to implement the method of using the comment generation model as described above.
[0136] An embodiment of the present application provides a computer device, which includes a training device for the above-mentioned review generation model or a method for using the above-mentioned review generation model.
[0137] It should be noted that: the training and use device of the review generation model provided in the above embodiment only uses the division of the above functional modules as an example when training and using the review generation model. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the training and use device of the review generation model is divided into different functional modules to complete all or part of the functions described above. In addition, the training and use device of the review generation model provided in the above embodiment and the training and use method embodiment of the review generation model belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.
[0138] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0139] The above description is not intended to limit the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.
Claims
1. A method for training a review generation model, characterized in that: The comment generation model includes a generation model and a reward model, the generation model is used to generate multiple predicted comments, and the reward model is used to score the multiple predicted comments, so that the multiple predicted comments are sorted according to the scores and then outputted. The method includes: Acquire a training data set, and train the generation model according to the training data set, wherein each set of training data in the training data set includes a prompt word and a plurality of real comments obtained after clustering and deduplication operations, and the prompt word is generated according to a preset prompt word template and text corresponding to the real comment; Using the generation model to generate multiple predicted comments for the acquired prompt words, marking each predicted comment as a superior comment or an inferior comment, and generating multiple groups of comparison data based on the multiple predicted comments, each group of comparison data including one superior comment and one inferior comment; The reward model is trained according to each set of comparison data, and the loss function of the reward model is used to rank the superior comments and inferior comments in each set of comparison data, and the scores of the superior comments and the inferior comments are classified into two categories based on a double threshold value to increase the gap between the superior comments and the inferior comments, and the loss is calculated according to the binary classification result, and the reward model is updated according to the loss calculation; The training of the reward model according to each group of comparison data includes: for each group of comparison data, using the reward model to generate a first score for the superior comments in the comparison data, and generating a second score for the inferior comments in the comparison data; using the loss function to calculate the difference between the first score and a first threshold to obtain a first difference, and the first threshold is positive; using the loss function to calculate the difference between the second score and a second threshold to obtain a second difference, and the second threshold is negative; using the loss function to calculate the cross entropy of the first difference and the second difference, and updating the reward model according to the cross entropy.
2. The method for training a review generation model according to claim 1, characterized in that: The step of obtaining a training data set includes: Obtain multiple texts and multiple real comments corresponding to each text, and cluster and remove duplicates of the multiple real comments corresponding to each text; Acquire a preset prompt word template library, wherein the prompt word template library includes at least one prompt word template; The text and a prompt word template selected from the prompt word template library are combined into a prompt word, and the prompt word and a plurality of screened real comments are combined into a set of training data.
3. The method for training a review generation model according to claim 2, characterized in that: The text and the prompt word template selected from the prompt word template library are combined into a prompt word, and the prompt word and the screened multiple real comments are combined into a set of training data, including: For each text, splitting multiple real comments corresponding to the text into multiple comment groups; For each comment group, a prompt word template is selected from the prompt word template library, the prompt word template and the text are combined into a prompt word, and the prompt word and the comment group are combined into a set of training data.
4. A method for using a review generation model, characterized in that: The method comprises: Acquire a target text, and generate a prompt word according to the target text and a preset prompt word template; Using a generation model in a comment generation model to generate a plurality of predicted comments for the prompt word, and performing clustering and deduplication operations on the plurality of predicted comments, wherein the comment generation model is obtained by training using the training method according to any one of claims 1 to 3; Scoring the screened multiple predicted reviews using the reward model in the review generation model; The plurality of predicted comments are sorted according to the scores and then outputted.
5. The method for using the review generation model according to claim 4, characterized in that: The outputting the plurality of predicted comments after sorting them according to the scores includes: Deleting predicted comments whose scores are less than a preset threshold from the plurality of predicted comments; Sort the remaining multiple predicted reviews in descending order of ratings; Output the sorted predicted comments.
6. A training device for a review generation model, characterized in that: The comment generation model includes a generation model and a reward model, wherein the generation model is used to generate a plurality of predicted comments, and the reward model is used to score the plurality of predicted comments, so as to output the plurality of predicted comments after being sorted according to the scores. The device includes: A first training module is used to obtain a training data set, and train the generation model according to the training data set, wherein each group of training data in the training data set includes a prompt word and a plurality of real comments obtained after clustering and deduplication operations, and the prompt word is generated according to a preset prompt word template and the text corresponding to the real comment; A generation module, used to generate multiple predicted comments for the acquired prompt words using the generation model, mark each predicted comment as a superior comment or an inferior comment, and generate multiple groups of comparison data based on the multiple predicted comments, each group of comparison data including one superior comment and one inferior comment; A second training module is used to train the reward model according to each set of comparison data, wherein the loss function of the reward model is used to rank the superior comments and inferior comments in each set of comparison data, and to perform binary classification on the scores of the superior comments and the inferior comments based on a double threshold value so as to increase the gap between the superior comments and the inferior comments, perform loss calculation according to the binary classification result, and update the reward model according to the loss calculation; The second training module is also used to generate a first score for the superior comments in the comparison data and a second score for the inferior comments in the comparison data for each group of comparison data using the reward model; calculate the difference between the first score and a first threshold value using the loss function to obtain a first difference, and the first threshold value is positive; calculate the difference between the second score and a second threshold value using the loss function to obtain a second difference, and the second threshold value is negative; calculate the cross entropy of the first difference and the second difference using the loss function, and update the reward model according to the cross entropy.
7. A device for using a review generation model, characterized in that: The device comprises: An acquisition module, used for acquiring a target text, and generating a prompt word according to the target text and a preset prompt word template; A generation module, used to generate multiple predicted comments for the prompt word using a generation model in a comment generation model, and to perform clustering and deduplication operations on the multiple predicted comments, wherein the comment generation model is obtained by training using the training method according to any one of claims 1 to 3; A scoring module, used to score the screened multiple predicted reviews using the reward model in the review generation model; The output module is used to output the plurality of predicted comments after sorting them according to the scores.
8. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the training method of the comment generation model as described in any one of claims 1 to 3; or, the at least one instruction is loaded and executed by the processor to implement the method of using the comment generation model as described in claim 4 or 5.
9. A computer device, characterized in that: The computer device includes: a training device for the review generation model described in claim 6 or a use device for the review generation model described in claim 7.