A text quality evaluation model training method

Abstract

This invention discloses a training method for a text quality assessment model. Firstly, during the sampling phase, it introduces supervision of the annotation score by adding suggestive information to the prompt words, thereby improving sampling efficiency. Simultaneously, it proposes a two-dimensional similarity calculation method to serve as an evaluation metric in scoring tasks and to provide fine-grained rewards for reinforcement learning. Finally, it proposes a suggestive self-distillation loss function to accelerate the convergence speed of the scoring model training. Compared with existing technologies, this application, through suggestive sampling and the suggestive self-distillation loss function, addresses the pain points of low positive sample rate and slow training convergence speed in traditional fine-tuning methods, significantly improving sampling efficiency, reducing computational resource consumption, and enhancing training efficiency, thus enabling efficient training of domain-specific evaluation models.

Description

Technical Field

[0001] This invention relates to the field of model training technology, and more specifically to a method for training a text quality assessment model. Background Technology

[0002] In recent years, with the rapid development of reasoning capabilities of large language models, generative score models have shown application potential in automated evaluation tasks in multiple professional fields, such as intelligent diagnosis and treatment, educational assessment, and customer service quality inspection.

[0003] Compared to traditional scalar scoring models or reward models, generative scoring models can better leverage reasoning capabilities and are less dependent on training data. To train such models, rejection sampling fine-tuning (RFT) and group relative policy optimization (GRPO) are commonly employed.

[0004] The RFT method samples standard prompts multiple times using a base model and selects high-quality answers manually or by rules to build a fine-tuning dataset, aiming to improve the model's performance on the target task. The GRPO method, on the other hand, uses an within-group answer comparison mechanism to encourage relatively good answers and penalize relatively poor answers, thereby optimizing the model and improving its performance on a specific task while preserving its original capabilities.

[0005] However, when existing RFT and GRPO methods are directly applied to specialized text quality scoring tasks in vertical domains, their inherent technical shortcomings are amplified, leading to severe challenges in training efficiency and the upper limit of model performance, specifically manifested as follows:

[0006] On the one hand, poor domain adaptability leads to low sampling efficiency: in specialized domains, high-quality evaluation requires deep domain knowledge. Existing methods use ordinary sampling prompts, requiring an extremely large number of sampling attempts to obtain only a few usable positive samples. If the standard answer is explicitly stated in the prompts, the model may reveal that it knows the correct answer in its thought process, resulting in unusable data.

[0007] On the other hand, insufficient utilization of domain knowledge leads to a learning ceiling bottleneck: these methods do not make full use of labeled answers, but merely use labeled answers to filter responses or calculate rewards. However, if the model is unable to obtain answers similar to the labeled answers on its own on some samples, the filtering mechanism will have difficulty obtaining effective positive samples no matter how many times it is sampled, causing the model to be unable to learn effectively from these high-quality labeled data, and the training process will fall into a bottleneck.

[0008] Therefore, overcoming the above problems is an urgent issue that needs to be addressed by those skilled in the art. Summary of the Invention

[0009] In view of the above problems, the present invention is proposed to provide a text quality assessment model training method that overcomes or at least partially solves the above problems.

[0010] To achieve the above objectives, the present invention adopts the following technical solution: This invention provides a method for training a text quality assessment model, comprising the following steps: Acquire text data containing environmental information and samples to be evaluated, as well as the labeled scores of each sample to be evaluated; construct prompt word groups for the samples to be evaluated, wherein each prompt word in the prompt word group contains a standard prompt word and a suggestive prompt word; the suggestive prompt words have the same structure and semantics as the standard prompt words, and embed numerical suggestive information consistent with the labeled scores; The base model is used to sample the sample to be evaluated based on the hints and prompts to generate a first group of candidate responses; the score sequence of the candidate responses in the group is extracted, and a two-dimensional similarity algorithm is used to calculate the similarity value between the score sequence and the labeled score of the sample to be evaluated corresponding to the current candidate response; positive samples are selected based on the similarity value, and the base model is supervised and fine-tuned using the positive samples; Using the fine-tuned base model, the remaining samples to be evaluated are sampled a second time based on the cue words to generate a second set of candidate responses. The advantage of the second set of candidate responses is calculated, a strategy loss term is constructed, and the average KL divergence loss is introduced. The fine-tuned base model is then jointly optimized to obtain the text quality assessment model.

[0011] The remaining samples to be evaluated include the samples remaining from the first sampling and the rejected samples from the first sampling.

[0012] The text quality assessment model evaluates text quality based on standard prompt words.

[0013] Preferably, a prompt word group is constructed for the sample to be evaluated, including constructing corresponding prompt words for all non-empty subsets in the sample to be evaluated.

[0014] Preferably, a two-dimensional similarity algorithm is used to calculate the similarity value between the score sequence and the labeled score of the sample to be evaluated corresponding to the current candidate response, including:

[0015] In the formula, Represents a fractional sequence. where n is the length of the fractional sequence. This represents the labeled score of the sample to be evaluated corresponding to the current candidate response. , and These are the weights for ranking similarity and numerical similarity, respectively. , This indicates the index of the sample to be evaluated in the candidate response. Let X be the ranking of the i-th sample to be evaluated. Let be the ranking of the i-th sample to be evaluated in Y. This represents the maximum sum of squares error when the rankings are completely opposite. Let i represent the i-th sample to be evaluated in X. Let Y represent the i-th sample to be evaluated.

[0016] Preferably, positive samples are selected based on the similarity value, including: When the similarity value meets the preset threshold, the sample to be evaluated corresponding to the current candidate response is taken as a positive sample; otherwise, it is rejected.

[0017] Preferably, in the supervised fine-tuning of the pedestal model using the positive samples, the objective function includes the standard supervised fine-tuning loss and the first implied self-distillation loss, expressed as follows:

[0018] In the formula, This indicates the loss from standard monitoring and fine-tuning. Indicates hyperparameters, The first implied loss from self-distillation is represented as:

[0019] In the formula, Indicates standard prompt words, Indicates suggestive or suggestive words, Indicates a positive sample response. This indicates a gradient cutoff operation. Represents the base model. This represents the KL divergence.

[0020] Preferably, the advantages of the second group of candidate responses are calculated, including:

[0021]

[0022]

[0023]

[0024] In the formula, This indicates the strength of the j-th candidate response in the second group of candidate responses. This represents the j-th candidate response. This represents the reward for the j-th candidate response. The value represents the average reward of the second group of candidate responses, and N represents the number of candidate responses in the second group. Represents the score sequence of the j-th candidate response. Corresponding labeled score sequence Similarity value between them This indicates that the answer should be given according to a preset format. Extract the fraction sequence.

[0025] As a preferred option, the constructed policy loss term is represented as follows:

[0026] In the formula, N represents the number of candidate responses in the second group of candidate responses. This represents the finely adjusted base model. Indicates standard prompt words; The introduced average KL divergence is expressed as:

[0027] In the formula, Indicates suggestive or suggestive words, This represents the j-th candidate response.

[0028] Preferably, in this embodiment, the text data is intelligent diagnosis and treatment data, the environmental information is patient health status description information, and the sample to be evaluated is a candidate diagnosis and treatment plan for the current patient.

[0029] Alternatively, the text data may be educational question bank data, the environmental information may be the knowledge points and difficulty requirements associated with the questions, and the sample to be evaluated may be question text with scoring.

[0030] Alternatively, the text data may be customer service dialogue data, the environmental information may be user query information and dialogue history, and the sample to be evaluated may be customer service reply text.

[0031] In summary, this invention provides a text quality assessment model training method. During the sampling phase, it introduces supervision of the labeled score by adding suggestive information to the prompt words, thereby improving sampling efficiency. Simultaneously, it proposes a two-dimensional similarity calculation method to serve as an evaluation metric in scoring tasks and to provide fine-grained rewards for reinforcement learning. Finally, it proposes a suggestive self-distillation loss function to accelerate the convergence speed of the scoring model training. This application addresses the pain points of low positive sample rate and slow training convergence speed in traditional fine-tuning methods through suggestive sampling and the suggestive self-distillation loss function.

[0032] Compared with existing technologies, the beneficial effects include at least the following: We propose a suggestive sampling method that stimulates the model's potential through suggestion, significantly improving the sampling efficiency of high-quality samples during training, reducing the number of samplings and computational resources required to obtain effective training samples, and raising the upper limit of model performance. We propose a suggestive self-distillation loss function, which effectively improves the model's performance when there are no labeled answer hints, making it closer to the performance when there are hints, thus accelerating training convergence and improving training stability. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0034] Figure 1 This is a schematic diagram of the text quality assessment model training method in an embodiment of the present invention; Figure 2 This is a flowchart illustrating the rejection sampling fine-tuning process in an embodiment of the present invention; Figure 3 This is a flowchart illustrating the relative strategy optimization process for the implied group in an embodiment of the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] This invention discloses a text quality assessment model training method, which mainly includes three parts: training data construction, implied rejection sampling fine-tuning, and implied group relative policy optimization.

[0037] Specific reference Figure 1 , Figure 1 This is a schematic diagram of the training method for the text quality assessment model in this application.

[0038] In one embodiment, training data construction includes the following steps: S11. Collect and label data: Obtain text data containing environmental information and samples to be evaluated, as well as the labeling scores of each sample to be evaluated; This application can be applied to text scoring tasks in various fields, such as scoring treatment plans in the medical field, scoring the quality of questions in the educational field, or scoring the quality of responses in the customer service field. Specifically, the text data is intelligent diagnosis and treatment data, the environmental information is descriptions of the patient's health status, and the sample to be evaluated is candidate treatment plans for the current patient; or the text data is educational question bank data, the environmental information is the knowledge points and difficulty requirements associated with the questions, and the sample to be evaluated is the question text with a score; or the text data is customer service dialogue data, the environmental information is user query information and dialogue history, and the sample to be evaluated is customer service response text.

[0039] Furthermore, environmental information and samples to be evaluated are first collected, and the labeled scores of each sample are obtained. Taking the scoring of medical treatment plans in the medical field as an example, the health status description of a specific patient is first collected as environmental information E, and several corresponding treatment plans to be scored are collected as samples. In the stage of obtaining annotation scores, As input information.

[0040] S12. Constructing Data: Constructing prompt word groups for the samples to be evaluated. Each prompt word in the prompt word group contains a standard prompt word and a suggestive prompt word. The suggestive prompt word has the same structural semantics as the standard prompt word and embeds numerical suggestive information consistent with the labeled score. In this embodiment, the prompt word is used to prompt the model to score each sample sequentially and put the score into the special tag / boxed{}, represented as Prompt. Using pre-written prompts, relatively reliable scoring labels can be obtained from more capable models with larger parameters, allowing the score of each sample to be extracted according to rules. Also known as If necessary, the scores can be adjusted by experts to obtain reliable scoring labels.

[0041] In a preferred embodiment of this example, Prompt is not used directly. Instead of using a single training sample as input, it constructs all possible non-empty subset combinations based on the set of schemes within it to form a new input prompt word. .in, for Any non-empty subset of the set. The corresponding annotation score is directly inherited from the score of the corresponding scheme in the original annotation data. For example, when obtaining annotation data, a data point is Prompt. Then, during the training phase, corresponding training samples can be constructed. , , , , , , There are a total of six training data sets.

[0042] Through the above method, the training set data is finally obtained. Each data point includes environmental information and several combinations of scoring schemes to be used. and the corresponding labeled scores ,Right now

[0043] Furthermore, for the cue words, each training data point contains standard cue words and suggestive cue words, denoted as follows: and Therefore, the data of the entire dataset can be represented as .in, and Maintaining consistency in structure and semantics, the difference lies in whether or not explicit hints about the labeled scores are included. For ordinary prompt words, include an output example to constrain the model's output format for the final score list; while Then replace the ambiguous values ​​in this output example with specific numerical values ​​that match the score Y.

[0044] This application subtly guides the model by embedding labeled scores within format constraints, aiming to increase the model's probability of correct answers without disrupting its thought process. For example, if a data point has a labeled score of Y = (3,6,8), the corresponding... for: "The patient's health status is described as follows: ... The treatment plans to be scored are as follows: ... Please analyze and score each treatment plan, and finally put the scores into / boxed{}, for example, / boxed{x,x,x}" Then the corresponding for: "The patient's health status is described as follows: ... The treatment plans to be scored are as follows: ... Please analyze and score each treatment plan, and finally put the scores into / boxed{}, for example / boxed{3,6,8}."

[0045] The only difference lies in the example of the score output format. In this way, each training sample provides both unhindredded and hinted versions of the prompts to support subsequent model training.

[0046] In one embodiment, the process of rejecting sampling fine-tuning is as follows: Figure 2 As shown, it includes: S21, Hinted Sampling: Based on the training data prompt words Using the base model Hint sampling is performed to generate the first set of candidate responses; the sampling process is represented as follows:

[0047] in Indicates the first A sampled response is a word sequence of length T. From The scoring list can be extracted from it. And use it in conjunction with the labeled score High-quality positive samples were selected through comparison and screening.

[0048] S22. Filtering Positive Samples Based on Dual-Metric Similarity: To measure the similarity between the model's score and the labeled score distributions, this paper proposes a dual-metric similarity (DMS) calculation method, which can simultaneously evaluate the similarity between the two distributions from both ranking and numerical perspectives. In this task, the relative score ranking of each input sample to be scored is crucial, directly determining the model's ability to compare different solutions under the same environmental conditions, as the optimal solution is often chosen when using the model. The absolute score is also important; it is desirable for the model to provide stable scores with low randomness, giving the same score to the same solution under the same environmental conditions. Therefore, the following similarity calculation method is proposed: Answer from the model The length of the parsed data is 0. A list of scores (the number of samples to be scored in the input). ,in The values ​​are non-negative integers. The list of labeled scores is as follows: The formulas for calculating the similarity between overall ranking similarity and numerical similarity are as follows:

[0049] In the formula, Represents a fractional sequence. where n is the length of the fractional sequence. This represents the labeled score of the sample to be evaluated corresponding to the current candidate response. , and These are the weights for ranking similarity and numerical similarity, respectively. , This indicates the index of the sample to be evaluated in the current candidate response. Let X be the ranking of the i-th sample to be evaluated. Let be the ranking of the i-th sample to be evaluated in Y. This represents the maximum sum of squares error when the rankings are completely opposite. , Let i represent the i-th sample to be evaluated in X. Let Y represent the i-th sample to be evaluated.

[0050] The final calculated similarity value is a decimal between [0:1]. A value of 1 indicates that the two distributions are the same. The smaller the value, the more different the two distributions are. The larger the value, the closer the two distributions are in terms of both ranking and numerical values.

[0051] S23. Implied supervised fine-tuning: In some implementations, positive samples are further screened based on similarity values, and the positive samples are used to supervise the fine-tuning of the base model. In this embodiment, when the similarity value meets a preset threshold, the sample to be evaluated corresponding to the current candidate response is taken as a positive sample; otherwise, it is rejected. For example, from k candidate responses... Select the optimal sample:

[0052] Accept if and only if Sim(X,Y)>0.9 Otherwise, reject all samples.

[0053] Furthermore, high-quality positive samples were used to... Building a supervised fine-tuning dataset The base model is subjected to supervised fine-tuning; the objective function includes standard supervised fine-tuning loss and implied self-distillation loss, i.e.

[0054] In the formula, This indicates the loss from standard monitoring and fine-tuning. This represents a hyperparameter used to control the weights implied by the KL divergence loss during the supervised fine-tuning phase. This indicates implied loss during distillation.

[0055] In this embodiment, the standard supervised fine-tuning loss is:

[0056] In the formula, Let T represent the aforementioned supervised fine-tuning dataset, where T represents the sequence. The length of the lexicon. , Representing sequences respectively The sequence of the t-th word and the first t-1 words.

[0057] To accelerate training convergence, this application proposes a suggestive self-distillation loss, which aims to improve model performance by narrowing the output distribution of the model under two cues. The "high performance" direct distillation to Below. To achieve this, in calculation It is necessary to cut off its gradient at times to ensure that the optimization process only updates the gradient. The parameters enable it to be frozen, high-performance. They are distributed close together. Let This represents the gradient clipping operation. The loss term is defined as:

[0058] In the formula, Indicates standard prompt words, Indicates suggestive or suggestive words, Indicates a positive sample. This indicates a gradient cutoff operation. Represents the base model. Let represent the KL divergence. In the formula, the word-level probabilities of suggestive prompts and ordinary prompts are respectively used as the Q-distribution and P-distribution when calculating the KL divergence. When calculating the Q-distribution, the model parameters are frozen (i.e., gradient cutoff is performed), while when calculating the P-distribution, they are not frozen.

[0059] In one embodiment, the group-relative strategy optimization process is as follows: Figure 3 As shown, it includes: S31. Hint Sampling: For the portion of data included in the training dataset but not used during the supervised fine-tuning phase. Using the finely tuned base model A second set of candidate responses is generated by sampling N times using cue words:

[0060] S32. Two-dimensional similarity reward calculation: Utilizing a verifiable answer mechanism, a reward is given for each sampled response. Calculate the reward value. Reward function. Based on response Extracted model score X and labeled score The degree of similarity is calculated using the two-measure similarity formula mentioned earlier:

[0061]

[0062] Calculate the average reward for this group of responses. and advantages ;

[0063]

[0064] Furthermore, this implies the calculation of self-distillation loss and strategy loss: The policy loss term is constructed as follows:

[0065] In the formula, N represents the number of candidate responses in the second group of candidate responses. This represents the finely adjusted base model. T represents the standard prompt word, and T represents the response sequence. The length of the lexicon. , Representing the response sequence The sequence of the t-th word and the first t-1 words.

[0066] During the optimization phase, the goal is for the model to learn standard cues that do not contain suggestive information. Even with this, the correct answer can still be generated, therefore the policy gradient is... Calculated under the given conditions.

[0067] To accelerate the model in standard prompts The behavior below provides a cue for high performance. To align the behavior, this application introduces the average KL divergence as the implied self-distillation loss.

[0068] Specifically, for the responses obtained from the same set of samples The computational model is in the standard prompt and hints The average KL divergence that generates these responses is as follows:

[0069] In the formula, Indicates suggestive or suggestive words, Let represent the sample corresponding to the j-th candidate response.

[0070] The final optimization goal is:

[0071] It is a hyperparameter used to control the weight of the implied KL divergence loss term in the total loss.

[0072] This application achieves joint optimization. This not only guides the model with policy gradients based on reward signals, but also subjects it to direct constraints from its own performance under cueing. Ultimately, this not only significantly accelerates convergence but also improves the model's performance under standard cues. The final performance.

[0073] In one specific embodiment, taking the evaluation and scoring task of traditional Chinese medicine dietary therapy as an example, the task is defined as giving a traditional Chinese medicine dietary therapy and a set of standardized evaluation indicators (such as the rationality of the combination, the adaptability of ingredients, and safety), requiring the model to output a total score, ranging from 1 to 10 points, and having manual annotation or a rule system as the annotation answer.

[0074] The first step is to obtain training data; Collect information describing the health status of several patients and several dietary therapy plans for each patient.

[0075] In the data annotation phase, each patient's information and all their potential dietary therapy recipes were used as input. Qwen3235B, Deepseek R1, and GPT5.2 were used as annotation models to obtain scoring annotations. An expert vote was then used to select one annotation from the three models as the final annotation. 2000 annotated data points were obtained.

[0076] In the data construction phase, patient information and all non-empty subsets of all potential dietary therapies were combined to obtain training data with a number several times greater than that in the annotation phase, totaling 20,000 training data points. The general prompt words and suggestive prompt words for one of these data points are as follows: Common prompts: Patient's health status description: Six months post-right breast cancer surgery. Symptoms: Six months post-right breast cancer surgery, currently undergoing oral valproic acid endocrine therapy, experiencing chest tightness and shortness of breath (asthma); hot flashes, poor sleep, joint pain, pale tongue, white and greasy tongue coating, and a thready and wiry pulse. Dietary therapy recipes to be scored: ① Walnut twig and pumpkin stem soup: 60g walnut twigs, 2 pumpkin stems, 9g motherwort, appropriate amount of rice wine; ② Coix seed, water chestnut, and half-branch lotus soup: 30g coix seed, 30g water chestnut, 30g half-branch lotus, decocted in water and taken twice daily; ③ Beef porridge: 100g beef, appropriate amount of rice, a little five-spice powder and salt, thinly sliced ​​beef, cooked with rice into porridge, seasoned with a little five-spice powder and salt. Please analyze and score each dietary therapy recipe, and finally put the scores into / boxed{}, for example / boxed{x,x,x}.

[0077] Hint / Prompt: Patient's health status description: Six months post-right breast cancer surgery. Symptoms: Six months post-right breast cancer surgery, currently undergoing oral valproate for endocrine therapy, experiencing chest tightness and shortness of breath (asthma); hot flashes, poor sleep, joint pain, pale tongue with a white, greasy coating, and a thready, wiry pulse. Dietary therapy recipes to be scored: ① Walnut twig and pumpkin stem soup: 60g walnut twigs, 2 pumpkin stems, 9g motherwort, appropriate amount of rice wine; ② Coix seed, water chestnut, and half-branch lotus soup: 30g coix seed, 30g water chestnut, 30g half-branch lotus, decocted in water and taken twice daily; ③ Beef porridge: 100g beef, appropriate amount of rice, a little five-spice powder and salt, thinly sliced ​​beef, cooked with rice into porridge, seasoned with a little five-spice powder and salt. Please analyze and score each dietary therapy recipe, and finally put the scores into / boxed{}, for example / boxed{3,3,9}.

[0078] The second step is to subtly reject the sampling fine-tuning. The base model is sampled multiple times using suggestive words to generate several candidate answers; This process used 11,000 samples to sample Qwen3 4B. Each sample was sampled four times, and the final extracted score X in each sample was compared with the labeled Y using the following two-dimensional similarity calculation. The largest similarity score, greater than 0.9, was retained as training data. A total of 2,000 high-quality answers were obtained.

[0079] A supervised fine-tuning dataset was created by sampling 2000 high-quality answers and their corresponding general and suggestive prompts, which was used to fine-tune the base model. Specifically, the 2000 data points were divided into batches of 32, and low-rank adaptation was used on four NVIDIA GeForce RTX 3090 GPUs. The model was trained for three epochs on all the data.

[0080] Step 3: Optimize the relative strategies of the suggestion group; The fine-tuned model was then further trained using reinforcement learning. The data used for reinforcement learning consisted of 9000 data points rejected during the suggestion rejection sampling process, plus another 9000 unused data points from the dataset, totaling 18000 data points. These 18000 data points were then divided into batches of 16 and trained on four NVIDIA GeForce RTX 3090 GPUs. This included: The model obtained by supervising fine-tuning under frozen conditions is used as... non-frozen as During training, for each data point in a batch, the model is first tested... Eight samples were taken.

[0081]

[0082] For each sampled response Calculate the reward value. Reward function. Based on response Extracted model score X and labeled score The degree of similarity is calculated using the two-measure similarity formula mentioned earlier:

[0083]

[0084] In the DMS formula, and All values ​​are 0.5. This indicates that the data is extracted according to the preset format rules. The content within the curly braces “\boxed{}” represents a list of scores. For example, if The final score for the three dietary therapy recipes is: / boxed{3,3,9}. The extracted values ​​are: Given a list of numbers in the range [3, 3, 9]. Next, calculate the average reward for this group of responses. and advantages ;

[0085]

[0086] Then, in and Top Perform forward propagation to obtain , and Then disable gradient computation, in Top Perform forward propagation to obtain Then calculate the loss using the following formula.

[0087]

[0088]

[0089]

[0090] Where N is 8, Let's take 0.1. The above describes the loss calculation process for a single data point. In practice, after performing the above loss calculation on all 16 data points in each batch, the arithmetic mean of the resulting loss values ​​is taken to obtain the final loss for that batch. Gradient backpropagation is then performed on the final loss to model the data. Parameters updated. The model was trained for a total of 5 epochs on all data to obtain the final model.

[0091] Using the methods described above, the final model significantly improved its accuracy in the task of scoring dietary therapy prescriptions, and its performance on this task basically reached the level of the most advanced models.

[0092] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium, which may be a ROM, RAM, disk, optical disk, or other storage medium. The storage medium stores one or more programs, which, when executed by a processor, implement the text quality assessment model training method of this application.

[0093] This embodiment provides a computing device, which may be a desktop computer, laptop computer, smartphone, PDA handheld terminal, tablet computer or other terminal device with display function. The computing device includes a processor and a memory. The memory stores one or more programs. When the processor executes the program stored in the memory, it implements the text quality assessment model training method of this application.

[0094] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0095] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training a text quality assessment model, characterized in that, include: Obtain text data containing environmental information and samples to be evaluated, as well as the labeled scores of each sample to be evaluated; A set of prompt words is constructed for the sample to be evaluated. Each prompt word in the set of prompt words contains a standard prompt word and a suggestive prompt word. The suggestive prompts are semantically consistent with the standard prompts and embed numerical suggestive information consistent with the labeled scores; The base model is used to sample the sample to be evaluated based on the hints and prompts to generate a first group of candidate responses; the score sequence of the candidate responses in the group is extracted, and a two-dimensional similarity algorithm is used to calculate the similarity value between the score sequence and the labeled score of the sample to be evaluated corresponding to the current candidate response; positive samples are selected based on the similarity value, and the base model is supervised and fine-tuned using the positive samples; Using the fine-tuned base model, the remaining samples to be evaluated are sampled a second time based on the cue words to generate a second set of candidate responses. The advantage of the second set of candidate responses is calculated, a strategy loss term is constructed, and the average KL divergence loss is introduced to jointly optimize the fine-tuned base model to obtain a text quality assessment model. The calculation of the similarity value between the score sequence and the labeled score of the sample to be evaluated corresponding to the current candidate response using a two-dimensional similarity algorithm includes: In the formula, Represents a fractional sequence. where n is the length of the fractional sequence. This represents the labeled score of the sample to be evaluated corresponding to the current candidate response. , and These are the weights for ranking similarity and numerical similarity, respectively. , This indicates the index of the sample to be evaluated in the candidate response. Let X be the ranking of the i-th sample to be evaluated. Let be the ranking of the i-th sample to be evaluated in Y. This represents the maximum sum of squares error when the rankings are completely opposite. Let i represent the i-th sample to be evaluated in X. Let Y represent the i-th sample to be evaluated.

2. The text quality assessment model training method according to claim 1, characterized in that, Construct prompt word groups for the sample to be evaluated, including constructing corresponding prompt words for all non-empty subsets in the sample to be evaluated.

3. The text quality assessment model training method according to claim 1, characterized in that, Filtering positive samples based on the similarity value includes: When the similarity value meets the preset threshold, the sample to be evaluated corresponding to the current candidate response is taken as a positive sample; otherwise, it is rejected.

4. The text quality assessment model training method according to claim 1, characterized in that, In the supervised fine-tuning of the pedestal model using the positive samples, the objective function includes the standard supervised fine-tuning loss and the implied self-distillation loss, expressed as: In the formula, This indicates the loss from standard monitoring and fine-tuning. Indicates hyperparameters, This implies a loss during self-distillation, expressed as: In the formula, Indicates standard prompt words, Indicates suggestive or suggestive words, Indicates a positive sample response. This indicates a gradient cutoff operation. Represents the base model. This represents the KL divergence.

5. The text quality assessment model training method according to claim 1, characterized in that, Calculating the advantage of the second group of candidate responses includes: In the formula, This indicates the strength of the j-th candidate response in the second group of candidate responses. This represents the j-th candidate response. This represents the reward for the j-th candidate response. The value represents the average reward of the second group of candidate responses, and N represents the number of candidate responses in the second group. Represents the score sequence of the j-th candidate response. Corresponding labeled score sequence Similarity value between them This indicates that the answer should be given according to a preset format. Extract the fraction sequence.

6. The text quality assessment model training method according to claim 1, characterized in that, The constructed policy loss term is represented as follows: In the formula, N represents the number of candidate responses in the second group of candidate responses. This represents the finely adjusted base model. T represents the standard prompt word; T represents the sequence. The length of the lexicon. , Representing sequences respectively The t-th word and the sequence of the first t-1 words; The introduced average KL divergence is expressed as: In the formula, Indicates suggestive or suggestive words, This represents the j-th candidate response.

7. The text quality assessment model training method according to any one of claims 1-6, characterized in that, The text data is intelligent diagnosis and treatment data, the environmental information is a description of the patient's health status, and the sample to be evaluated is a candidate diagnosis and treatment plan for the current patient.

8. The text quality assessment model training method according to any one of claims 1-6, characterized in that, The text data is educational question bank data, the environmental information is the knowledge points and difficulty requirements associated with the questions, and the sample to be evaluated is the question text with scoring.

9. The text quality assessment model training method according to any one of claims 1-6, characterized in that, The text data is customer service dialogue data, the environmental information is user query information and dialogue history, and the sample to be evaluated is customer service reply text.

Images