Scoring model training method and apparatus, storage medium, and electronic device

Abstract

The application provides a score model training method and device, a storage medium and an electronic device, and the method comprises the following steps: performing reverse thinking chain synthesis on each supervised training sample in a supervised training sample set respectively to obtain a thinking chain label of each supervised training sample; calling an initial score model to determine the prediction probability data of each supervised training sample respectively; calculating the model loss value of the initial score model based on the prediction probability data of each supervised training sample and a model training label, wherein the model training label of one supervised training sample comprises the training sample label and the thinking chain label of the corresponding supervised training sample; optimizing the model parameters in the initial score model in the direction of reducing the model loss value of the initial score model to obtain an initial score model after model optimization; and determining a target score model based on the initial score model after model optimization. The embodiments of the application can improve the model performance of the score model.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a scoring model training method, apparatus, storage medium, and electronic device. Background Technology

[0002] Currently, when scoring answers to questions, related technologies typically use generalized large language models for direct scoring. This results in scores being influenced by surface language features and the arrangement of prompts, leading to low accuracy. Therefore, there is currently no satisfactory solution for improving the performance of scoring models to enhance accuracy. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a scoring model training method, apparatus, storage medium, and electronic device to solve the problems of low accuracy of scoring results caused by related technologies; that is, embodiments of the present invention can train the scoring model through a supervised training sample set to obtain a target scoring model with high model performance, thereby effectively improving the performance of the scoring model, and the input of the scoring model does not need to include prompt data, etc., and the accuracy of the scoring results can be effectively improved through the target scoring model.

[0004] According to one aspect of the present invention, a scoring model training method is provided, the method comprising: Obtain a set of supervised training samples. A supervised training sample includes a model training input data and a corresponding supervised training sample label. A model training input data includes at least one of the following: a question, a candidate answer, and a question scoring criterion. A training sample label includes at least one of the following: a score label and a scoring reason label. A question scoring criterion includes the scoring criteria for the corresponding question in each score range. Reverse thinking chain synthesis is performed on each supervised training sample in the supervised training sample set to obtain the thinking chain label of each supervised training sample; The initial scoring model is invoked, and the model training input data is determined based on the model training input data in each of the supervised training samples to determine the predicted probability data of each supervised training sample. Based on the predicted probability data and model training labels of each supervised training sample, the model loss value of the initial scoring model is calculated; wherein, the model training label of a supervised training sample includes the training sample label and the thought chain label of the corresponding supervised training sample. The model parameters in the initial scoring model are optimized in the direction of reducing the model loss value of the initial scoring model to obtain an optimized initial scoring model; and based on the optimized initial scoring model, a target scoring model is determined, wherein the predicted score output by a scoring model belongs to the multiple score levels.

[0005] According to another aspect of the present invention, a scoring model training apparatus is provided, the apparatus comprising: The acquisition unit is used to acquire a set of supervised training samples. A supervised training sample includes a model training input data and a corresponding supervised training sample label. A model training input data includes at least one of the following: a question, a candidate answer, and a question scoring criterion. A training sample label includes at least one of the following: a score label and a scoring reason label. A question scoring criterion includes the scoring criteria for the corresponding question in each score level among multiple score levels. The processing unit is used to perform reverse thinking chain synthesis on each supervised training sample in the supervised training sample set to obtain the thinking chain label of each supervised training sample. The processing unit is also used to call the initial scoring model, and determine the predicted probability data of each supervised training sample based on the model training input data in each supervised training sample; The processing unit is further configured to calculate the model loss value of the initial scoring model based on the predicted probability data and model training labels of each supervised training sample; wherein, the model training label of a supervised training sample includes the training sample label and the thinking chain label of the corresponding supervised training sample. The processing unit is further configured to optimize the model parameters in the initial scoring model in the direction of reducing the model loss value of the initial scoring model, thereby obtaining an optimized initial scoring model; and based on the optimized initial scoring model, determine the target scoring model, wherein the predicted score output by a scoring model belongs to the multiple score levels.

[0006] According to another aspect of the present invention, an electronic device is provided, the electronic device including a processor and a memory storing a program, wherein the program includes instructions that, when executed by the processor, cause the processor to perform the methods mentioned above.

[0007] According to another aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods mentioned above is provided.

[0008] This invention provides an embodiment of a supervised training sample set. Each supervised training sample includes model training input data and a corresponding training sample label. The model training input data includes at least one of the following: a question, candidate answers, and question scoring criteria. Each training sample label includes at least one of the following: a score label and a scoring reason label. The question scoring criteria for a question include the scoring criteria for each score level across multiple score levels. The invention also performs reverse thinking chain synthesis on each supervised training sample in the supervised training sample set to obtain the thinking chain label for each supervised training sample. Then, an initial scoring model can be invoked, and based on the model training input data in each supervised training sample, the predicted probability data for each supervised training sample can be determined. Based on the predicted probability data and model training labels of each supervised training sample, the model loss value of the initial scoring model can be calculated. The model training label for each supervised training sample includes the corresponding training sample label and thinking chain label. Accordingly, the model parameters in the initial scoring model can be optimized in the direction of reducing the model loss value of the initial scoring model, resulting in an optimized initial scoring model. Based on the optimized initial scoring model, a target scoring model is determined, where the predicted score output by a scoring model belongs to multiple score levels. It is evident that this embodiment of the invention can train the scoring model using a supervised training sample set to obtain a target scoring model with higher performance, effectively improving the performance of the scoring model. Furthermore, the input of the scoring model does not need to include prompt data, and the accuracy of the scoring results can be effectively improved through the target scoring model. Attached Figure Description

[0009] Further details, features, and advantages of the invention are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which: Figure 1 A flowchart illustrating a scoring model training method according to an exemplary embodiment of the present invention is shown; Figure 2 A flowchart illustrating another scoring model training method according to an exemplary embodiment of the present invention is shown; Figure 3 A flowchart illustrating another scoring model training method according to an exemplary embodiment of the present invention is shown; Figure 4 A flowchart illustrating a reward reinforcement training process according to an exemplary embodiment of the present invention is shown; Figure 5 A schematic block diagram of a scoring model training apparatus according to an exemplary embodiment of the present invention is shown; Figure 6 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present invention is shown. Detailed Implementation

[0010] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the invention. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the invention.

[0011] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0012] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in this invention are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0013] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0014] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0015] It should be noted that the execution subject of the scoring model training method provided in this embodiment of the invention can be one or more electronic devices, and this embodiment of the invention does not limit this; wherein, the electronic device can be a terminal (i.e., a client) or a server. Therefore, when the execution subject includes multiple electronic devices, and among the multiple electronic devices includes at least one terminal and at least one server, the scoring model training method provided in this embodiment of the invention can be jointly executed by the terminal and the server. Accordingly, the terminal mentioned herein may include, but is not limited to: smartphones, tablets, laptops, desktop computers, intelligent voice interaction devices, etc. The server mentioned herein can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, etc.

[0016] Based on the above description, this embodiment of the invention proposes a scoring model training method, which can be executed by the aforementioned electronic device (terminal or server); or, the scoring model training method can be executed jointly by the terminal and the server, etc. For ease of explanation, the following description will use the execution of the scoring model training method by an electronic device as an example; such as Figure 1 As shown, the training method for this scoring model may include the following steps S101-S105: S101, Obtain a set of supervised training samples. A supervised training sample includes a model training input data and the training sample label of the corresponding supervised training sample.

[0017] The model training input data may include, but is not limited to, at least one of the following: questions, candidate answers, and question scoring criteria. A training sample label may include, but is not limited to, at least one of the following: score label and scoring reason label. A question's scoring criteria include the scoring criteria for each score level across multiple score levels. It should be understood that candidate answers in the model training input data may be candidate answers for the questions included in the corresponding model training input data, and question scoring criteria in the model training input data may be the question scoring criteria for the questions included in the corresponding model training input data. Correspondingly, the training sample label of a supervised training sample may represent the label of the candidate answers in the corresponding supervised training sample, such as the score label and scoring reason label of the candidate answer, etc.

[0018] Optionally, a question (also referred to as a problem) can be any question in the question set, that is, the questions in a training sample (such as a supervised training sample) can be any question in the question set. This embodiment of the invention does not limit this. Optionally, the question set can be set according to experience or actual needs. This embodiment of the invention does not limit this. Optionally, a question can include, but is not limited to, at least one of the following: a question stem and a question stem context (such as question type, materials, etc.). This embodiment of the invention does not limit this. For example, when a question x includes a question stem q and a question stem context p, it can be denoted as x=(q,p).

[0019] Optionally, multiple score levels can be set based on experience or actual needs, and this embodiment of the invention does not limit this. For example, multiple score levels can include 1 to N, where N is an integer greater than 1, such as 1, 2, ..., N, etc. Optionally, a scoring criterion can also be called a judgment clause, which provides an executable judgment clause for each score level. Optionally, the scoring criterion under a score level may include, but is not limited to, at least one of the following: key points required to be covered by the corresponding score level, common errors and deduction points, boundary descriptions, and output format constraints (such as whether a conclusion must be given, whether the basis must be given, whether it must be output in a specified field, etc.), etc.; this embodiment of the invention does not limit this. Optionally, the scoring criterion for a question may also include explicit distinguishing conditions for the corresponding question in any two adjacent score levels (such as 3 points and 4 points, etc.), etc.; this embodiment of the invention does not limit this. Optionally, the question scoring criterion can be a preset unified structural expression form; optionally, the preset unified structural expression form can be set based on experience or actual needs, and this embodiment of the invention does not limit this. Optionally, the scoring criteria for a question can be stored in a criterion database, and each question corresponds to a unique criterion index. The criterion index corresponding to a question can be used to indicate the scoring criteria for the corresponding question.

[0020] Optionally, embodiments of the present invention can realize the construction and solidification of criteria, thereby realizing the construction of a question scoring criterion bound to each question in the question set. Optionally, the generation of candidate question scoring criteria for a question can be obtained by multi-source generation and merging (e.g., generated through multiple large language models, and merged through a large language model to obtain candidate question scoring criteria, etc.), and then finalized by manual structured revision; or, the question scoring criteria for the corresponding question can be directly formulated manually, etc.; embodiments of the present invention do not limit this. Optionally, during the finalization, general statements (e.g., relatively complete, relatively clear, basically correct, etc.) can be rewritten into a verifiable list of terms, giving clear distinguishing conditions for adjacent levels and a unified structural expression form of criteria (e.g., unified to a preset unified structural expression form, etc.), etc.; embodiments of the present invention do not limit this. Optionally, after the scoring criteria for a question are finalized, they can be stored in a criteria library. The electronic device can generate a unique criterion index for each question, enabling the retrieval of the scoring criteria corresponding to the unique criterion index during the training and inference phases. This allows the scoring criteria for any question to be determined from the criteria library based on the criterion index corresponding to any question. One criterion index can be used to indicate one scoring criterion for a question, and one question corresponds to one criterion index. Optionally, the electronic device can also display a criteria revision interface, allowing users (such as administrators or reviewers) to perform manual revisions and / or finalization on the criteria revision interface. The finalized scoring criteria can then be stored in the criteria library to establish a unique index between questions and their scoring criteria, etc. Figure 2 As shown.

[0021] Optionally, a question scoring criterion can be stored in structured text, such as a parsable format that includes at least one field. Optionally, the at least one field may include, but is not limited to, at least one of the following: score range, scoring criteria for each score level, a list of deduction items, output requirements, etc. This embodiment of the invention does not limit this. In other words, this embodiment of the invention does not limit the specific content and structure of the question scoring rules. Based on this, this embodiment of the invention can ensure stable parsing for subsequent training and inference through structured text. Correspondingly, after adding the question scoring criteria of each question in the question set to the criterion library, the criterion library can be established.

[0022] Optionally, after the criterion library is established, a supervised training sample set can be constructed based on the criterion library, and so on. It should be understood that the scoring reason can be used to express the basis for scoring and the reasons for deduction, so that the scoring model can learn the judgment logic of "scoring according to the criteria", rather than just learning the score label; for example, a scoring reason (such as a scoring reason label) may include, but is not limited to, at least one of the following: explanation of the hit of the scoring criteria, key defect location, explanation of the missing essential points, etc., which are not limited in this embodiment of the invention; based on this, the scoring reason can characterize the judgment basis on the boundary of adjacent score levels, such as characterizing the judgment basis of manual review on the boundary of adjacent score levels, etc.

[0023] In this embodiment of the invention, the methods for obtaining the supervised training sample set may include, but are not limited to, the following: The first method of acquisition: The electronic device has a set of supervised training samples stored in its own storage space. In this case, the electronic device can obtain the set of supervised training samples from its own storage space.

[0024] The second method of acquisition: The electronic device can obtain a download link for the supervised training sample set. In this case, the electronic device can use the download link to download the supervised training sample set to obtain the supervised training sample set.

[0025] The third acquisition method: Electronic devices can construct a supervised training sample set through a criterion library to acquire the supervised training sample set, etc. Optionally, when constructing the supervised training sample set through the criterion library, the electronic device can acquire at least one question (e.g., determine at least one question from a question set) and acquire candidate answers for each question in the at least one question, thereby adding each question and its corresponding candidate answer to a supervised training sample, so that a supervised training sample includes one question and one candidate answer for the corresponding question; optionally, the number of candidate answers for a question can be one or more, and this embodiment of the invention does not limit this; wherein, the number of supervised training samples including any question can be the number of candidate answers for any question, so that the candidate answers in different supervised training samples including any question are different. Further, for any supervised training sample in the supervised training sample set, the question scoring criteria for the question in any supervised training sample can be determined from the criterion library, and the determined question scoring criteria can be added to any supervised training sample, so that the model training input data in any supervised training sample includes the question, candidate answer, and question scoring criteria. Optionally, the electronic device can also acquire the training sample label of any supervised training sample, thereby adding the training sample label of any supervised training sample to any supervised training sample to complete the acquisition of any supervised training sample, and thus realize the acquisition of the supervised training sample set.

[0026] Optionally, candidate answers to a question can be output by any large language model, obtained through online sampling, or generated by multiple large language models, etc.; this embodiment of the invention does not limit this. Optionally, candidate answers to a question can be obtained in different ways to cover different quality levels; optionally, to avoid excessive concentration of score ranges in the supervised training sample set (e.g., a large proportion of medium-to-high score samples), supplementary measures can be taken for scarce score ranges. Optionally, this embodiment of the invention can generate multiple candidate answers for the same question and can introduce a consistency screening mechanism to retain only candidate answers with high score consistency for the construction of supervised training samples; optionally, consistency can be determined by the majority vote percentage of multiple reviewers (e.g., the higher the percentage of the maximum number of votes, the higher the consistency), etc.; for example, candidate answers with a maximum vote percentage less than a preset vote percentage threshold can be filtered out. The preset vote percentage threshold can be set according to experience or actual needs, and this embodiment of the invention does not limit this. Optionally, the score label of the supervised training sample containing the selected candidate answers can be the majority vote score (i.e., the score with the largest percentage of votes); or, the score label of a supervised training sample can be manually set, etc. Optionally, multiple reviewers may include, but are not limited to, at least one of the following: at least one rater and at least one large language model, etc., which are not limited in this embodiment of the invention. Optionally, the rating reason label of a supervised training sample can be generated by at least one large language model, or it can be manually labeled, etc., which are not limited in this embodiment of the invention.

[0027] Based on this, the embodiments of the present invention can realize the construction of supervised training samples, which can also be referred to as the construction of training samples.

[0028] It should be noted that the supervised training sample set can cover all score levels and may include candidate answers for adjacent score levels as well as the reasons for defining adjacent score levels, thereby effectively improving the coverage of the supervised training sample set on the boundaries of each score level, making the scoring model more stable in distinguishing adjacent score levels.

[0029] S102, perform reverse thinking chain synthesis on each supervised training sample in the supervised training sample set to obtain the thinking chain label of each supervised training sample.

[0030] Among them, reverse thinking chain synthesis can also be expressed as reverse CoT (Chain-of-Thought) synthesis, or reverse thinking chain synthesis; optionally, a thinking chain can also be called a thinking chain.

[0031] Optionally, for any supervised training sample in the supervised training sample set, the electronic device can invoke the reverse thinking chain synthesis model to synthesize the reverse thinking chain for any supervised training sample, thereby obtaining the thinking chain label for any supervised training sample. It should be noted that the specific structure of the reverse thinking chain synthesis model is not limited in this embodiment of the invention; for example, the reverse thinking chain synthesis model can be any large language model (i.e., a large model), or any lightweight semantic model, etc. Optionally, the model structure of the reverse thinking chain synthesis model can be the same as or different from the model structure of the current scoring model; this embodiment of the invention does not limit this. Optionally, reverse thinking chain synthesis can be completed by an independent large language model, or by the same model (such as the current scoring model) under a reverse derivation prompt template, etc.; this embodiment of the invention does not limit this. Optionally, the reverse derivation prompt template can be set according to experience or actual needs; this embodiment of the invention does not limit this.

[0032] Optionally, the electronic device can also construct prompt text for any supervised training sample using a standard reverse prompting algorithm (also known as a reverse prompting algorithm), thereby invoking a reverse thinking chain synthesis model. Based on the prompt text of any supervised training sample, the model performs reverse thinking chain synthesis on any supervised training sample to obtain a thinking chain label for that supervised training sample, and so on. For example, the electronic device can obtain a reverse derivation prompt template and embed any supervised training sample into the template to form prompt text for that supervised training sample. This prompt text is then input into the reverse thinking chain synthesis model, driving it to generate an initial thinking chain (also known as a reverse thinking chain).

[0033] Optionally, the electronic device can also use a cosine similarity algorithm to verify the consistency between the forward derivation result of the initial thinking chain and the candidate answers in any supervised training sample. Then, it can correct the thinking chain through a multi-round iterative optimization algorithm to obtain a standard inverse thinking chain, which can then be used as the thinking chain label for any supervised training sample, and so on. Optionally, the electronic device can also use the thinking chain of any supervised training sample output by the inverse thinking chain synthesis model as the thinking chain label for any supervised training sample to achieve inverse thinking chain synthesis for any supervised training sample. For example, the aforementioned initial thinking chain can be used as the thinking chain label for any supervised training sample, and so on. The specific implementation of inverse thinking chain synthesis is not limited in this invention. Optionally, the inverse thinking chain synthesis model can be a model trained using a set of inverse synthesis training samples. For example, an inverse synthesis training sample may include inverse thinking chain synthesis input training data and inverse synthesis thinking chain labels; or, the inverse thinking chain synthesis model can be set according to experience or actual needs, and so on. Optionally, the set of inverse synthesis training samples can be set according to experience or actual needs, and this embodiment of the invention does not limit this.

[0034] In this embodiment of the invention, for any supervised training sample in the supervised training sample set, the input for the reverse thinking chain synthesis can be any supervised training sample, such as the question, candidate answer, question scoring criteria, score label, and scoring reason label in any supervised training sample, and then output the thinking chain label of any supervised training sample. Optionally, a thinking chain (such as a thinking chain label) can be a constrained reasoning text, the content of which can be organized according to a preset reasoning order; optionally, the preset reasoning order can be set according to experience or actual needs, and this embodiment of the invention does not limit this; for example, the preset reasoning order can be "checking each item against the question scoring criteria (such as checking whether it meets the scoring criteria under each score level, etc.) - locating the satisfied items and deducted items - summarizing the score boundaries (such as how to define scores for adjacent score levels, etc.) - obtaining the final score", etc.

[0035] It should be understood that, for any supervised training sample in the supervised training sample set, when synthesizing the reverse thinking chain in this embodiment of the invention, the scoring reason label in any supervised training sample can be used as an anchor point. This requires that the generated thinking chain label and the scoring reason label be consistent in terms of deduction points, hit clauses, and score labels, thereby making the thinking chain controllable and traceable. Based on this, this embodiment of the invention can effectively limit the synthesis of thinking chain labels, rather than allowing a general model to generate thinking freely, thus effectively improving the accuracy of the thinking chain labels.

[0036] S103, call the initial scoring model, and train the model input data based on each supervised training sample to determine the predicted probability data for each supervised training sample.

[0037] Optionally, a scoring model (such as an initial scoring model) can be any large language model, etc., and the embodiments of the present invention do not limit this; that is, the embodiments of the present invention do not limit the model structure of the scoring model.

[0038] In this embodiment of the invention, for any supervised training sample in the supervised training sample set, the electronic device can input the model training input data in any supervised training sample into the initial scoring model, so as to output the predicted probability data of any supervised training sample through the initial scoring model.

[0039] S104, Calculate the model loss value of the initial scoring model based on the predicted probability data and model training labels of each supervised training sample; wherein, the model training label of a supervised training sample includes the training sample label and the thinking chain label of the corresponding supervised training sample.

[0040] In this embodiment of the invention, the thought chain label can be used as an intermediate supervision signal in the model training process to improve the performance of the scoring model.

[0041] Optionally, the predicted probability data of a supervised training sample may include, but is not limited to, the conditional probability of each word in the model training label of the corresponding supervised training sample. Based on this, when calculating the model loss value of the initial scoring model based on the predicted probability data and model training labels of each supervised training sample, the electronic device may determine the conditional probability of each word in the model training label of each supervised training sample from the predicted probability data of each supervised training sample; and use the conditional probability of each word in the model training label of each supervised training sample to calculate the model loss value of the initial scoring model. Here, a word can be a token, or it can be represented as a text unit, encoding unit, etc.

[0042] Optionally, the supervision objective in this embodiment of the invention can be to maximize the conditional probability of the scoring model for the target output sequence (i.e., the model training labels). Optionally, the electronic device can use the conditional probability of each word in the model training labels of each supervised training sample to calculate the model loss value for each supervised training sample, and use the sum of the model loss values ​​for each supervised training sample as the model loss value of the initial scoring model. For example, when the model loss value of the initial scoring model is calculated using cross-entropy, the electronic device can use Formula 1.1 to calculate the model loss value for any supervised training sample using the conditional probability of each word in the model training labels of any supervised training sample: Formula 1.1 Where L can represent the model loss value under any supervised training sample, u i p(u) can represent the i-th word in the model training labels of any supervised training sample. i |x,a,R) can represent the conditional probability of the i-th word (i.e., the conditional probability of the i-th word under the model training input data in any supervised training sample), x can represent the question in any supervised training sample, a can represent the candidate answer in any supervised training sample, and R can represent the question scoring criterion in any supervised training sample.

[0043] S105, optimize the model parameters in the initial scoring model in the direction of reducing the model loss value of the initial scoring model to obtain the optimized initial scoring model; and determine the target scoring model based on the optimized initial scoring model, wherein the predicted score output by a scoring model belongs to the multiple score levels.

[0044] Optionally, the electronic device can continue to train the optimized initial scoring model using the supervised training sample set until the supervised training convergence condition is met (such as the number of iterations reaching a preset supervised training number threshold, or the model loss value being less than a preset supervised training loss threshold, etc.). The scoring model that reaches the supervised training convergence condition is then used as the supervised training scoring model. Optionally, both the preset supervised training number threshold and the preset supervised training loss threshold can be set based on experience or actual needs; this embodiment of the invention does not limit this. Optionally, the electronic device can use the supervised training scoring model as the target scoring model, or it can continue to perform reward reinforcement training on the supervised training scoring model to obtain the target scoring model (i.e., further determine the target scoring model based on the supervised training scoring model), etc.; this embodiment of the invention does not limit this. Optionally, reward reinforcement training can also be called Reinforcement Learning with Verifiable Rewards (RLVR), or reinforcement learning based on verifiable rewards, etc.

[0045] In this embodiment of the invention, through the above-mentioned model training method, the scoring model not only learns "how many points to give", but also learns "why to give that point, on what terms to give that point, and how to distinguish adjacent score levels", which can provide a stable initialization capability for subsequent verifiable reward reinforcement training and can significantly reduce the scale drift caused by relying solely on the general model to understand the criteria in real time.

[0046] Optionally, the electronic device can also acquire data to be scored, which may include, but is not limited to, the target question and the answer to the target question to be scored. Correspondingly, based on the data to be scored, target model input data can be constructed. For example, the scoring criteria for the target question can be obtained from a criterion library, and the target question, the answer to the target question to be scored, and the scoring criteria can be used to construct the target model input data. Further, the target scoring model can be invoked, and based on the target model input data, the scoring result of the answer to be scored can be determined. The scoring result may include, but is not limited to, at least one of the following: the predicted score of the answer to be scored and the scoring reason. In other words, the electronic device can input the target model input data into the target scoring model to output the scoring result of the answer to be scored through the target scoring model. Optionally, the scoring result may also include the thought process of the answer to be scored. The predicted score of the answer to be scored can be any of multiple score levels, that is, the predicted score of the answer to be scored can belong to multiple score levels.

[0047] Optionally, a scoring model can output scoring results according to a specified output template; optionally, the specified output template can be set according to experience or actual needs, and this embodiment of the invention does not limit this. For example, the specified output template can enable the scoring model to first output the thought process, and then output the predicted score and the scoring reason, so as to prompt the scoring model to explicitly learn the judgment path of "verifying each item according to the question scoring criteria and converging to the score level".

[0048] Optionally, in other embodiments, the data to be scored may also include the scoring criteria for the target question, in which case the data to be scored can be used as input data for the target model, and so on.

[0049] In summary, the embodiments of the present invention can retain the scoring reasons during the manual annotation stage, and based on the "reverse CoT synthesis" mechanism, use the scoring reason labels and score labels as anchor points to synthesize the thinking chain, so that the training samples simultaneously have a consistent expression of "conclusion (score) - basis (reason) - reasoning process (thinking chain)". This enables the scoring model to learn the judgment path of verifying the criteria of manual review one by one and converging to the graded score during the supervised training stage, which significantly improves the alignment problem caused by the lack of calibration of general large language models without scoring data.

[0050] This invention provides an embodiment of a supervised training sample set. Each supervised training sample includes model training input data and a corresponding training sample label. The model training input data includes at least one of the following: a question, candidate answers, and question scoring criteria. Each training sample label includes at least one of the following: a score label and a scoring reason label. The question scoring criteria for a question include the scoring criteria for each score level across multiple score levels. The invention also performs reverse thinking chain synthesis on each supervised training sample in the supervised training sample set to obtain the thinking chain label for each supervised training sample. Then, an initial scoring model can be invoked, and based on the model training input data in each supervised training sample, the predicted probability data for each supervised training sample can be determined. Based on the predicted probability data and model training labels of each supervised training sample, the model loss value of the initial scoring model can be calculated. The model training label for each supervised training sample includes the corresponding training sample label and thinking chain label. Accordingly, the model parameters in the initial scoring model can be optimized in the direction of reducing the model loss value of the initial scoring model, resulting in an optimized initial scoring model. Based on the optimized initial scoring model, a target scoring model is determined, where the predicted score output by a scoring model belongs to multiple score levels. It is evident that this embodiment of the invention can train the scoring model using a supervised training sample set to obtain a target scoring model with higher performance, effectively improving the performance of the scoring model. Furthermore, the input of the scoring model does not need to include prompt data, and the accuracy of the scoring results can be effectively improved through the target scoring model.

[0051] Based on the above description, this embodiment of the invention also proposes another method for training a scoring model. Accordingly, this scoring model training method can be executed by the aforementioned electronic device (terminal or server); or, it can be executed jointly by the terminal and the server, etc. For ease of explanation, the following description will use the execution of this scoring model training method by an electronic device as an example; please refer to [link to relevant documentation]. Figure 3 The training method for this scoring model may include the following steps S301-S309: S301, Obtain a set of supervised training samples. A supervised training sample includes a model training input data and the training sample label of the corresponding supervised training sample.

[0052] S302, perform reverse thinking chain synthesis on each supervised training sample in the supervised training sample set to obtain the thinking chain label of each supervised training sample.

[0053] S303, invoke the initial scoring model, and determine the predicted probability data for each supervised training sample based on the model training input data in each supervised training sample.

[0054] S304, Calculate the model loss value of the initial scoring model based on the predicted probability data and model training labels of each supervised training sample; wherein, the model training label of a supervised training sample includes the training sample label and the thinking chain label of the corresponding supervised training sample.

[0055] S305, optimize the model parameters in the initial scoring model in the direction of reducing the model loss value of the initial scoring model to obtain the optimized initial scoring model; and determine the supervised training scoring model based on the optimized initial scoring model.

[0056] The supervised training scoring model can be the scoring model that has completed supervised training. Based on this, embodiments of the present invention can use the scoring model that has completed supervised training to initialize the scoring model in the verifiable reward reinforcement training process. That is, embodiments of the present invention can continue to perform verifiable reward reinforcement training (i.e., reward reinforcement training, or simply reinforcement training) on ​​the supervised training scoring model to obtain the target scoring model.

[0057] S306, determine the target reward reinforcement training sample set, and call the supervised training scoring model to determine the K scoring candidate output results corresponding to each reward reinforcement training sample in the target reward reinforcement training sample set, where K is a positive integer; wherein, a scoring candidate output result includes a predicted score and / or a scoring reason.

[0058] Optionally, the target reward reinforcement training sample set can be determined from the supervised training sample set. In this case, the initial reward reinforcement training sample set may include the model training input data from each supervised training sample. That is, one reward reinforcement training sample in the initial reward reinforcement training sample set can be the model training input data from a supervised training sample. The target reward reinforcement training sample set can be determined from the supervised training sample set by sampling the target reward reinforcement training sample set from the initial reward reinforcement training sample set. Alternatively, the electronic device can directly obtain the initial reward reinforcement training sample set (e.g., from its own storage space, or by downloading from a download link of the reward reinforcement training sample set), and sample the target reward reinforcement training sample set from the initial reward reinforcement training sample set, etc. The embodiments of the present invention do not limit this. Optionally, the data included in a reward reinforcement training sample may be the same as the model training input data in a supervised training sample, or may be different from the model training input data in any supervised training sample. The embodiments of the present invention do not limit this.

[0059] Optionally, a reward-reinforced training sample may include, but is not limited to, at least one of the following: questions, candidate answers, and question scoring criteria.

[0060] Optionally, for any reward reinforcement training sample in the target reward reinforcement training sample set, the electronic device can call the supervised training scoring model to determine the K candidate scoring outputs corresponding to any reward reinforcement training sample. In other words, any reward reinforcement training sample can be input into the supervised training scoring model K times, thereby outputting the K candidate scoring outputs corresponding to any reward reinforcement training sample through the supervised training scoring model.

[0061] Optionally, in other embodiments, a scoring candidate output may also include a thought chain, etc.; the present invention does not limit this.

[0062] S307, determine the reward selection output result pair corresponding to each reward reinforcement training sample from the K score candidate output results corresponding to each reward reinforcement training sample; wherein, the reward selection output result pair corresponding to a reward reinforcement training sample includes the preferred output result and the inferior output result corresponding to the corresponding reward reinforcement training sample.

[0063] Optionally, the preferred output result can also be called the first output result, and the inferior output result can also be called the second output result. The first output result is better than the second output result.

[0064] Optionally, for any reward reinforcement training sample in the target reward reinforcement training sample set, the electronic device can determine the comprehensive reward of each of the K candidate scoring outputs corresponding to any reward reinforcement training sample; and based on the comprehensive reward of each candidate scoring output, determine the reward screening output result pair corresponding to any reward reinforcement training sample from the K candidate scoring outputs corresponding to any reward reinforcement training sample; wherein, the preferred output result corresponding to any reward reinforcement training sample is the candidate scoring output with the largest comprehensive reward among the K candidate scoring outputs corresponding to any reward reinforcement training sample, and the unsuitable output result corresponding to any reward reinforcement training sample is the candidate scoring output with the smallest comprehensive reward among the K candidate scoring outputs corresponding to any reward reinforcement training sample.

[0065] Optionally, to ensure that the reward is calculable and repeatable, a deterministic validator can be set to verify and score the scoring candidate output results. Preferably, the deterministic validator may include structural verification and / or score consistency verification. Structural verification can be used to confirm that the scoring candidate output results can be stably parsed downstream. Optionally, the deterministic validator can be set according to experience or actual needs, and this embodiment of the invention does not limit this. Accordingly, when determining the comprehensive reward of each of the K candidate output results corresponding to any reward-enhanced training sample, the electronic device can traverse each of the K candidate output results corresponding to any reward-enhanced training sample and take the currently traversed candidate output result as the current candidate output result; and determine whether the current candidate output result meets the preset output verification rules, and when the current candidate output result meets the preset output verification rules, take the first structural reward as the structural reward of the current candidate output result; when the current candidate output result does not meet the preset output verification rules, take the second structural reward as the structural reward of the current candidate output result, and the first structural reward is greater than the second structural reward; furthermore, the comprehensive reward of the current candidate output result can be determined based on the structural reward of the current candidate output result; after traversing each of the K candidate output results corresponding to any reward-enhanced training sample, the comprehensive reward of each of the K candidate output results corresponding to any reward-enhanced training sample is obtained. Optionally, the preset output verification rules may include a preset structure; optionally, the preset output verification rules may be set based on experience or based on actual needs, and this embodiment of the invention does not limit this. For example, the preset output verification rules can be used to verify whether the scoring candidate output results meet the preset structure, whether they contain necessary fields, and whether the predicted scores are compliant, etc. Optionally, the electronic device may also be equipped with a rule verification function, which can be used to verify whether the preset output verification rules are met, that is, to determine whether the current scoring candidate output results meet the preset output verification rules.

[0066] Optionally, both the first structural reward and the second structural reward can be set according to experience or actual needs, and this embodiment of the invention does not limit this; for example, the first structural reward can be 1, the second structural reward can be 0, and so on. For example, assume that the rule verification function can be represented as V fmt If the current candidate output is the kth candidate output among the k candidate outputs corresponding to any reward-enhanced training sample, the electronic device can use Formula 2.1 to determine the structural reward of the current candidate output: Equation 2.1 Where, r fmt(k) The structural reward z can represent the output result of the k-th rating candidate (i.e., the current rating candidate output result). o (k) This can represent the output result of the k-th scoring candidate; that is, the rule verification function can be used to determine whether the current scoring candidate output result meets the preset output verification rule. If it meets the rule, the rule verification function output can be true (i.e., true), and the structural reward can be 1. If it does not meet the rule, the rule verification function output can be false, which is not equal to true, and the structural reward can be 0.

[0067] Optionally, the electronic device may use the structural reward of the current scoring candidate output result as the comprehensive reward of the current scoring candidate output result; or, the electronic device may also perform a score consistency check on the predicted score in the current scoring candidate output result to obtain the accuracy reward of the current scoring candidate output result, and determine the comprehensive reward of the current scoring candidate output result based on the structural reward and accuracy reward of the current scoring candidate output result; or, the electronic device may also determine whether the structural reward of the current scoring candidate output result is zero. If the structural reward of the current scoring candidate output result is zero, the structural reward of the current scoring candidate output result is used as the comprehensive reward of the current scoring candidate output result; if the structural reward of the current scoring candidate output result is not zero, the comprehensive reward of the current scoring candidate output result is determined based on the structural reward and accuracy reward of the current scoring candidate output result, and so on; the embodiments of the present invention do not limit this.

[0068] Optionally, when performing score consistency verification on the predicted scores in the current scoring candidate output results to obtain the accuracy reward of the current scoring candidate output results, the electronic device can obtain the score label corresponding to the current scoring candidate output results (here, the score label of any reward reinforcement training sample), and can use the score label corresponding to the current scoring candidate output results to perform score consistency verification on the predicted scores in the current scoring candidate output results to obtain the accuracy reward of the current scoring candidate output results. Optionally, the electronic device can use the difference between the score label corresponding to the current scoring candidate output results and the predicted scores in the current scoring candidate output results as the proximity difference, and use this proximity difference and the number of score levels in multiple score levels to calculate the accuracy reward of the current scoring candidate output results; for example, taking the current scoring candidate output results as the k-th scoring candidate output results as an example, the electronic device can use Formula 2.2 to calculate the accuracy reward of the current scoring candidate output results: Equation 2.2 Where, r acc (k) d can represent the accuracy reward of the current candidate output (i.e., the kth candidate output),(k) The value can represent the difference in closeness, where N represents the number of score levels in multiple score ranges; and the difference in closeness d represents the value of closeness. (k) Can be |y o (k) -y|,y o (k) y can represent the predicted score in the current score candidate output result, and y can represent the score label corresponding to the current score candidate output result.

[0069] Optionally, to avoid awarding higher rewards to structurally unqualified outputs, the accuracy reward only takes effect when the structural verification passes, i.e., only when the current candidate output meets the preset output verification rules. Based on this, the accuracy reward of the current candidate output is effective when the structural reward is not zero; the accuracy reward is ineffective when the structural reward is zero, thus setting the overall reward to zero. For example, taking the kth candidate output as an example, the electronic device can use Formula 2.3 to determine the overall reward of the current candidate output based on its structural and accuracy rewards: Equation 2.3 Where α and β can be weighting coefficients, r (k) This can represent the comprehensive reward of the current scoring candidate output result. Optionally, the weighting coefficients can be set according to experience or actual needs, and this embodiment of the invention does not limit this.

[0070] S308. Based on the reward filtering output pairs corresponding to each reward reinforcement training sample, determine the reinforcement training loss value of the supervised training scoring model.

[0071] Optionally, the electronic device can determine the set of reward reinforcement training samples to be used from the set of target reward reinforcement training samples based on the comprehensive reward of the preferred output results and undesired output results corresponding to each reward reinforcement training sample; and can determine the reinforcement training loss value of the supervised training scoring model based on the reward screening output result pairs corresponding to each reward reinforcement training sample to be used in the set of reward reinforcement training samples to be used.

[0072] Optionally, for any reward reinforcement training sample in the target reward reinforcement training sample set, if the comprehensive reward of the optimal output result corresponding to any reward reinforcement training sample is the same as the comprehensive reward of the inferior output result of any reward reinforcement training sample, then the reward reinforcement training sample will not be added to the reward reinforcement training sample set to be used; if the comprehensive reward of the optimal output result corresponding to any reward reinforcement training sample is different from the comprehensive reward of the inferior output result of any reward reinforcement training sample, then the reward reinforcement training sample will be added to the reward reinforcement training sample set to be used, so as to determine the reward reinforcement training sample set to be used from the target reward reinforcement training sample set.

[0073] Optionally, when determining the reinforcement training loss value of the supervised-trained scoring model based on the reward selection output pairs corresponding to each reinforcement training sample in the set of reinforcement training samples to be used, the DAPO (Decoupled Clip and Dynamic sAmpling Policy Optimization) algorithm can be used. This algorithm determines the reinforcement training loss value of the supervised-trained scoring model based on the reward selection output pairs corresponding to each reinforcement training sample in the set of reinforcement training samples to be used. Based on this, embodiments of the present invention can use algorithms such as DAPO to update model parameters, such as... Figure 4 As shown. It should be noted that the embodiments of the present invention do not limit the hyperparameters in the DAPO algorithm; that is, the hyperparameters in the DAPO algorithm can be set according to experience or actual needs. Optionally, after traversing each reward-enhanced training sample in the target reward-enhanced training sample set, if the number of unused reward-enhanced training samples in the unused reward-enhanced training sample set reaches the specified unused number, then sampling can continue to be performed from the initial reward-enhanced training sample set to add to the target reward-enhanced training sample set, and it can continue to determine whether the newly added samples in the target reward-enhanced training sample set support being added to the unused reward-enhanced training sample set, until the number of unused reward-enhanced training samples in the unused reward-enhanced training sample set reaches the specified unused number; optionally, the specified unused number can be set according to experience or actual needs, and the embodiments of the present invention do not limit this.

[0074] For example, when using the DAPO algorithm to calculate the model loss value to achieve model training via the DAPO algorithm, for any training sample in the set of training samples to be reinforced with rewards, the advantage value of each word in the preferred output result corresponding to any training sample to be reinforced with rewards can be determined, as well as the advantage value of each word in the undesired output result corresponding to any training sample to be reinforced with rewards can be determined; furthermore, the importance sampling ratio of each word in the preferred output result corresponding to any training sample to be reinforced with rewards can be calculated, and the importance sampling ratio of each word in the preferred output result corresponding to any training sample to be reinforced with rewards can be determined. The importance sampling ratio of each word is decoupled and pruned to obtain the decoupled pruned result of each word in the optimal output result corresponding to any training sample to be reinforced by the reward (which can encourage the model to learn the optimal output result); correspondingly, the importance sampling ratio of each word in the inferior output result corresponding to any training sample to be reinforced by the reward can be calculated, and the importance sampling ratio of each word in the inferior output result corresponding to any training sample to be reinforced by the reward can be tightened and pruned to obtain the tightened pruned result of each word in the inferior output result corresponding to any training sample to be reinforced by the reward (which can limit the probability of words in the inferior output result). Based on this, the loss value under the optimal output result corresponding to any reward-enhanced training sample can be calculated based on the advantage value, importance sampling ratio, and decoupling pruning result of each word in the optimal output result corresponding to any reward-enhanced training sample. Similarly, the loss value under the inferior output result corresponding to any reward-enhanced training sample can be calculated based on the advantage value, importance sampling ratio, and tightening pruning result of each word in the inferior output result corresponding to any reward-enhanced training sample. The sum of the loss values ​​under the optimal and inferior output results corresponding to any reward-enhanced training sample is then used as the loss value under any reward-enhanced training sample. Based on this, the sum of the loss values ​​under each reward-enhanced training sample can be used as the reinforcement training loss value of the supervised training scoring model, and so on. For example, taking the optimal output result corresponding to any training sample to be used for reward reinforcement as an example, for any word in the optimal output result corresponding to any training sample to be used for reward reinforcement, the minimum product under any word can be determined from the product between the advantage value and importance sampling ratio of any word and the product between the advantage value and decoupling pruning result of any word. The negative number of the sum of the minimum products under each word in the optimal output result corresponding to any training sample to be used for reward reinforcement is taken as the loss value under the optimal output result corresponding to any training sample to be used for reward reinforcement, and so on.Optionally, in other embodiments, the loss value under any reward-enhanced training sample can be divided by the length of the reward selection output pair corresponding to any reward-enhanced training sample to update the loss value under any reward-enhanced training sample. The length of the reward selection output pair corresponding to any reward-enhanced training sample can be the sum of the number of lexical units in the preferred output result and the number of lexical units in the undesired output result corresponding to any reward-enhanced training sample, etc. The present invention does not limit the specific method for determining the reinforcement training loss value.

[0075] Optionally, the electronic device can also add each unused reward-enhanced training sample from the unused reward-enhanced training sample set to the target cache queue. The unused reward-enhanced training sample set includes all reward-enhanced training samples from the target reward-enhanced training sample set, excluding the set of reward-enhanced training samples to be used. The reward-enhanced training samples in the target cache queue are used for priority sampling in the next iteration of model training to determine the target reward-enhanced training sample set in the next iteration. Based on this, embodiments of the present invention can introduce a DAPO method with empirical replacement to prioritize sampling the target cache queue in the next iteration, thus avoiding the traditional DAPO algorithm from discarding redundant samples in each sampling, thereby effectively accelerating training efficiency.

[0076] S309. Optimize the model parameters in the supervised training scoring model in the direction of reducing the reinforcement training loss value to obtain the initial reinforcement training model; and determine the target scoring model based on the initial reinforcement training model.

[0077] Optionally, when determining the target scoring model based on the initial reinforcement training model, the electronic device can iteratively execute the aforementioned determination of the target reward reinforcement training sample set to continue training the initial reinforcement training model until the reinforcement training convergence condition is met (such as the number of iterations reaching a preset reinforcement training number threshold, or the reinforcement training loss value being less than a preset reinforcement training loss threshold, etc.), thereby using the scoring model that meets the reinforcement training convergence condition as the target scoring model. Optionally, both the preset reinforcement training number threshold and the preset reinforcement training loss threshold can be set according to experience or actual needs, and this embodiment of the invention does not limit this.

[0078] Based on this, electronic devices can use the DAPO algorithm for parameter optimization, i.e., parameter update. In this case, the same reward can be used to reinforce the K candidate output results corresponding to the training sample. The reward-filtered output result pair is selected based on the comprehensive reward, and the scoring model is optimized for preference without introducing length penalty, making the scoring model more inclined to generate structured scoring results with higher comprehensive rewards. Based on this, through the above DAPO update, the scoring model can be continuously calibrated under constraints such as a large number of score labels, thereby achieving calibration between the mapping relationship of "criterion clause - score boundary - human scale", which can improve the stability and consistency of adjacent classification.

[0079] It should be understood that, in the embodiment of the present invention, a verifiable reward (RLVR) is introduced in the reward reinforcement training stage (also known as the reinforcement training stage), and preference optimization is performed using DAPO with experience replacement: a computable comprehensive reward is constructed through structure verification and score consistency verification, and a preference signal is formed by the reward difference of multiple candidate outputs under the same input to update the scoring model. This allows the model to maintain the stability of the structured output while continuously aligning the score boundary to the human scoring scale, thereby improving the discrimination stability of adjacent tiers and reducing score fluctuations.

[0080] Optionally, in other embodiments, the electronic device may also employ other preference optimization or policy optimization methods to determine the reinforcement training loss value of the supervised-trained scoring model based on the reward selection output pairs corresponding to each reinforcement training sample in the set of reinforcement training samples to be used, thereby updating the parameters of the scoring model, i.e., performing reward reinforcement training on the scoring model. For example, model parameters can also be updated using algorithms such as DPO (Direct Preference Optimization) and ORPO (Odds Ratio Preference Optimization), etc.; this embodiment of the invention does not limit this. Based on this, the electronic device may employ a target preference optimization algorithm to determine the reinforcement training loss value of the supervised-trained scoring model based on the reward selection output pairs corresponding to each reinforcement training sample in the set of reinforcement training samples to be used, thereby optimizing the model parameters of the scoring model. The target preference optimization algorithm can be any policy optimization method based on preference pairs, i.e., any preference optimization method based on verifiable rewards, such as DAPO, DPO, and ORPO.

[0081] In summary, this invention proposes a scoring model training method based on question scoring criteria, enabling the scoring process to be stably executed under the constraints of question-related criteria, and ensuring that the predicted scores of the scoring model are consistent with and reproducible human scoring standards. Specifically, this invention effectively solves the problems of unclear criterion boundaries and score drift caused by reliance on subjective interpretation in "scoring based on fuzzy criteria." It addresses how to transform the scoring criteria from "fuzzy grading descriptions" into a stable criterion system, ensuring that scorers maintain consistent scoring results for the same answer at different times and under different subjects. Furthermore, stable scoring can be achieved through clear question scoring criteria and fixed structured output requirements, effectively reducing randomness and sensitivity to prompts. In addition, this invention can utilize question scoring criteria and human scoring data to form training samples, enabling targeted calibration and alignment of the scoring model, allowing the scoring model to learn a stable mapping relationship from "criterion text + question / answer" to "human-consistent score." Based on this, the embodiments of the present invention can use the "1-N score criterion related to the question" as a unified scoring standard, and take the question, candidate answer and corresponding question scoring criteria as the model input. By constructing supervised training samples containing score labels and scoring reason labels, supervised learning is performed on the scoring model, and verifiable reward reinforcement training can be further performed to enable the scoring model to obtain a stable "criterion-score" mapping ability, thereby obtaining more reliable scoring results.

[0082] In this embodiment of the invention, after obtaining a set of supervised training samples, the reverse thinking chain synthesis is performed on each supervised training sample in the set to obtain a thinking chain label for each supervised training sample. The initial scoring model can then be invoked, and the predicted probability data for each supervised training sample can be determined based on the model training input data from each supervised training sample. Based on this, the model loss value of the initial scoring model can be calculated using the predicted probability data and model training labels of each supervised training sample; wherein, the model training label of a supervised training sample includes the training sample label and the thinking chain label of the corresponding supervised training sample. Accordingly, the model parameters in the initial scoring model can be optimized in the direction of reducing the model loss value of the initial scoring model to obtain an optimized initial scoring model; and based on the optimized initial scoring model, the supervised training scoring model is determined. Furthermore, a target reward reinforcement training sample set can be determined, and the supervised-trained scoring model can be invoked to determine K candidate scoring outputs corresponding to each reward reinforcement training sample in the target reward reinforcement training sample set. Additionally, reward selection output pairs corresponding to each reward reinforcement training sample can be determined from the K candidate scoring outputs corresponding to each reward reinforcement training sample. Each reward selection output pair corresponding to a reward reinforcement training sample includes the preferred output and the undesired output for that sample. Based on these reward selection output pairs, the reinforcement training loss value of the supervised-trained scoring model can be determined. Then, the model parameters in the supervised-trained scoring model are optimized in the direction of reducing the reinforcement training loss value to obtain the initial reinforcement training model. Based on the initial reinforcement training model, the target scoring model is determined. Therefore, this embodiment of the invention can introduce verifiable reward reinforcement training into the scoring model on the basis of supervised training, thereby further reducing the deviation between the scoring model and the human scoring scale, and improving the stability of the predicted score under the same input, thus further improving the model performance of the scoring model.

[0083] Based on the description of the relevant embodiments of the scoring model training method above, this invention also proposes a scoring model training device, which can be a computer program (including program code) running in an electronic device; such as Figure 5 As shown, the scoring model training device may include an acquisition unit 501 and a processing unit 502. The scoring model training device can perform... Figure 1 or Figure 3 The scoring model training method shown, i.e., the scoring model training device, can run the above-mentioned unit: The acquisition unit 501 is used to acquire a set of supervised training samples. A supervised training sample includes a model training input data and a corresponding supervised training sample label. A model training input data includes at least one of the following: a question, a candidate answer, and a question scoring criterion. A training sample label includes at least one of the following: a score label and a scoring reason label. A question scoring criterion includes the scoring criteria for the corresponding question in each score level among multiple score levels. Processing unit 502 is used to perform reverse thinking chain synthesis on each supervised training sample in the supervised training sample set to obtain the thinking chain label of each supervised training sample. The processing unit 502 is further configured to call the initial scoring model, and determine the predicted probability data of each supervised training sample based on the model training input data in each supervised training sample; The processing unit 502 is further configured to calculate the model loss value of the initial scoring model based on the predicted probability data and model training labels of each supervised training sample; wherein, the model training label of a supervised training sample includes the training sample label and the thinking chain label of the corresponding supervised training sample. The processing unit 502 is further configured to optimize the model parameters in the initial scoring model in the direction of reducing the model loss value of the initial scoring model, so as to obtain an optimized initial scoring model; and based on the optimized initial scoring model, determine the target scoring model, wherein the predicted score output by a scoring model belongs to the multiple score levels.

[0084] In one implementation, the predicted probability data of a supervised training sample includes the conditional probability of each word in the model training labels of the corresponding supervised training sample; when the processing unit 502 calculates the model loss value of the initial scoring model based on the predicted probability data and model training labels of each supervised training sample, it may specifically be used to: The conditional probability of each word in the model training label of each supervised training sample is determined from the predicted probability data of each supervised training sample. The conditional probability of each word in the model training label is used from each supervised training sample to calculate the model loss value of the initial scoring model.

[0085] In another implementation, when determining the target scoring model based on the optimized initial scoring model, the processing unit 502 may specifically be used to: Based on the optimized initial scoring model, the supervised training scoring model is determined. A target reward reinforcement training sample set is determined, and the supervised training scoring model is invoked to determine K candidate scoring outputs corresponding to each reward reinforcement training sample in the target reward reinforcement training sample set, where K is a positive integer; wherein, a candidate scoring output includes a predicted score and / or a scoring reason; From the K candidate output results corresponding to each reward reinforcement training sample, the reward selection output result pair corresponding to each reward reinforcement training sample is determined; wherein, the reward selection output result pair corresponding to a reward reinforcement training sample includes the preferred output result and the unpreferred output result corresponding to the corresponding reward reinforcement training sample; Based on the reward filtering output pairs corresponding to each reward reinforcement training sample, the reinforcement training loss value of the supervised training scoring model is determined. The model parameters in the supervised training scoring model are optimized in the direction of reducing the reinforcement training loss value to obtain the initial reinforcement training model; and the target scoring model is determined based on the initial reinforcement training model.

[0086] In another embodiment, when the processing unit 502 determines the reward selection output result pair corresponding to each reward reinforcement training sample from the K candidate score output results corresponding to each reward reinforcement training sample, it may specifically be used to: For any reward enhancement training sample in the target reward enhancement training sample set, determine the comprehensive reward of each of the K candidate output results corresponding to the reward enhancement training sample. Based on the comprehensive reward of each of the rating candidate output results, the reward screening output result pair corresponding to any reward reinforcement training sample is determined from the K rating candidate output results corresponding to any reward reinforcement training sample; Wherein, the preferred output result corresponding to any reward-enhanced training sample is the candidate output result with the largest comprehensive reward among the K candidate output results corresponding to any reward-enhanced training sample, and the unsuitable output result corresponding to any reward-enhanced training sample is the candidate output result with the smallest comprehensive reward among the K candidate output results corresponding to any reward-enhanced training sample.

[0087] In another implementation, when determining the comprehensive reward of each of the K candidate output results corresponding to any reward-enhanced training sample, the processing unit 502 may specifically be used to: Iterate through each of the K candidate output results corresponding to any reward-enhanced training sample, and take the currently iterated candidate output result as the current candidate output result. Determine whether the current scoring candidate output result meets the preset output verification rules. If the current scoring candidate output result meets the preset output verification rules, use the first structural reward as the structural reward of the current scoring candidate output result. If the current scoring candidate output result does not meet the preset output verification rules, use the second structural reward as the structural reward of the current scoring candidate output result. The first structural reward is greater than the second structural reward. Based on the structural reward of the current candidate score output, determine the comprehensive reward of the current candidate score output; After traversing through all the candidate outputs of the K candidate outputs corresponding to any reward-enhanced training sample, the comprehensive reward of each candidate output is obtained.

[0088] In another implementation, when determining the reinforcement training loss value of the supervised training scoring model based on the reward filtering output result pairs corresponding to each reward reinforcement training sample, the processing unit 502 may specifically be used for: Based on the comprehensive reward of the preferred and unpreferred output results corresponding to each reward reinforcement training sample, a set of reward reinforcement training samples to be used is determined from the target reward reinforcement training sample set. Based on the reward filtering output pairs corresponding to each reward reinforcement training sample in the set of reward reinforcement training samples to be used, the reinforcement training loss value of the supervised training scoring model is determined. Processing unit 502 can also be used for: Each unused reward-enhanced training sample in the unused reward-enhanced training sample set is added to the target cache queue; wherein, the unused reward-enhanced training sample set includes all reward-enhanced training samples in the target reward-enhanced training sample set except for the reward-enhanced training sample set to be used, and the reward-enhanced training samples in the target cache queue are used for priority sampling in the next iteration of model training, so as to realize the determination of the target reward-enhanced training sample set in the next iteration.

[0089] In another embodiment, the acquisition unit 501 can also be used for: Obtain the data to be scored, which includes the target question and the answers to the target question to be scored; Processing unit 502 can also be used for: Based on the data to be scored, construct the input data for the target model; The target scoring model is invoked, and the scoring result of the answer to be scored is determined based on the input data of the target model. The scoring result includes at least one of the following: the predicted score of the answer to be scored and the scoring reason.

[0090] According to one embodiment of the present invention, Figure 5 Each unit in the scoring model training device shown can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effect of the embodiments of the present invention. The above units are based on logical function division. In practical applications, the function of one unit can be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of the present invention, any scoring model training device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0091] According to another embodiment of the present invention, it is possible to perform operations such as those described above by running on a general-purpose electronic device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). Figure 1 or Figure 3 The computer program (including program code) involved in each step of the corresponding method shown, to construct such... Figure 5 The diagram illustrates a scoring model training apparatus and a scoring model training method for implementing embodiments of the present invention. The computer program may be stored on, for example, a computer storage medium, loaded onto the aforementioned electronic device via the computer storage medium, and run therein.

[0092] Based on the description of the method and apparatus embodiments above, an exemplary embodiment of the present invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, which, when executed by the at least one processor, causes the electronic device to perform the method according to an embodiment of the present invention.

[0093] An exemplary embodiment of the present invention also provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0094] An exemplary embodiment of the present invention also provides a computer program product, including a computer program, wherein, when executed by a computer's processor, the computer program is used to cause the computer to perform a method according to an embodiment of the present invention.

[0095] refer to Figure 6 The present invention will now be described in the form of a structural block diagram of an electronic device 600 that can serve as a server or client of the present invention, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0096] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. The RAM 603 may also store various programs and data required for the operation of the electronic device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0097] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, output unit 607, storage unit 608, and communication unit 609. Input unit 606 can be any type of device capable of inputting information to electronic device 600. Input unit 606 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 607 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 608 may include, but is not limited to, disks and optical discs. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0098] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above. For example, in some embodiments, the scoring model training method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 600 via ROM 602 and / or communication unit 609. In some embodiments, the computing unit 601 can be configured to perform the scoring model training method by any other suitable means (e.g., by means of firmware).

[0099] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0100] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0101] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0102] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0103] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0104] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0105] Furthermore, it should be understood that the above-disclosed embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, any equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for training a scoring model, characterized in that, include: Obtain a set of supervised training samples. A supervised training sample includes a model training input data and a corresponding supervised training sample label. A model training input data includes at least one of the following: a question, a candidate answer, and a question scoring criterion. A training sample label includes at least one of the following: a score label and a scoring reason label. A question scoring criterion includes the scoring criteria for the corresponding question in each score range. Reverse thinking chain synthesis is performed on each supervised training sample in the supervised training sample set to obtain the thinking chain label of each supervised training sample; The initial scoring model is invoked, and the model training input data is determined based on the supervised training samples to determine the predicted probability data for each supervised training sample. Based on the predicted probability data and model training labels of each supervised training sample, the model loss value of the initial scoring model is calculated; wherein, the model training label of a supervised training sample includes the training sample label and the thought chain label of the corresponding supervised training sample. The model parameters in the initial scoring model are optimized in the direction of reducing the model loss value of the initial scoring model to obtain an optimized initial scoring model; and based on the optimized initial scoring model, a target scoring model is determined, wherein the predicted score output by a scoring model belongs to the multiple score levels.

2. The method according to claim 1, characterized in that, The predicted probability data of a supervised training sample includes the conditional probability of each word in the model training labels of the corresponding supervised training sample; the calculation of the model loss value of the initial scoring model based on the predicted probability data and model training labels of each supervised training sample includes: The conditional probability of each word in the model training label of each supervised training sample is determined from the predicted probability data of each supervised training sample. The conditional probability of each word in the model training label is used from the supervised training samples to calculate the model loss value of the initial scoring model.

3. The method according to claim 1 or 2, characterized in that, The process of determining the target scoring model based on the optimized initial scoring model includes: Based on the optimized initial scoring model, the supervised training scoring model is determined. A target reward reinforcement training sample set is determined, and the supervised training scoring model is invoked to determine K candidate scoring outputs corresponding to each reward reinforcement training sample in the target reward reinforcement training sample set, where K is a positive integer; wherein, a candidate scoring output includes a predicted score and / or a scoring reason; From the K candidate output results corresponding to each reward reinforcement training sample, the reward selection output result pair corresponding to each reward reinforcement training sample is determined; wherein, the reward selection output result pair corresponding to a reward reinforcement training sample includes the preferred output result and the unpreferred output result corresponding to the corresponding reward reinforcement training sample; Based on the reward filtering output pairs corresponding to each reward reinforcement training sample, the reinforcement training loss value of the supervised training scoring model is determined. The model parameters in the supervised training scoring model are optimized in the direction of reducing the reinforcement training loss value to obtain the initial reinforcement training model; and the target scoring model is determined based on the initial reinforcement training model.

4. The method according to claim 3, characterized in that, The step of determining the reward selection output pair corresponding to each reward enhancement training sample from the K candidate score outputs corresponding to each reward enhancement training sample includes: For any reward enhancement training sample in the target reward enhancement training sample set, determine the comprehensive reward of each of the K candidate output results corresponding to the reward enhancement training sample. Based on the comprehensive reward of each of the rating candidate output results, the reward screening output result pair corresponding to any reward reinforcement training sample is determined from the K rating candidate output results corresponding to any reward reinforcement training sample; Wherein, the preferred output result corresponding to any reward-enhanced training sample is the candidate output result with the largest comprehensive reward among the K candidate output results corresponding to any reward-enhanced training sample, and the unsuitable output result corresponding to any reward-enhanced training sample is the candidate output result with the smallest comprehensive reward among the K candidate output results corresponding to any reward-enhanced training sample.

5. The method according to claim 4, characterized in that, The step of determining the comprehensive reward of each of the K candidate output results corresponding to any reward-enhanced training sample includes: Iterate through each of the K candidate output results corresponding to any reward-enhanced training sample, and take the currently iterated candidate output result as the current candidate output result. Determine whether the current scoring candidate output result meets the preset output verification rules. If the current scoring candidate output result meets the preset output verification rules, use the first structural reward as the structural reward of the current scoring candidate output result. If the current scoring candidate output result does not meet the preset output verification rules, use the second structural reward as the structural reward of the current scoring candidate output result. The first structural reward is greater than the second structural reward. Based on the structural reward of the current candidate score output, determine the comprehensive reward of the current candidate score output; After traversing through all the K candidate output results corresponding to any reward-enhanced training sample, the comprehensive reward of each candidate output result in the K candidate output results corresponding to any reward-enhanced training sample is obtained.

6. The method according to claim 3, characterized in that, The step of determining the reinforcement training loss value of the supervised training scoring model based on the reward filtering output pairs corresponding to each reward reinforcement training sample includes: Based on the comprehensive reward of the preferred and unpreferred output results corresponding to each reward reinforcement training sample, a set of reward reinforcement training samples to be used is determined from the target reward reinforcement training sample set. Based on the reward filtering output pairs corresponding to each reward reinforcement training sample in the set of reward reinforcement training samples to be used, the reinforcement training loss value of the supervised training scoring model is determined. The method further includes: Each unused reward-enhanced training sample in the unused reward-enhanced training sample set is added to the target cache queue; wherein, the unused reward-enhanced training sample set includes all reward-enhanced training samples in the target reward-enhanced training sample set except for the reward-enhanced training sample set to be used, and the reward-enhanced training samples in the target cache queue are used for priority sampling in the next iteration of model training, so as to realize the determination of the target reward-enhanced training sample set in the next iteration.

7. The method according to claim 1 or 2, characterized in that, The method further includes: Obtain the data to be scored, which includes the target question and the answers to the target question to be scored; Based on the data to be scored, construct the input data for the target model; The target scoring model is invoked, and the scoring result of the answer to be scored is determined based on the input data of the target model. The scoring result includes at least one of the following: the predicted score of the answer to be scored and the scoring reason.

8. A scoring model training device, characterized in that, The device includes: The acquisition unit is used to acquire a set of supervised training samples. A supervised training sample includes a model training input data and a corresponding supervised training sample label. A model training input data includes at least one of the following: a question, a candidate answer, and a question scoring criterion. A training sample label includes at least one of the following: a score label and a scoring reason label. A question scoring criterion includes the scoring criteria for the corresponding question in each score level among multiple score levels. The processing unit is used to perform reverse thinking chain synthesis on each supervised training sample in the supervised training sample set to obtain the thinking chain label of each supervised training sample. The processing unit is also used to call the initial scoring model, and determine the predicted probability data of each supervised training sample based on the model training input data in each supervised training sample; The processing unit is further configured to calculate the model loss value of the initial scoring model based on the predicted probability data and model training labels of each supervised training sample; wherein, the model training label of a supervised training sample includes the training sample label and the thinking chain label of the corresponding supervised training sample. The processing unit is further configured to optimize the model parameters in the initial scoring model in the direction of reducing the model loss value of the initial scoring model, thereby obtaining an optimized initial scoring model; and based on the optimized initial scoring model, determine the target scoring model, wherein the predicted score output by a scoring model belongs to the multiple score levels.

9. An electronic device, characterized in that, include: processor; as well as Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

Images