Scoring method, apparatus, device, and computer medium

Abstract

The application discloses a scoring method and device, equipment and a computer medium. The method comprises the following steps: acquiring a plurality of groups of first evaluation characteristics of a plurality of to-be-scored test questions; acquiring a plurality of groups of second evaluation characteristics of a plurality of calibration test questions, a plurality of reference scores of the plurality of calibration test questions, and a plurality of initial predicted scores of the plurality of calibration test questions; sorting the plurality of groups of first evaluation characteristics and the plurality of groups of second evaluation characteristics to obtain a target sorting result; and determining a scoring result of the plurality of to-be-scored test questions based on the target sorting result, the plurality of reference scores, and the plurality of initial predicted scores. The method can improve the efficiency of scoring oral English in oral English evaluation.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent scoring, and particularly relates to a scoring method and device, equipment and a computer medium. BACKGROUND

[0002] With the development of education technology and the increasing emphasis on language ability in society, oral ability has become one of the important components for evaluating the comprehensive quality of students. In some examinations, oral evaluation has gradually become an indispensable link.

[0003] Compared with artificial scoring, in the face of a large group of examinees, machine scoring can not only quickly process a large amount of data and shorten the scoring period, but also can reduce the influence of subjective factors. With the gradual development of oral evaluation technology, oral artificial evaluation is gradually being replaced by machines. How to ensure the fairness and accuracy of machine scoring becomes more and more important. In order to ensure the accuracy of machine scoring, the process of first calibration and then machine scoring is introduced, that is, in an oral examination, a part of the data is selected, experts give artificial scores according to the scoring standard, and the machine learns from the part of the data with artificial labels and then evaluates the whole set of data.

[0004] The existing calibration scoring is generally divided into four stages. The first stage: a certain amount of examination data is selected, a general scoring scheme is used to output a general score and output the features for calculating the general score. The second stage: a certain amount of data is selected according to the general score and sent to experts for artificial scoring. The third stage: the features output in the first stage and the artificial score in the second stage are used to retrain the ranking model. The fourth stage: the new ranking model is replaced, and the whole set of data of the current examination is retested to output the machine score.

[0005] However, the existing technology is dependent on feature engineering, and the extracted evaluation features are regressed. The conventional calibration method basically adopts the feature engineering method, learns the related regression model according to the features and artificial scores of the calibration data. The conventional regression model may have the following problems:

[0006] Linear hypothesis problem: Many regression models assume a linear relationship between the score and the features, but in reality this relationship may be non-linear.

[0007] Overfitting / underfitting problem: If the number of training samples is insufficient or the features are not selected properly, the model may have overfitting (overfitting the training data, poor generalization ability) or underfitting (failure to capture patterns in the data) problems.

[0008] Complexity: Oral evaluation involves many factors, such as pronunciation, fluency, grammar, and vocabulary richness, and a single regression model may not be able to fully capture these complex factors.

[0009] Data imbalance: the sample data of the calibration may have an imbalance problem, and the sample number in some interval is too large, which is more likely to cause the model to fit better when training the sample data.

[0010] In summary, in the related art, the related oral evaluation scoring scheme has low efficiency. SUMMARY

[0011] The embodiments of the present application provide an implementation scheme different from the prior art to solve the technical problem of low efficiency of the related oral evaluation scoring scheme in the related art.

[0012] In a first aspect, the present application provides a scoring method, comprising:

[0013] obtaining a plurality of groups of first evaluation features of a plurality of to-be-scored test questions;

[0014] obtaining a plurality of groups of second evaluation features of a plurality of calibration test questions, a plurality of reference scores of the plurality of calibration test questions, and a plurality of initial predicted scores of the plurality of calibration test questions;

[0015] sorting the plurality of groups of first evaluation features and the plurality of groups of second evaluation features to obtain a target sorting result;

[0016] determining a scoring result of the plurality of to-be-scored test questions based on the target sorting result, the plurality of reference scores, and the plurality of initial predicted scores.

[0017] In a second aspect, the present application provides a scoring device, comprising:

[0018] an obtaining unit configured to obtain a plurality of groups of first evaluation features of a plurality of to-be-scored test questions;

[0019] the obtaining unit is configured to obtain a plurality of groups of second evaluation features of a plurality of calibration test questions, a plurality of reference scores of the plurality of calibration test questions, and a plurality of initial predicted scores of the plurality of calibration test questions;

[0020] a sorting unit configured to sort the plurality of groups of first evaluation features and the plurality of groups of second evaluation features to obtain a target sorting result;

[0021] a determining unit configured to determine a scoring result of the plurality of to-be-scored test questions based on the target sorting result, the plurality of reference scores, and the plurality of initial predicted scores.

[0022] In a third aspect, the present application provides an electronic device, comprising:

[0023] a processor; and

[0024] a memory configured to store executable instructions of the processor;

[0025] The processor is configured to execute the executable instructions to perform any of the methods of the first aspect or any of the possible implementation manners of the first aspect.

[0026] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement any of the methods of the first aspect or any of the possible implementation manners of the first aspect.

[0027] In a fifth aspect, an embodiment of the present application provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement any of the methods of the first aspect or any of the possible implementation manners of the first aspect.

[0028] The application provides a plurality of groups of first evaluation features of a plurality of to-be-scored test questions; a plurality of groups of second evaluation features of a plurality of calibration test questions, a plurality of reference scores of the plurality of calibration test questions, and a plurality of initial predicted scores of the plurality of calibration test questions; sorting the plurality of groups of first evaluation features and the plurality of groups of second evaluation features to obtain a target sorting result; and determining a score result of the plurality of to-be-scored test questions based on the target sorting result, the plurality of reference scores, and the plurality of initial predicted scores. The scheme of scoring test questions based on feature sorting can avoid a series of problems caused by scoring test questions based on a regression model, and improves the efficiency of scoring oral tests in oral test evaluation. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor. In the drawings:

[0030] Figure 1 The flowchart of the scoring method provided by an embodiment of the present application is shown in the figure;

[0031] Figure 2a The flowchart of the scoring method provided by an embodiment of the present application is shown in the figure;

[0032] Figure 2b The schematic diagram of the process of determining the calibration test question set provided by an embodiment of the present application is shown in the figure;

[0033] Figure 3 The structural schematic diagram of the scoring device provided by an embodiment of the present application is shown in the figure;

[0034] Figure 4A structural schematic diagram of an electronic device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0035] Embodiments of the present application are described in detail below with reference to examples illustrated in the accompanying drawings. The embodiments described below by reference to the drawings are exemplary and are intended to explain the present application, and should not be understood as limiting the present application.

[0036] The terms "first" and "second" and the like in the specification, claims and drawings of the embodiments of the present application are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented, for example, in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to such a process, method, product or device.

[0037] First, the following explains and describes some of the terms in the embodiments of the present application, so as to facilitate understanding by those skilled in the art.

[0038] In machine learning and statistics, overfitting and underfitting are two common concepts that describe the performance of a model on training data and unseen data. Overfitting refers to a model that performs too well on training data, capturing noise and random fluctuations in the training data rather than underlying patterns in the data. As a result, the model's performance tends to be poor when it faces new, unseen data. Underfitting refers to a model that performs poorly on training data, failing to capture underlying patterns in the data. As a result, the model's performance is also poor when it faces new, unseen data.

[0039] Text-based features and BERT-based features each have their own unique nature and importance in the field of natural language processing (NLP).

[0040] Text-based features refer to basic units or attributes used to represent text data, which play a crucial role in text analysis and processing. Common text-based features include:

[0041] Bag of Words: Treats text as an unordered collection of words, without considering the order and context of the words, and only counts the frequency of word occurrences.

[0042] TF-IDF (Term Frequency-Inverse Document Frequency): Based on the bag-of-words model, it considers the frequency of a word in a document and the inverse document frequency in the document collection to evaluate the importance of a word in a document.

[0043] Word Embedding: Representing words as vectors in a high-dimensional space that can capture semantic and syntactic relationships between words. Common word embedding models include Word2Vec, GloVe, etc.

[0044] N-gram: Considering the sequence of N consecutive words in the text to capture local contextual information in the text.

[0045] Syntactic features: Such as part-of-speech tagging (POS Tagging), named entity recognition (NER), etc., which provide grammatical and semantic role information of words in the text.

[0046] BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model based on the Transformer architecture, which has significant features and advantages in the field of natural language processing. BERT-based features mainly include the following aspects:

[0047] Bidirectional encoding capability: BERT uses a bidirectional Transformer encoder to encode text, which can simultaneously utilize context information and capture richer semantic relationships. Deep language representation: BERT learns deep language representation knowledge through pre-training on large-scale text data, which can capture different levels of language information from shallow syntactic features to deep semantic features.

[0048] Pre-training tasks: BERT adopts two pre-training tasks, namely Masked Language Model (MLM) and Next Sentence Prediction (NSP), which help the model understand the complex semantic relationships of words in sentences and sentence-level information.

[0049] Wide adaptability: BERT model has wide adaptability and can be applied to various NLP tasks through fine-tuning, such as text classification, sequence labeling, question answering system, named entity recognition, etc.

[0050] Parameter sharing and efficiency: During fine-tuning, most of the parameters of the BERT model are shared, and only the output layer parameters related to the specific task need to be trained. This way not only saves computing resources but also improves the generalization ability of the model. At the same time, since BERT has learned rich language representation knowledge in the pre-training stage, it can converge faster and achieve better performance in the fine-tuning stage.

[0051] RankSVM is a ranking learning algorithm based on support vector machine (SVM), mainly used to solve ranking problems, especially in the field of information retrieval. RankSVM determines a ranking model by learning a series of object pairs, aiming to minimize ranking errors. It uses linear programming in dual space to optimize the objective function, making it possible to handle large-scale datasets. This algorithm takes into account the partial order information between different object pairs, striving to maintain these relative relationships to improve the accuracy of ranking. RankSVM treats the ranking problem as an ordered multi-class problem. From the core idea, the goal of RankSVM is to find a hyperplane to distinguish different ranking objects. These objects are represented by feature vectors, while the hyperplane is composed of coefficient weights. In specific implementation, RankSVM adjusts the hyperplane by minimizing the cost function, which imposes a penalty on the occurrence of ranking errors.

[0052] With the development of educational technology and the increasing emphasis on language ability in society, oral ability has become one of the important components of evaluating students' comprehensive quality. In some exams, oral evaluation has gradually become an indispensable part.

[0053] Compared with manual scoring, in the face of a large group of examinees, machine scoring not only can quickly process a large amount of data and shorten the scoring period, but also can reduce the influence of subjective factors. With the gradual development of oral evaluation technology, oral artificial evaluation is gradually being replaced by machines. How to ensure the fairness and accuracy of machine scoring becomes increasingly important. In order to ensure the accuracy of machine scoring, the process of first labeling and then machine scoring is introduced, that is, in one oral exam, a part of the data is selected, and experts give manual scores according to the scoring standard, and the machine learns from the data with manual labels and then evaluates the whole set of data.

[0054] Existing labeling scoring generally consists of four stages. The first stage: select a certain amount of test data, use the general scoring scheme to output the general score and output the features for calculating the general score. The second stage: select a certain amount of data according to the general score and send it to the expert for manual scoring. The third stage: use the features output in the first stage and the manual score in the second stage to retrain the ranking model. The fourth stage: replace the new ranking model and retest the whole set of data in the current exam to output the machine score.

[0055] However, the existing technologies are all dependent on feature engineering, and the evaluation features extracted are regressed. The conventional calibration methods basically adopt the feature engineering method, and learn the relevant regression model according to the features of the calibration data and manual division. The conventional regression model may have the following problems:

[0056] Linear hypothesis problem: Many regression models assume a linear relationship between scores and features, but in reality, such a relationship may be nonlinear.

[0057] Overfitting / underfitting problem: If the number of training samples is insufficient or the feature selection is improper, the model may have overfitting (overfitting the training data, poor generalization ability) or underfitting (failure to capture the patterns in the data) problems.

[0058] Complexity: Spoken language evaluation involves many factors, such as pronunciation, fluency, grammar, and vocabulary richness, and a single regression model may not be able to fully capture these complex factors.

[0059] Data imbalance: The sample data for calibration may have an imbalance problem, and in some intervals, the number of samples is biased, which is more likely to cause the model to fit better for the data with more samples and not fit well for the data with fewer samples during model training.

[0060] In summary, in the related technology, the related spoken language evaluation scoring scheme has low efficiency.

[0061] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0062] Figure 1 A flowchart of a scoring method provided by an exemplary embodiment of the present application is shown. The scoring method can include: obtaining a plurality of original test questions; selecting part of the original test questions of a certain type as a plurality of initial test questions from the plurality of original test questions; inputting the plurality of initial test questions into an evaluation unit to obtain evaluation results, wherein the evaluation results include evaluation features and initial predicted scores of each initial test question. According to the initial predicted scores, a calibration test question set and a test question set to be scored are determined from the plurality of initial test questions; the calibration test question set is sent to a target device, so that relevant personnel manually score the calibration test question set; based on the manual scores of the calibration test question set and the evaluation results of each calibration test question in the calibration test question set, an initial ranking model is trained to obtain a target ranking model; the target ranking model is used to score each test question in the test question set to be scored. Further, the total score of the test question set to be scored can also be calculated.

[0063] Optionally, the original test question in the present application refers to the test question itself and the answer to the test question given by the testee.

[0064] Optionally, the test question in the present application refers to the test question of the oral test.

[0065] Figure 2a A flowchart of a scoring method provided by an exemplary embodiment of the present application is shown. The subject of the method can be any electronic device, and the method includes at least the following steps S201-S204:

[0066] S201, obtaining a plurality of sets of first evaluation features of a plurality of test questions to be scored;

[0067] S202, obtaining a plurality of sets of second evaluation features of a plurality of calibration test questions, a plurality of reference scores of the plurality of calibration test questions, and a plurality of initial predicted scores of the plurality of calibration test questions;

[0068] Optionally, the plurality of test questions to be scored and the plurality of calibration test questions can be initial test questions in the same test paper.

[0069] Optionally, the plurality of test questions to be scored and the plurality of calibration test questions can be initial test questions of a single type in the same test paper.

[0070] Optionally, the plurality of test questions to be scored and the plurality of calibration test questions can be initial test questions in a plurality of test papers of the same subject.

[0071] Optionally, the plurality of test questions to be scored and the plurality of calibration test questions can be initial test questions of a single type in a plurality of test papers of the same subject.

[0072] Optionally, the types of test questions can be divided into reading of fixed text content and non-reading of non-fixed text.

[0073] Specifically, the specific classification rules of the types of test questions are not limited in the present application.

[0074] Optionally, the reading of fixed text content can be further divided into reading of words, sentences, chapters, etc.; and the non-reading of non-fixed text can be further divided into open, semi-open and closed types, and can be combined into many types of test questions.

[0075] In some optional embodiments of the present application, the plurality of test questions to be scored and the plurality of calibration test questions are included in a plurality of initial test questions.

[0076] Optionally, the method further comprises: inputting the plurality of initial test questions into an evaluation unit to obtain a plurality of sets of first evaluation features of the plurality of test questions to be scored, a plurality of sets of second evaluation features of the plurality of calibration test questions, an initial predicted score of each test question to be scored in the plurality of test questions to be scored, and an initial predicted score of each calibration test question in the plurality of calibration test questions.

[0077] Optionally, with respect to the evaluation features in the present application, for the reading of fixed text type, the relevant pronunciation features calculated after forced alignment can be used; for the non-fixed text type, text features or bert features can be used.

[0078] In some optional embodiments of the present application, the method further comprises the following steps S01-04:

[0079] S01, obtaining a plurality of initial test questions;

[0080] Optionally, the plurality of initial test questions can be a plurality of test questions in a same test paper and a plurality of answers to the plurality of test questions replied by a test taker.

[0081] Optionally, the plurality of initial test questions can be a plurality of test questions in a plurality of test papers of a same subject and a plurality of answers to the plurality of test questions replied by a test taker.

[0082] S02, determining a preset number of partial test questions from the plurality of initial test questions as the plurality of calibration test questions;

[0083] In some optional embodiments of the present application, S02, determining a preset number of partial test questions from the plurality of initial test questions as the plurality of calibration test questions, comprises the following steps S021-S024:

[0084] S021, obtaining a preset total score;

[0085] Optionally, the preset total score can be 100 points.

[0086] S022, obtaining a total number of initial test questions in the plurality of initial test questions;

[0087] S023, dividing the total score into a plurality of score segments;

[0088] Optionally, the plurality of score segments can be: [0, 10], (10, 20], (20, 30]... (90, 100].

[0089] S024、for each score segment in the plurality of score segments, counting the number of initial test questions in the plurality of initial test questions whose initial predicted scores are located in the score segment; determining a target ratio of the number to the total number; selecting, from the initial test questions whose initial predicted scores are located in the score segment, target initial test questions whose ratio to the preset number is the target ratio as the calibration test questions corresponding to the score segment.

[0090] The target initial test questions whose ratio to the preset number is the target ratio can be selected randomly from the initial test questions whose initial predicted scores are located in the score segment.

[0091] The total number of calibration test questions corresponding to each score segment is the preset number. All calibration test questions corresponding to each score segment are the plurality of calibration test questions determined from the plurality of initial test questions. The plurality of calibration test questions form a calibration test question set. Specifically, the process of determining the calibration test question set can be referred to in Figure 2b .

[0092] S03、taking at least part of the test questions in the plurality of initial test questions other than the plurality of calibration test questions as the plurality of test questions to be scored;

[0093] Optionally, all test questions in the plurality of initial test questions other than the plurality of calibration test questions can be taken as the plurality of test questions to be scored.

[0094] S04, sending the plurality of calibration test questions to a first device, so that relevant personnel score the plurality of calibration test questions to obtain a plurality of reference scores of the plurality of calibration test questions.

[0095] The reference score is a manual score.

[0096] S203, sorting the plurality of groups of first evaluation features and the plurality of groups of second evaluation features to obtain a target sorting result;

[0097] In some optional embodiments of the present application, in the foregoing S203, the sorting the plurality of groups of first evaluation features and the plurality of groups of second evaluation features to obtain a target sorting result comprises: inputting the plurality of groups of first evaluation features and the plurality of groups of second evaluation features into a target sorting model to obtain a target sorting result; wherein the target sorting model is used for sorting the plurality of groups of first evaluation features and the plurality of groups of second evaluation features.

[0098] In some optional embodiments of the present application, in the foregoing S203, the sorting the plurality of groups of first evaluation features and the plurality of groups of second evaluation features to obtain a target sorting result comprises:

[0099] For each group of first evaluation features in the plurality of groups of first evaluation features, a feature pair of the first evaluation feature and each group of second evaluation features in the plurality of groups of second evaluation features is constructed to obtain a plurality of feature pairs corresponding to the first evaluation feature; the plurality of feature pairs are input into a target ranking model to obtain a plurality of ranking results corresponding to the plurality of feature pairs, and then a target ranking result is obtained.

[0100] In the target ranking result, a plurality of ranking results corresponding to a plurality of feature pairs corresponding to each group of first evaluation features are included.

[0101] Optionally, each first evaluation feature corresponds to a plurality of feature pairs and a plurality of ranking results. The plurality of ranking results corresponding to each first evaluation feature can be regarded as a group of ranking results; and the plurality of groups of ranking results corresponding to the plurality of first evaluation features are regarded as a target ranking result.

[0102] Optionally, each first evaluation feature corresponds to a plurality of feature pairs and a plurality of ranking results. The plurality of ranking results corresponding to each first evaluation feature can be regarded as a group of ranking results; and the total ranking result determined based on the plurality of groups of ranking results corresponding to the plurality of first evaluation features is regarded as a target ranking result.

[0103] S204, determining a score result of the plurality of to-be-scored test questions based on the target ranking result, the plurality of reference scores, and the plurality of initial predicted scores.

[0104] In some optional embodiments of the present application, in the foregoing S204, the determining of the score result of the plurality of to-be-scored test questions based on the target ranking result, the plurality of reference scores, and the plurality of initial predicted scores includes: for each to-be-scored test question in the plurality of to-be-scored test questions,

[0105] If the target ranking result indicates that the first evaluation feature of the to-be-scored test question is located between two groups of second evaluation features, the score result of the to-be-scored test question is determined based on a first preset algorithm, initial predicted scores and reference scores of two calibration test questions corresponding to the two groups of second evaluation features.

[0106] If the target ranking result indicates that the first evaluation feature of the to-be-scored test question is located before the plurality of groups of second evaluation features, the score result of the to-be-scored test question is determined based on a second preset algorithm, initial predicted scores and reference scores of two calibration test questions corresponding to two second evaluation features located after the first evaluation feature and most adjacent to the first evaluation feature in the plurality of groups of second evaluation features.

[0107] Optionally, the score result of the to-be-scored test question is determined based on the first preset algorithm, initial predicted scores and reference scores of two calibration test questions corresponding to the two groups of second evaluation features, and can be realized by the following formula:

[0108]

[0109] wherein, x test denotes the first evaluation feature of the to-be-scored test question, x train1 denotes the second evaluation feature before the first evaluation feature, x train2 denotes the second evaluation feature after the first evaluation feature, f(x train1 ) is the initial predicted score of the calibration test question corresponding to x train1 , f(x train2 ) is the initial predicted score of the calibration test question corresponding to x train2 , h(x train1 ) is the reference score of the calibration test question corresponding to x train1 , and h(x train2 ) is the reference score of the calibration test question corresponding to x train2 . denotes the score result of the to-be-scored test question.

[0110] Optionally, the score result of the to-be-scored test question is determined based on the second preset algorithm, initial predicted scores and reference scores of two calibration test questions corresponding to two second evaluation features which are arranged after the first evaluation feature and are most adjacent to the first evaluation feature, and can be realized by the following formula:

[0111]

[0112] wherein, x test denotes the first evaluation feature of the to-be-scored test question, x train11 denotes the second evaluation feature which is arranged after the first evaluation feature and is most adjacent to the first evaluation feature, x train12 denotes the second evaluation feature which is arranged after the first evaluation feature and is second most adjacent to the first evaluation feature, f(x train11 ) is the initial predicted score of the calibration test question corresponding to x train11 , f(x train12 ) is the initial predicted score of the calibration test question corresponding to x train12 , h(x train11 ) is the reference score of the calibration test question corresponding to x train11 , and h(x train12 ) is the reference score of the calibration test question corresponding to x train12 . denotes the score result of the to-be-scored test question.

[0113] In some optional embodiments of the present application, the method further comprises: training an initial ranking model based on the multiple sets of second evaluation features, the multiple reference scores of the multiple calibration test questions, and the multiple initial predicted scores of the multiple calibration test questions to obtain the target ranking model.

[0114] Optionally, the initial ranking model in the present application can be a RankSVM model.

[0115] In some optional embodiments of the present application, the training of the initial ranking model based on the multiple sets of second evaluation features, the multiple reference scores of the multiple calibration test questions, and the multiple initial predicted scores of the multiple calibration test questions to obtain the target ranking model comprises the following steps S1-S5:

[0116] S1, two sets of second evaluation features are taken out from the multiple sets of second evaluation features to form a sample feature pair;

[0117] Optionally, in S1, the two sets of second evaluation features are taken out from the multiple sets of second evaluation features to form a sample feature pair, comprising: taking out any two sets of second evaluation features from the multiple sets of second evaluation features to form a sample feature pair.

[0118] Optionally, in S1, the two sets of second evaluation features are taken out from the multiple sets of second evaluation features to form a sample feature pair, comprising: taking out any two sets of second evaluation features that have not been taken out from the multiple sets of second evaluation features to form a sample feature pair.

[0119] S2, obtaining the real ranking result corresponding to the sample feature pair;

[0120] Optionally, each calibration test question corresponds to a set of second evaluation features, and each set of second evaluation features can be represented in the following manner: X1, X2,..., X m ;

[0121] X1 = {x 11 ,x 12 ,...,x 1n}; wherein x 11 ,x 12 ,...,x 1n are multi-dimensional evaluation features in X1. X2 = {x 21 ,x 22 ,...,x 2n};...

[0122] X m = {x m1 ,x m2 ,...,x mn};

[0123] X1 corresponds to the reference score Y1 of the calibration test question;

[0124] X2 corresponds to the reference score Y2 of the calibration test question;...

[0125] X m corresponds to the reference score Y m ;

[0126] When the real ranking of X1 is before X2, Y1 is greater than Y2, and the corresponding label can be +1;

[0127] When the real ranking of X1 is after X2, Y1 is less than Y2, and the corresponding label can be -1;

[0128] The feature of the sample feature pair can refer to the difference between the two second evaluation features in the sample feature pair, and the label can refer to their relative advantage and disadvantage relationship.

[0129] S3, inputting the sample feature pair into the initial ranking model to obtain a predicted ranking result;

[0130] Optionally, before inputting the sample feature pair into the initial ranking model, the sample feature pair can be processed as follows: calculating the difference feature value between the two second evaluation features in the sample feature pair; and taking the difference feature value and the corresponding label as sample data for inputting into the initial ranking model.

[0131] S4, determining the corresponding loss information based on the real ranking result and the predicted ranking result;

[0132] Optionally, in the foregoing S4, the corresponding loss information is determined based on the real ranking result and the predicted ranking result, which can be realized by the following formula:

[0133]

[0134] where X i is the i-th second evaluation feature, X j is the i-th second evaluation feature, f(X i ,X j )=w T (X i -X j )+b, w and b are adjustable parameters in the initial ranking model. C is a hyperparameter, ||w|| 2 is a regularization term, max(0,1-y ij f(X i ,X j )) is the hinge loss of the ranking pair, which controls the trade-off between regularization and loss.

[0135] S5, when the loss information meets the preset condition, taking the initial ranking model determined last time as a target ranking model, when the loss information does not meet the preset condition, adjusting parameters of the initial ranking model, and returning to execute the step of taking two groups of second evaluation features from the plurality of groups of second evaluation features to form a sample feature pair until a target ranking model is determined.

[0136] Optionally, when the loss information is less than a preset threshold, the loss information is considered to meet the preset condition.

[0137] The present application provides a pairwise-based oral test scheme, which no longer uses a regression method to calculate the total score, and removes the possible nonlinear relationship between the regression model and the features. The problems of overfitting (overfitting the training data, poor generalization ability) or underfitting (failure to capture the patterns in the data) of the model due to insufficient number of training samples or improper feature selection are avoided. The single regression model cannot comprehensively capture the complex phonemes involved in oral test, such as pronunciation, fluency, grammar, and vocabulary richness. The scoring scheme no longer trains a regression model, solving the problem of model effect caused by data imbalance. In addition, different ranking calculation schemes can be selected according to the difference in device computing capacity.

[0138] The present application provides a scheme for obtaining a plurality of groups of first evaluation features of a plurality of to-be-scored test questions, a plurality of groups of second evaluation features of a plurality of calibration test questions, a plurality of reference scores of the plurality of calibration test questions, and a plurality of initial predicted scores of the plurality of calibration test questions, ranking the plurality of groups of first evaluation features and the plurality of groups of second evaluation features to obtain a target ranking result, and determining a scoring result of the plurality of to-be-scored test questions based on the target ranking result, the plurality of reference scores, and the plurality of initial predicted scores. The features are ranked to score the test questions, which can avoid a series of problems caused by scoring the test questions based on a regression model, and improves the efficiency of scoring oral tests.

[0139] Figure 3 A structural schematic diagram of a scoring device provided by an exemplary embodiment of the present application is provided; wherein the device comprises:

[0140] The obtaining unit 31 is configured to obtain a plurality of groups of first evaluation features of a plurality of to-be-scored test questions.

[0141] The obtaining unit 31 is further configured to obtain a plurality of groups of second evaluation features of a plurality of calibration test questions, a plurality of reference scores of the plurality of calibration test questions, and a plurality of initial predicted scores of the plurality of calibration test questions.

[0142] The ranking unit 32 is configured to rank the plurality of groups of first evaluation features and the plurality of groups of second evaluation features to obtain a target ranking result.

[0143] The determining unit 33 is configured to determine the score results of the multiple to-be-scored examination questions based on the target ranking result, the multiple reference scores, and the multiple initial predicted scores.

[0144] In some optional embodiments of the present application, when the apparatus is used to rank the multiple groups of first evaluation features and the multiple groups of second evaluation features to obtain a target ranking result, the apparatus is specifically configured to:

[0145] input the multiple groups of first evaluation features and the multiple groups of second evaluation features into a target ranking model to obtain the target ranking result.

[0146] The target ranking model is configured to rank the multiple groups of first evaluation features and the multiple groups of second evaluation features.

[0147] In some optional embodiments of the present application, when the apparatus is used to rank the multiple groups of first evaluation features and the multiple groups of second evaluation features to obtain a target ranking result, the apparatus is specifically configured to:

[0148] for each group of first evaluation features in the multiple groups of first evaluation features, construct a feature pair of the first evaluation feature and each group of second evaluation features in the multiple groups of second evaluation features to obtain multiple feature pairs corresponding to the first evaluation feature; input the multiple feature pairs into a target ranking model to obtain multiple ranking results corresponding to the multiple feature pairs, and then obtain the target ranking result.

[0149] In the target ranking result, multiple ranking results corresponding to multiple feature pairs corresponding to each group of first evaluation features are included.

[0150] In some optional embodiments of the present application, when the apparatus is used to determine the score results of the multiple to-be-scored examination questions based on the target ranking result, the multiple reference scores, and the multiple initial predicted scores, the apparatus is specifically configured to:

[0151] for each to-be-scored examination question in the multiple to-be-scored examination questions,

[0152] if the target ranking result indicates that the first evaluation feature of the to-be-scored examination question is located between two groups of second evaluation features, determine the score result of the to-be-scored examination question based on a first preset algorithm and initial predicted scores and reference scores of two calibration examination questions corresponding to the two groups of second evaluation features.

[0153] If the target ranking result indicates that the first evaluation feature of the to-be-scored test question is located before the plurality of second evaluation features, the initial predicted scores and the reference scores of two calibration test questions corresponding to two second evaluation features that are most adjacent to the first evaluation feature and located after the first evaluation feature in the plurality of second evaluation features are used to determine the score result of the to-be-scored test question based on a second preset algorithm.

[0154] In some optional embodiments of the present application, the foregoing apparatus is further configured to:

[0155] obtain a plurality of initial test questions;

[0156] determine a preset number of partial test questions from the plurality of initial test questions as the plurality of calibration test questions;

[0157] use at least partial test questions from the plurality of initial test questions other than the plurality of calibration test questions as the plurality of to-be-scored test questions;

[0158] send the plurality of calibration test questions to a first device, so that relevant personnel score the plurality of calibration test questions to obtain a plurality of reference scores of the plurality of calibration test questions.

[0159] In some optional embodiments of the present application, the foregoing apparatus is further configured to:

[0160] train an initial ranking model based on the plurality of second evaluation features, the plurality of reference scores of the plurality of calibration test questions, and the plurality of initial predicted scores of the plurality of calibration test questions to obtain the target ranking model.

[0161] In some optional embodiments of the present application, when the foregoing apparatus is configured to train an initial ranking model based on the plurality of second evaluation features, the plurality of reference scores of the plurality of calibration test questions, and the plurality of initial predicted scores of the plurality of calibration test questions to obtain the target ranking model, the apparatus is specifically configured to:

[0162] select two second evaluation features from the plurality of second evaluation features to form a sample feature pair;

[0163] obtain a real ranking result corresponding to the sample feature pair;

[0164] input the sample feature pair into an initial ranking model to obtain a predicted ranking result;

[0165] determine corresponding loss information based on the real ranking result and the predicted ranking result;

[0166] When the loss information meets the preset condition, the initial ranking model determined last time is taken as the target ranking model, and when the loss information does not meet the preset condition, the parameters of the initial ranking model are adjusted, and the step of taking out two sets of second evaluation features from the plurality of sets of second evaluation features to form a sample feature pair is returned to be executed until the target ranking model is determined.

[0167] It should be understood that the device embodiments and the method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, details are not described here. Specifically, the device can perform the above-mentioned method embodiments, and the foregoing and other operations and / or functions of each module in the device are respectively for the corresponding flow in each method in the above-mentioned method embodiments, and for the sake of brevity, details are not described here.

[0168] The device of the embodiments of the present application is described above from the perspective of functional modules in combination with the drawings. It should be understood that the functional modules can be realized by hardware, or by instructions in the form of software, or by a combination of hardware and software modules. Specifically, each step of the method embodiments in the embodiments of the present application can be completed by integrated logic circuits of hardware in the processor and / or instructions in the form of software, and the steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware code processor execution completion, or executed by a combination of hardware and software modules in the code processor. Alternatively, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and combines the hardware to complete the steps in the above-mentioned method embodiments.

[0169] Figure 4 is a schematic block diagram of an electronic device provided by the embodiments of the present application, which can include:

[0170] The memory 301 is used to store computer programs and transmit the program codes to the processor 302. In other words, the processor 302 can call and run the computer programs from the memory 301 to implement the method in the embodiments of the present application.

[0171] For example, the processor 302 can be used to execute the above-mentioned method embodiments according to the instructions in the computer program.

[0172] In some embodiments of the present application, the processor 302 can include but is not limited to:

[0173] A general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc.

[0174] In some embodiments of the present application, the memory 301 includes, but is not limited to:

[0175] volatile memory and / or non-volatile memory. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM) used as an external cache. By way of example, and not limitation, many forms of RAM can be used, such as a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a synch link DRAM (SLDRAM), and a Direct Rambus RAM (DR RAM).

[0176] In some embodiments of the present application, the computer program can be divided into one or more modules, which are stored in the memory 301 and executed by the processor 302 to complete the method provided by the present application. The one or more modules can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the electronic device.

[0177] As shown in Figure 4 , the electronic device can further include:

[0178] A transceiver 303, which can be connected to the processor 302 or the memory 301.

[0179] The processor 302 can control the transceiver 303 to communicate with other devices, specifically, can send information or data to other devices, or receive information or data sent by other devices. The transceiver 303 can include a transmitter and a receiver. The transceiver 303 can further include an antenna, and the number of antennas can be one or more.

[0180] It should be understood that various components in the electronic device are connected through a bus system, which includes a data bus, a power supply bus, a control bus, and a state signal bus in addition to a data bus.

[0181] The present application also provides a computer readable storage medium, which stores a computer program. When the computer program is executed by a computer, the computer can execute the method of the above method embodiments. Alternatively, the present application embodiment also provides a computer program product containing instructions, which, when executed by a computer, causes the computer to execute the method of the above method embodiments.

[0182] When implemented by using software, the computer program product can be implemented in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the flow or function according to the present application embodiment is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. containing one or more available media sets. The available medium can be a magnetic medium (for example, floppy disk, hard disk, magnetic tape), an optical medium (for example, digital video disc (DVD)), or a semiconductor medium (for example, solid state disk (SSD)) and the like.

[0183] Those skilled in the art can understand that the modules and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0184] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are merely illustrative, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed modules can be indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0185] The modules described as separate components can or can not be physically separated, and the components shown as modules can or can not be physical modules, i.e. they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiments according to actual needs. For example, the functional modules in each embodiment of the present application can be integrated in one processing module, or each module can be physically present separately, or two or more modules can be integrated in one module.

[0186] The above is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any skilled person in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A scoring method, characterized in that, include: Obtain multiple sets of first assessment features for multiple test questions to be scored; Obtain multiple sets of second evaluation features for multiple calibrated test questions, multiple reference scores for the multiple calibrated test questions, and multiple initial predicted scores for the multiple calibrated test questions; The multiple sets of first evaluation features and the multiple sets of second evaluation features are sorted to obtain the target sorting result; The scoring results of the multiple test questions to be scored are determined based on the target ranking results, the multiple reference scores, and the multiple initial predicted scores. The step of sorting the multiple sets of first evaluation features and the multiple sets of second evaluation features to obtain a target sorting result includes: inputting the multiple sets of first evaluation features and the multiple sets of second evaluation features into a target sorting model to obtain a target sorting result; Based on the multiple sets of second evaluation features, the multiple reference scores of the multiple calibrated test questions, and the multiple initial predicted scores of the multiple calibrated test questions, the initial ranking model is trained to obtain the target ranking model, including: Two sets of second evaluation features are selected from the multiple sets of second evaluation features to form a sample feature pair; Obtain the true ranking results corresponding to the sample feature pairs; The sample feature pairs are input into the initial ranking model to obtain the predicted ranking result; Based on the actual ranking results and the predicted ranking results, the corresponding loss information is determined; When the loss information meets the preset conditions, the most recently determined initial ranking model is used as the target ranking model. When the loss information does not meet the preset conditions, the parameters of the initial ranking model are adjusted, and the process returns to the step of taking two sets of second evaluation features from the multiple sets of second evaluation features to form a sample feature pair, until the target ranking model is determined.

2. The method according to claim 1, characterized in that, The target ranking model is used to rank the multiple sets of first evaluation features and the multiple sets of second evaluation features.

3. The method according to claim 1, characterized in that, The process of sorting the multiple sets of first evaluation features and the multiple sets of second evaluation features to obtain the target ranking result includes: For each of the multiple sets of first evaluation features, feature pairs are constructed between the first evaluation features and each of the multiple sets of second evaluation features, resulting in multiple feature pairs corresponding to the first evaluation features; the multiple feature pairs are input into the target ranking model to obtain multiple ranking results corresponding to the multiple feature pairs, and then the target ranking result is obtained; The target ranking result includes multiple ranking results corresponding to multiple feature pairs corresponding to the first evaluation features of each group.

4. The method according to claim 1, characterized in that, The step of determining the scoring results for the multiple test questions to be scored based on the target ranking result, the multiple reference scores, and the multiple initial predicted scores includes: For each of the multiple unscored test questions, If the target ranking result indicates that the first evaluation feature of the test question to be scored is located between two sets of second evaluation features, then the scoring result of the test question to be scored is determined based on the first preset algorithm, the initial predicted score and the reference score of the two calibration test questions corresponding to the two sets of second evaluation features; If the target ranking result indicates that the first evaluation feature of the test question to be scored is located before the multiple sets of second evaluation features, then the scoring result of the test question to be scored is determined based on the second preset algorithm and the initial predicted score and reference score of the two calibration test questions corresponding to the two second evaluation features that are ranked after the first evaluation feature and are most adjacent to the first evaluation feature.

5. The method according to claim 1, characterized in that, The method further includes: Obtain multiple initial test questions; From the plurality of initial test questions, a predetermined number of test questions are determined as the plurality of standard test questions; At least a portion of the initial test questions, excluding the standard test questions, will be used as the test questions to be scored. The multiple calibration test questions are sent to the first device, so that relevant personnel can score the multiple calibration test questions and obtain multiple reference scores for the multiple calibration test questions.

6. A scoring device, characterized in that, include: The acquisition unit is used to acquire multiple sets of first evaluation features for multiple test questions to be scored. The acquisition unit is used to acquire multiple sets of second evaluation features of multiple calibration test questions, multiple reference scores of the multiple calibration test questions, and multiple initial predicted scores of the multiple calibration test questions. The sorting unit is used to sort the multiple sets of first evaluation features and the multiple sets of second evaluation features to obtain the target sorting result; The determining unit is used to determine the scoring results of the multiple test questions to be scored based on the target ranking result, the multiple reference scores, and the multiple initial predicted scores; The step of sorting the multiple sets of first evaluation features and the multiple sets of second evaluation features to obtain a target sorting result includes: inputting the multiple sets of first evaluation features and the multiple sets of second evaluation features into a target sorting model to obtain a target sorting result; Based on the multiple sets of second evaluation features, the multiple reference scores of the multiple calibrated test questions, and the multiple initial predicted scores of the multiple calibrated test questions, the initial ranking model is trained to obtain the target ranking model, including: Two sets of second evaluation features are selected from the multiple sets of second evaluation features to form a sample feature pair; Obtain the true ranking results corresponding to the sample feature pairs; The sample feature pairs are input into the initial ranking model to obtain the predicted ranking result; Based on the actual ranking results and the predicted ranking results, the corresponding loss information is determined; When the loss information meets the preset conditions, the most recently determined initial ranking model is used as the target ranking model. When the loss information does not meet the preset conditions, the parameters of the initial ranking model are adjusted, and the process returns to the step of taking two sets of second evaluation features from the multiple sets of second evaluation features to form a sample feature pair, until the target ranking model is determined.

7. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1-5 by executing the executable instructions.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-5.

Images