Spoken language assessment methods, apparatuses, related devices, and computer program products

Abstract

The application discloses a spoken language evaluation method and device, related equipment and a computer program product. The method comprises the following steps: obtaining the answer data of a testee, wherein the answer data comprises a question, the answer audio of the testee and a reference answer; identifying the answer text corresponding to the answer audio; obtaining the reasoning score of the testee by combining the answer text and the answer data and through a configured reasoning scoring model; obtaining a configured calibration model, wherein the calibration model is obtained by pre-training based on the answer text of a calibration testee, the reasoning score of the calibration testee and an expert score; the calibration testee is part of the testees participating in the current oral test; and obtaining the final score of each testee by using the calibration model to score according to the answer text and the reasoning score of each testee. Compared with the prior art which determines the score by simply calculating the similarity between the answer text and the reference answer, the oral evaluation result obtained by the application is more accurate.

Description

Technical Field

[0001] This application relates to the field of spoken language assessment technology, and more specifically, to a spoken language assessment method, apparatus, related equipment, and computer program product. Background Technology

[0002] In speaking tests, test takers typically demonstrate their oral communication skills through a series of tasks. These tasks may include self-introduction, describing pictures or charts, answering questions, and participating in conversations.

[0003] With the development of artificial intelligence technology, many oral assessment systems can now automatically score responses. These systems use speech recognition technology to convert test takers' spoken expressions into text and use natural language processing algorithms to analyze various aspects of the text. However, in actual exams, test takers need to demonstrate not only language skills but also confidence, logical thinking, and adaptability. Therefore, oral assessments are not only a test of language ability but also a test of comprehensive abilities. However, due to the diverse performances of test takers, standard answer keys cannot encompass all suitable responses. Current oral assessment systems heavily rely on the content of answer keys, but the number of answer keys is limited, and the test paper creation stage cannot include all correct answers. This leads to situations where a test taker provides a correct answer, but because that answer was not included in the answer key beforehand, the text similarity between the test taker's answer and the answer key is low, resulting in a lower score and reducing the accuracy of the oral assessment. Summary of the Invention

[0004] In view of the above problems, this application is made to provide a spoken language assessment method, device, related equipment, and computer program product to improve the accuracy of spoken language assessment. The specific solution is as follows:

[0005] Firstly, it provides a method for oral assessment, including:

[0006] The system acquires the candidates' answer data, which includes questions, audio recordings of the candidates' answers, and reference answers.

[0007] Identify the answer text corresponding to the answer audio, and combine the answer text and the answer data to obtain the candidate's reasoning score through the configured reasoning scoring model;

[0008] Obtain the configured calibration model, which is pre-trained based on the answer texts of the calibration candidates, the reasoning scores of the calibration candidates, and the expert scores. The calibration candidates are a portion of the candidates selected from all candidates participating in this oral exam.

[0009] Based on each candidate's answer text and reasoning score, the calibration model is used to score each candidate and obtain their final score.

[0010] In one possible design, in another implementation of the first aspect of the embodiments of this application, the reasoning scoring model includes a pronunciation scoring model and a completeness scoring model;

[0011] The process of obtaining the candidate's reasoning score through a configured reasoning scoring model, combining the answer text and the answer data, includes:

[0012] The pronunciation scoring model is used to process the answer audio to obtain the pronunciation score output by the model;

[0013] The completeness scoring model is used to process the question, the reference answer, and the answer text to obtain the completeness score output by the model.

[0014] The candidate's reasoning score is composed of the pronunciation score and the completeness score.

[0015] In one possible design, in another implementation of the first aspect of the embodiments of this application, when the reference answer includes two or more, the process of using the completeness scoring model to process the question, the reference answer, and the answer text to obtain the completeness score output by the model includes:

[0016] Calculate the similarity between the answer text and each of the reference answers, and select the target reference answer with the highest similarity.

[0017] The completeness scoring model is used to process the question, the target reference answer, and the answer text to obtain the completeness score output by the model.

[0018] In one possible design, in another implementation of the first aspect of the embodiments of this application, the process of selecting benchmark candidates from all candidates participating in this oral examination includes:

[0019] All candidates who participated in this oral exam were ranked according to their reasoning scores.

[0020] Select a portion of the qualified candidates from each of the established scoring intervals to form the complete list of qualified candidates.

[0021] In one possible design, in another implementation of the first aspect of the embodiments of this application, the calibration model includes: a feature extraction model for each examinee, and a regression model;

[0022] The feature extraction model for each candidate is a model trained using the candidate's answer text in an unsupervised training manner. The feature extraction model is used to extract the text features of the corresponding candidate's answer text.

[0023] The regression model is trained using the textual features and inference scores of the benchmarked candidates' answer texts as input and the expert scores of the benchmarked candidates as output.

[0024] In one possible design, in another implementation of the first aspect of the embodiments of this application, the regression model includes: a first regression model and a second regression model;

[0025] The first regression model is trained by taking the text features of the test-takers' answer texts as input and the expert scores of the test-takers as output.

[0026] The second regression model is trained using the expert ratings predicted by the first regression model for the calibrated candidates and the inference ratings of the calibrated candidates as inputs, and the expert ratings of the calibrated candidates as outputs.

[0027] In one possible design, in another implementation of the first aspect of the embodiments of this application, the process of scoring each candidate using the calibration model based on the candidate's answer text and the inference score to obtain the final score for each candidate includes:

[0028] For each candidate:

[0029] The candidate's answer text is fed into the corresponding feature extraction model to extract the text features of the candidate's answer text;

[0030] The text features of the candidate's answer text are fed into the first regression model to obtain the output first score;

[0031] The first score and the candidate's reasoning score are fed into the second regression model to obtain the final output score.

[0032] In one possible design, in another implementation of the first aspect of this application, the process of identifying the answer text corresponding to the answer audio includes:

[0033] Extract the acoustic features of the answer audio;

[0034] The acoustic features are input into a configured speech recognition model, which includes an encoder, a decoder, and a keyword activation module. The keyword activation module encodes the features of each keyword in the reference answer, predicts the sub-word score of each keyword, performs a weighted fusion of the sub-word scores predicted by the decoder and the sub-word scores predicted by the keyword activation module, and determines the recognition result based on the fused sub-word scores, which is used as the answer text.

[0035] In one possible design, in another implementation of the first aspect of the embodiments of this application, when there are multiple candidates to be evaluated, a recognition core is configured for each candidate, and a reasoning core corresponding to the multiple recognition cores is configured.

[0036] The process of identifying the text of each candidate's answer audio includes:

[0037] The input features of the speech recognition model are constructed using the recognition core corresponding to each candidate. The inference sequence number of the recognition core is marked and sent to the inference core. The inference result returned by the inference core is received, and the final recognized answer text is output based on the inference result.

[0038] The inference core obtains the input features carrying inference sequence numbers sent by each of the recognition cores, concatenates the input features from multiple channels according to the inference sequence number, sends them into the speech recognition model for inference, and returns the inference result to the corresponding recognition core according to the inference sequence number.

[0039] Secondly, a spoken language assessment device is provided, including:

[0040] The answer data acquisition unit is used to acquire the candidate's answer data, which includes questions, audio recordings of the candidate's answers, and reference answers.

[0041] An audio recognition unit is used to recognize the answer text corresponding to the answer audio.

[0042] The reasoning scoring unit is used to combine the answer text and the answer data to obtain the candidate's reasoning score through a configured reasoning scoring model;

[0043] The calibration model acquisition unit is used to acquire the configured calibration model, which is pre-trained based on the answer text of the calibration candidates, the reasoning scores of the calibration candidates, and the expert scores. The calibration candidates are a portion of the candidates selected from all candidates participating in this oral exam.

[0044] The final scoring unit is used to score each candidate's answer text and reasoning score using the calibration model to obtain the final score for each candidate.

[0045] Thirdly, an electronic device is provided, comprising: a memory and a processor;

[0046] The memory is used to store programs;

[0047] The processor is configured to execute the program to implement the steps of the oral assessment method described in any of the first aspects of this application.

[0048] Fourthly, a readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the oral assessment method described in any of the preceding first aspects of this application.

[0049] Fifthly, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of the oral assessment method described in any of the first aspects of this application.

[0050] By employing the aforementioned technical solution, this application identifies the answer text corresponding to the candidate's audio answer, and combines the answer text with the reference answers and audio answers in the answer data to determine the candidate's inference score through an inference scoring model. Building upon this, to improve the accuracy of the oral assessment, a calibration model is further introduced. This involves selecting a subset of calibration candidates from all participants in the oral exam, and having experts score these candidates to obtain expert scores, which are more accurate. Based on this, the calibration model is trained using the calibration candidates' answer texts, inference scores, and expert scores. This calibration model can predict more accurate expert scores based on the candidate's answer texts and inference scores. Therefore, for each candidate, a final score can be obtained by using the calibration model based on their answer texts and inference scores. In determining the candidate's final score, this application combines both the reference answers and the expert calibration scores, giving the calibration model a certain degree of scoring freedom based on the reference answers. Compared to existing technologies that simply calculate the similarity between the answer text and the reference answers to determine the score, the oral assessment results obtained by this application are more accurate. Furthermore, since the calibration model is trained based on expert scores from calibrated test takers, it can be understood as having undergone customized training and learning for the scoring scale and student responses of each oral exam. Therefore, it is more suitable for evaluating the oral performance of each test taker in this oral exam, and further improves the accuracy of the oral assessment results. Attached Figure Description

[0051] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0052] Figure 1 A schematic diagram of an implementation system architecture for the oral assessment method provided in this application embodiment;

[0053] Figure 2 A schematic diagram of a spoken language assessment method provided in this application embodiment;

[0054] Figure 3 This is a schematic diagram of the structure of a pronunciation scoring model provided in an embodiment of this application;

[0055] Figure 4 This is a schematic diagram of a completeness scoring model structure provided in an embodiment of this application;

[0056] Figure 5 This is a schematic diagram of a speech recognition model structure provided in an embodiment of this application;

[0057] Figure 6 A schematic flowchart illustrating a method for recognizing the audio answers of multiple examinees, provided in an embodiment of this application;

[0058] Figure 7 A schematic diagram of another oral assessment method provided in this application embodiment;

[0059] Figure 8 A schematic diagram of the structure of a spoken language assessment device provided in this application embodiment;

[0060] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0061] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0062] This application provides a spoken language assessment method that can be applied to, for example... Figure 1 The system architecture shown may include a terminal 100 and a server 200. The server 200 may include one or more servers (…). Figure 1(This example uses a server as an illustration).

[0063] In this embodiment, the terminal 100 can specifically be a device with data processing capabilities, such as a tablet computer, a learning machine, a teaching terminal, or a laptop computer.

[0064] Either terminal 100 or server 200 can be used independently to execute the oral assessment method provided in the embodiments of this application. Alternatively, terminal 100 and server 200 can also be used collaboratively to execute the oral assessment method provided in the embodiments of this application.

[0065] For example, terminal 100 acquires the candidate's answer data and completes the oral assessment of the candidate to obtain the oral assessment result.

[0066] In another example, terminal 100 obtains the candidate's answer data, uploads the answer data to server 200, and server 200 completes the oral assessment of the candidate and returns the oral assessment result to terminal 100.

[0067] In another example, terminal 100 and server 200 can cooperate to complete the oral assessment of the examinee. That is, some steps of the oral assessment are implemented on terminal 100, and other steps are implemented on server 200. The two coordinate and cooperate to complete the oral assessment.

[0068] The oral assessment scheme provided in this application can assess oral skills on both offline transmitted candidate answer data (such as answer audio) and online transmitted candidate answer data.

[0069] The oral assessment scheme proposed in this application can be applied to oral examination scenarios in various subjects, such as English oral examination assessment and Chinese oral examination assessment.

[0070] This application provides a spoken language assessment method, illustrated by applying the method to a computer device, specifically which may be... Figure 1 The system consists of terminal 100 or a combination of terminal 100 and server 200. (Refer to...) Figure 2 The oral assessment method specifically includes the following steps:

[0071] Step S100: Obtain the candidate's answer data, which includes questions, audio recordings of the candidate's answers, and reference answers.

[0072] Specifically, for all candidates participating in this oral exam, each candidate's answer data includes exam paper information and corresponding audio recordings. The exam paper information refers to the file created according to the exam paper production guidelines, which may include questions and corresponding answer keys.

[0073] Depending on the type of speaking test question, there may be one or more suggested answers. For example, questions involving sentence translation or paraphrasing may have more than one suggested answer.

[0074] Step S110: Identify the answer text corresponding to the answer audio, and combine the answer text and answer data to obtain the candidate's reasoning score through the configured reasoning scoring model.

[0075] Specifically, for each candidate's audio answer, the recognition result, i.e., the answer text, is obtained through a speech recognition model.

[0076] Furthermore, in this step, the answer text and answer data can be combined to use a pre-trained reasoning scoring model to predict the score and obtain the candidate's reasoning score output by the model.

[0077] In this embodiment, the reasoning scoring model is configured to predict the corresponding reasoning score based on the answer text and answer data.

[0078] The reasoning scoring model can at least refer to the reference answers in the answer data, determining the reasoning score based on the matching between the answer text and the reference answers. Building on this, the model can further analyze and process the audio of the answer data to score the candidate's pronunciation, and then comprehensively consider the pronunciation, the matching between the answer text and the reference answers to determine the final reasoning score.

[0079] Step S120: Obtain the configured calibration model, which is pre-trained based on the calibration candidates' answer texts, the calibration candidates' reasoning scores, and expert scores.

[0080] Among them, the qualified candidates are a portion of the candidates selected from all candidates who participated in this oral exam. The selection of qualified candidates can be based on pre-defined selection rules, such as randomly selecting a set number of qualified candidates from all candidates.

[0081] This embodiment provides an optional extraction method:

[0082] First, all candidates who participated in this oral exam were ranked according to their reasoning scores.

[0083] Furthermore, a portion of the qualified candidates are selected from each of the established scoring intervals to form the complete list of qualified candidates.

[0084] The extraction method used in this embodiment can ensure that the extracted standardized candidates cover as many different scoring methods as possible, which facilitates a comprehensive and fair assessment of the overall level of the candidates.

[0085] After selecting the benchmark candidates, experts can be arranged to score their answers, resulting in expert scores for the benchmark candidates. Understandably, expert scores are more accurate and can be used as the standard scores.

[0086] Based on this, a calibration model is trained using the answer texts, reasoning scores, and expert scores of the calibrated candidates. The calibration model is configured to predict the expert scores based on the candidates' answer texts and reasoning scores.

[0087] Step S130: Based on each candidate's answer text and reasoning score, use the calibration model to score and obtain the final score for each candidate.

[0088] Specifically, the aforementioned steps yield each candidate's answer text and reasoning score. Based on this, a calibration model can be used to predict the final score, thus obtaining the final score for each candidate.

[0089] The oral assessment method provided in this application identifies the answer text corresponding to the candidate's audio response, combines the answer text with the reference answers and audio responses in the answer data, and determines the candidate's inference score through an inference scoring model. To further improve the accuracy of the oral assessment, a calibration model is introduced. This involves selecting a subset of candidates from all participants in the oral exam for expert scoring, resulting in more accurate expert scores. The calibration model is then trained using the answer texts, inference scores, and expert scores of these candidates. This calibration model can predict more accurate expert scores based on the candidates' answer texts and inference scores. Therefore, for each candidate, a final score can be obtained by using the calibration model based on their answer text and inference score. This application combines the reference answers and expert calibration scores in determining the final score, giving the calibration model a degree of scoring freedom based on the reference answers. Compared to existing methods that simply calculate the similarity between the answer text and the reference answers to determine the score, the oral assessment results obtained by this application are more accurate. Furthermore, since the calibration model is trained based on expert scores from calibrated test takers, it can be understood as having undergone customized training and learning for the scoring scale and student responses of each oral exam. Therefore, it is more suitable for evaluating the oral performance of each test taker in this oral exam, and further improves the accuracy of the oral assessment results.

[0090] In some embodiments of this application, the process of obtaining the candidate's reasoning score through a reasoning scoring model is described.

[0091] The reasoning scoring model can measure the completeness of the answer text at the text level. Furthermore, it can measure the pronunciation of the answer audio at the audio level to obtain a pronunciation score. The reasoning score is determined by comprehensively considering both the completeness and pronunciation scores.

[0092] Taking the reasoning score as an example, which includes both pronunciation and completeness scores:

[0093] Inference scoring models can include pronunciation scoring models and completeness scoring models.

[0094] The pronunciation scoring model is trained end-to-end using a data-driven approach.

[0095] The pronunciation scoring model is used to process the test takers' audio answers and obtain the pronunciation score output by the model.

[0096] Combination Figure 3 An example of an optional structure for a pronunciation scoring model is provided, which may include an audio encoder (Encoder) and a fully connected layer (FC). The Encoder may employ a Conformer-based encoder.

[0097] The input to the pronunciation scoring model can include acoustic features of the answer audio, such as Fbank features. In addition, it can further include a frame mask corresponding to the acoustic features in the answer audio.

[0098] like Figure 3 As shown, a "CLS" marker can be added before the input acoustic features. Based on this, the output of the encoder at the "CLS" position can be fed into the fully connected layer to predict the pronunciation score.

[0099] The pronunciation scoring model is trained using audio samples labeled with pronunciation scores as training data, following a predefined loss function. For example, the Huber loss function can be used. The specific calculation process is as follows:

[0100] For each audio sample i, calculate the difference between the predicted value and the true value:

[0101] The loss is calculated in two cases based on the absolute value of the difference and the magnitude of δ:

[0102] 1) If The loss is half the square of the difference, that is:

[0103]

[0104] 2) If The loss is:

[0105]

[0106] 3) Averaging the loss over all audio samples:

[0107]

[0108] Where n represents the number of audio samples, and yi represents the true pronunciation score (i.e., pronunciation score label) of the i-th audio sample. The value represents the model's prediction. δ is a parameter of the Huber loss, used to control the trade-off between the mean absolute error (MAE) and the mean squared error (MSE) of the loss function. A larger δ makes the loss function closer to the MSE at large errors, while a smaller δ makes the loss function closer to the MAE.

[0109] Of course, the above only illustrates the Huber loss function. Other types of loss functions can also be used in the pronunciation scoring model during the training phase, which will not be listed in this embodiment.

[0110] In this embodiment, by training an end-to-end pronunciation scoring model, the trained pronunciation scoring model can be used to predict the corresponding pronunciation score for the candidate's answer audio, thereby measuring the candidate's level from the perspective of pronunciation.

[0111] For completeness scoring models, they can also be trained end-to-end in a data-driven manner.

[0112] The completeness scoring model is used to process the questions, reference answers, and answer texts in the candidates' answer data to obtain the completeness score output by the model.

[0113] Completeness scoring models can use BERT or other backbone networks as the main structure. Figure 4 An example of an integrity scoring model using BERT as the main structure is given.

[0114] The input text sequence for the completeness scoring model can include question information, reference answers, and answer text.

[0115] Separators can be set between different types of input. For example, a sequence of input text can be formatted as follows:

[0116] [CLS]Question sequence[SEP]Ref sequence[SEP]ASR sequence[SEP].

[0117] In this context, Question represents the question, Ref represents the reference answer, and ASR represents the answer text.

[0118] It should be noted that if there are more than two reference answers for the current question in the answer data, the similarity between the candidate's answer text and each reference answer can be calculated, and the target reference answer with the highest similarity can be selected.

[0119] When calculating the similarity between a candidate's answer text and each reference answer, the Levenstein distance between the two can be calculated. The smaller the distance, the higher the similarity. Therefore, the reference answer with the smallest distance can be selected as the target reference answer (Ref).

[0120] When constructing the input text sequence for a completeness scoring model, the question, the target reference answer, and the answer text can be used to form the input text sequence.

[0121] After obtaining the input text sequence, the corresponding token sequence can be determined. For example, a tokenizer can be used to process the input text sequence to obtain the token sequence, i.e., token_id.

[0122] Furthermore, the input to BERT can also include token_id_type and token_mask.

[0123] Among them, token_id_type can be a sequence of all zeros with the same length as token_id, and the position value of the token that belongs to the answer text in token_id is set to 1, that is, the answer text in token_id is marked.

[0124] token_mask represents a mask used to preserve valid data across multiple batches. token_mask can be a sequence of all 1s with the same length as token_id.

[0125] The token_id, token_id_type, and token_mask constructed above are added together to obtain the input to BERT. BERT then processes the data and predicts the completeness score as the output.

[0126] In this embodiment, by training an end-to-end completeness scoring model, the trained completeness scoring model can be used to predict the corresponding completeness scores for questions, reference answers, and answer texts in the candidate's answer data, thereby measuring the completeness level of the answer text at the text level.

[0127] After obtaining the candidate's pronunciation score and completeness score in the above embodiments, the two can be combined to form the candidate's reasoning score.

[0128] In some embodiments of this application, the process of identifying the answer text corresponding to the candidate's answer audio in the aforementioned step S110 is described.

[0129] In one possible implementation, a traditional speech recognition model can be used to recognize the candidate's audio answers and obtain the answer text.

[0130] In some other possible implementations, in order to improve the accuracy of speech recognition results, this embodiment introduces a speech recognition model structure that adopts an encoder-decoder structure and includes a keyword activation module.

[0131] When building a speech recognition model, subwords can be used as the modeling unit. Compared to using whole words as the modeling unit, using subwords as the modeling unit can reduce false triggers.

[0132] For the audio of the test taker's answers, acoustic features are extracted and then fed into an acoustic encoder for encoding. The encoded features are then fed into a decoder, which predicts the word score P based on the encoded features. ed .

[0133] The keyword incentive module encodes the features of each keyword in the reference answer and predicts the sub-word score P for each keyword. hot .

[0134] In one possible implementation, when predicting the sub-word score for each keyword, the keyword activation module can further refer to the hidden features extracted by the decoder. That is, the keyword activation module can predict the sub-word score P for each keyword based on the feature encoding results of each keyword in the reference answer and the hidden features extracted by the decoder. hot .

[0135] Finally, the word score P predicted by the decoder is... ed The sub-word score P predicted by the keyword incentive module hot Perform weighted fusion:

[0136] P total =P ed ×(1-coeff)+P hot ×coeff

[0137] Among them, P total This represents the score of the merged sub-words, and coeff represents the score weight of the keyword incentive module. For example, coeff can be equal to 0.9.

[0138] Based on the fused sub-word score P total The identification result is used as the answer text.

[0139] The speech recognition model provided in this embodiment, by adding a keyword incentive module, can incentivize the sub-word scores of keywords in the reference answer, thereby increasing the probability that the recognition result hits the keywords in the reference answer and avoiding the impact of homophone recognition differences on subsequent evaluation results.

[0140] The input to the keyword activation module mentioned above includes the feature encoding of each keyword in the reference answer. Considering that conventional speech recognition models are prone to confusion with homophones, mainly because they do not consider pronunciation information, the feature encoding of keywords in this embodiment may include the feature encoding of subwords (hotword_sub) and the phoneme-level encoding of subwords (hotword_phn).

[0141] In one possible implementation, the words in the reference answer are first deduplicated, and then the following processing steps are performed on each deduplicated word:

[0142] 1) Obtain the subword sequence of a word;

[0143] 2) Record the length and position of each sub-word:

[0144] 3) Obtain the pronunciation sequence of the word and its corresponding subword pronunciation sequence;

[0145] 4) Record the length of the pronunciation sequence of each sub-word and the position of the pronunciation sequence of the sub-word.

[0146] The above sub-word sequence, sub-word length, and sub-word position are concatenated together to form the sub-word feature code hotword_sub. The above sub-word pronunciation sequence, sub-word pronunciation sequence length, and sub-word pronunciation sequence position are concatenated together to form the sub-word phoneme-level code hotword_phn.

[0147] Understandably, for the different data splicing processes described above, padding can be set using a specific value, such as -1, to maintain dimensional consistency during data splicing.

[0148] like Figure 5 As shown, an optional component structure for a speech recognition model is provided, which illustrates the optional component structure of the decoder and the keyword activation module.

[0149] For the keyword activation module, when performing feature processing on the feature encoding hotword_sub and the phoneme-level encoding hotword_phn of subwords, Conformer can be used for feature encoding. Conformer performs better than LSTM.

[0150] Furthermore, the keyword activation module can also include a RAM (Real Additive Margin)-softmax layer. Traditional softmax methods only consider the class information of data points during training, ignoring the distance relationships between different data points. RAM-softmax builds upon this by introducing the concept of marginal distance. By adding margins, it increases the classification robustness of data points; simply put, it increases the distance between data points of the same class and reduces the distance between data points of different classes, thus improving classification accuracy.

[0151] In some embodiments of this application, considering that in actual oral assessment scenarios, it is necessary to assess all candidates participating in the oral exam, i.e., there are multiple candidates to be assessed. Traditional methods require sequential recognition of each candidate's answer audio, which severely reduces the overall efficiency of the oral assessment. This embodiment provides a solution:

[0152] In this embodiment, a recognition core is configured for each examinee, and a reasoning core corresponding to the multiple recognition cores is also configured. The reasoning core deploys a speech recognition model, which may include an encoder and a decoder.

[0153] The process of identifying the text of each candidate's answer audio can include:

[0154] The input features of the speech recognition model are constructed using the recognition core corresponding to each candidate. The inference sequence number of the recognition core is marked and sent to the inference core. The inference result returned by the inference core is received, and the final recognized answer text is output based on the inference result.

[0155] The inference core obtains the input features carrying the inference sequence number sent by each recognition core, and concatenates the input features from multiple channels according to the inference sequence number. The data is then sent into the speech recognition model for inference, and the inference result is returned to the corresponding recognition core according to the inference sequence number.

[0156] Combination Figure 6 It demonstrates the process of recognizing the audio of each candidate's answer through a multi-channel recognition core and a single-channel reasoning core.

[0157] The input features of the speech recognition model are constructed using the recognition core corresponding to each examinee, and the inference sequence number of the recognition core is marked and sent to the inference core. After detecting the decoder output (the probability score of the recognition result at the current moment) returned by the inference core, the set search algorithm is executed. Figure 6Taking the BeamSearch algorithm as an example, it searches for the recognition result path based on the probability score of the recognition result at the current moment according to the set search algorithm, and determines whether the set search algorithm should stop searching, that is, whether the search stopping condition has been met. If not, the output of the search algorithm is constructed and sent to the inference core for the decoder of the speech recognition model to infer again. If yes, the final recognized answer text is output.

[0158] The inference core obtains the input features carrying inference sequence numbers sent by each recognition core, and concatenates the multi-channel data according to the inference sequence number. That is, it concatenates the input features from multiple channels and sends them to the encoder in the speech recognition model for inference. The encoder output is recorded according to the inference sequence number. The encoder output is concatenated again and sent to the decoder for inference. The current recognition result probability score is recorded by the decoder according to the inference sequence number, and the decoder output is returned to the corresponding recognition core according to the inference sequence number.

[0159] Using the system configuration of a multi-channel recognition core and a single-channel inference core provided in this embodiment, the recognition process of audio data from multiple different examinees can be executed in parallel by the multi-channel recognition core. After the multi-channel input features are concatenated, the inference core performs inference. Given the characteristics of matrix operations in deep learning, the number of batches can be increased to improve the overall computational efficiency. The inference core grasps the original position of the data according to the inference sequence number, performs multi-batch inference, and returns the inference result to the recognition core with the corresponding inference sequence number, thus improving the overall computational efficiency.

[0160] In another possible implementation, if the speech recognition model deployed in the inference core adopts... Figure 5 The architecture shown includes an encoder, a decoder, and a keyword activation module. Therefore, the input features of the speech recognition model built in the recognition core can include: the acoustic features of the answer audio (as input features to the encoder), and the features of each keyword in the reference answer (as input features to the keyword activation module).

[0161] Correspondingly, in the inference core, multiple acoustic features are concatenated according to the inference sequence number, and multiple keyword features are concatenated. The concatenated acoustic feature result is sent to the encoder for inference, and the concatenated keyword feature result is sent to the keyword excitation module for inference. The outputs of the encoder and the keyword recording module are recorded according to the inference sequence number. The encoder output is concatenated again and sent to the decoder for inference. The decoder output is recorded according to the inference sequence number, and the decoder output and the keyword recording module output are weighted and fused according to the inference sequence number to obtain the probability score of the recognition result at the current time. This score is then returned to the corresponding recognition core according to the inference sequence number so that the recognition core can search for the recognition result path using the search algorithm.

[0162] In some embodiments of this application, the calibration model obtained in the aforementioned step S120 is described.

[0163] In one possible implementation, the calibration model may include: a feature extraction model for each examinee, and a regression model.

[0164] The feature extraction model for each candidate is a model trained using the candidate's answer text in an unsupervised manner. The feature extraction model is used to extract the text features of the corresponding candidate's answer text.

[0165] For example, the feature extraction model can employ a Restricted Boltzmann Machine (RBM), an autoencoder, or other types of neural network models. Taking the RBM model as an example:

[0166] The uni / bi / tri ternary modeling unit can be used to train the RBM model corresponding to each candidate's answer text in an unsupervised manner.

[0167] The regression model in the calibration model is trained by taking the textual features and inference scores of the calibration candidates' answer texts as input and the expert scores of the calibration candidates as output.

[0168] In other words, for the answer text of the benchmarked candidates, the text features of the answer text are first extracted through the feature extraction model corresponding to the candidates. The text features of the benchmarked candidates and their inference scores are used as input. The expert scores of the benchmarked candidates are predicted through the regression model, and the final score results are obtained.

[0169] In one possible implementation, there can be only one regression model. This regression model can be trained using the textual features and inference scores of the benchmarked candidates' answer texts as training samples and the expert scores of the benchmarked candidates as sample labels.

[0170] In another possible implementation, the regression model may include a first regression model and a second regression model.

[0171] The first regression model is trained by using the textual features of the test takers' answers as input and the expert scores of the test takers as output.

[0172] The second regression model is trained using the expert ratings predicted by the first regression model for the calibrated candidates and the inferred ratings of the calibrated candidates as inputs, and the expert ratings of the calibrated candidates as outputs.

[0173] By setting up a first and a second regression model, the first regression model predicts a coarse range of expert scores based on the textual features of the candidate's answer text, and the second regression model predicts a more accurate expert score based on the coarse range of expert scores and inference scores predicted by the first regression model. This can improve the accuracy of the scoring results.

[0174] The first regression model can employ various types of regression models, such as the ridge regression model.

[0175] The number of secondary regression models can be one or more, and they can also be of various types. For example, secondary regression models include three different types of regression models: linear regression model, decision tree model, and support vector machine model.

[0176] The scoring results are then predicted using each type of secondary regression model, and the prediction results of each type of secondary regression model are combined, such as by averaging, to obtain the final scoring result.

[0177] In one possible implementation, taking a calibration model that includes a feature extraction model, a first regression model, and a second regression model as an example, step S130, the process of scoring each candidate using the calibration model based on their answer text and inference score to obtain the final score for each candidate, may include:

[0178] For each candidate:

[0179] The candidate's answer text is fed into the corresponding feature extraction model to extract the text features of the candidate's answer text.

[0180] The textual features of the candidate's answer text are fed into the first regression model to obtain the first score output.

[0181] The first score and the candidate's reasoning score are fed into the second regression model to obtain the final output score.

[0182] Reference Figure 7 , Figure 7 A flowchart of a spoken language assessment method provided in this application embodiment.

[0183] Oral assessment methods can be divided into three stages: pre-calibration processing, calibration process, and post-calibration processing.

[0184] Pre-calibration processing:

[0185] Collect all candidates' answer data, including questions, reference answers, and audio recordings. The answer data is analyzed using the test paper parsing module to obtain the questions and reference answers. A silence detection module detects non-silent segments in the answer audio, and an audio feature extraction module extracts the acoustic features of these non-silent segments, which are then fed into the recognition module.

[0186] The recognition module can utilize the input acoustic features and combine them with keywords in the reference answer to perform recognition using a speech recognition model, thereby obtaining the recognized answer text.

[0187] Combining the candidates' answer texts and answer data, the reasoning scoring module generates the candidates' reasoning scores. The initial assessment results for all candidates consist of their reasoning scores and answer texts.

[0188] During the calibration process:

[0189] Selected candidates for expert evaluation are chosen from all candidates. Experts score the answers of these selected candidates, resulting in expert scores. The final evaluation result for each selected candidate comprises their expert scores, answer texts, and reasoning scores.

[0190] From the initial evaluation results of all candidates, the answer text of each candidate is extracted, and the RBM model corresponding to each candidate's answer text is trained using an unsupervised training method.

[0191] For the benchmarked candidates, the expert scores of the benchmarked candidates are extracted, and the text features of the benchmarked candidates' answer texts are extracted using the RBM model corresponding to the answer texts of the benchmarked candidates. The ridge regression model is trained in a supervised manner using the text features and expert scores.

[0192] Furthermore, for the calibrated test takers, the scores and inference scores output by the ridge regression model are used as inputs, and the expert scores are used as outputs to train a supervised three-regression model. Here, the three-regression model can include three different types of regression models, such as linear regression models, decision tree models, and support vector machine models.

[0193] Save the RBM model, ridge regression model, and triple regression model corresponding to each candidate after the above training as a calibration model.

[0194] Post-calibration processing:

[0195] For each candidate to be evaluated (since the benchmark candidates have already been scored by experts, all candidates excluding the benchmark candidates can be considered as candidates to be evaluated), RBM features are extracted using the RBM model and fed into the Ridge Regression model to predict the first score. Further, the first score and the candidate's inference score are fed into a three-regression model to obtain the output scores of the three regression models. The average of these three output scores is then used as the candidate's final score.

[0196] The oral language assessment device provided in the embodiments of this application is described below. The oral language assessment device described below can be referred to in correspondence with the oral language assessment method described above.

[0197] See Figure 8 , Figure 8 This is a schematic diagram of the structure of a spoken language assessment device disclosed in an embodiment of this application.

[0198] like Figure 8 As shown, the device may include:

[0199] Answer data acquisition unit 11 is used to acquire the candidate's answer data, which includes questions, audio recordings of the candidate's answers, and reference answers;

[0200] The audio recognition unit 12 is used to recognize the answer text corresponding to the answer audio;

[0201] Reasoning scoring unit 13 is used to combine the answer text and the answer data to obtain the candidate's reasoning score through a configured reasoning scoring model;

[0202] The calibration model acquisition unit 14 is used to acquire the configured calibration model, which is pre-trained based on the answer text of the calibration candidates, the reasoning scores of the calibration candidates, and the expert scores. The calibration candidates are a portion of the candidates selected from all candidates participating in this oral exam.

[0203] The final scoring unit 15 is used to score each candidate's answer text and reasoning score using the calibration model to obtain the final score for each candidate.

[0204] In one possible implementation, the reasoning scoring model includes a pronunciation scoring model and a completeness scoring model. The process by which the reasoning scoring unit combines the answer text and the answer data to obtain the examinee's reasoning score through the configured reasoning scoring model includes:

[0205] The pronunciation scoring model is used to process the answer audio to obtain the pronunciation score output by the model;

[0206] The completeness scoring model is used to process the question, the reference answer, and the answer text to obtain the completeness score output by the model.

[0207] The candidate's reasoning score is composed of the pronunciation score and the completeness score.

[0208] In one possible implementation, when the reference answer includes two or more, the reasoning scoring unit processes the question, the reference answer, and the answer text using the completeness scoring model to obtain the completeness score output by the model, including:

[0209] Calculate the similarity between the answer text and each of the reference answers, and select the target reference answer with the highest similarity.

[0210] The completeness scoring model is used to process the question, the target reference answer, and the answer text to obtain the completeness score output by the model.

[0211] In one possible implementation, the apparatus of this application may further include:

[0212] The qualifying candidate selection unit is used to select qualifying candidates from all candidates who participated in this speaking test. This process includes:

[0213] All candidates who participated in this oral exam were ranked according to their reasoning scores.

[0214] Select a portion of the qualified candidates from each of the established scoring intervals to form the complete list of qualified candidates.

[0215] In one possible implementation, the calibration model includes: a feature extraction model for each examinee, and a regression model;

[0216] The feature extraction model for each candidate is a model trained using the candidate's answer text in an unsupervised training manner. The feature extraction model is used to extract the text features of the corresponding candidate's answer text.

[0217] The regression model is trained using the textual features and inference scores of the benchmarked candidates' answer texts as input and the expert scores of the benchmarked candidates as output.

[0218] In one possible implementation, the regression model includes: a first regression model and a second regression model;

[0219] The first regression model is trained by taking the text features of the test-takers' answer texts as input and the expert scores of the test-takers as output.

[0220] The second regression model is trained using the expert ratings predicted by the first regression model for the calibrated candidates and the inference ratings of the calibrated candidates as inputs, and the expert ratings of the calibrated candidates as outputs.

[0221] Based on this, the final scoring unit uses the calibration model to score each candidate's answer text and reasoning score, and obtains the final score for each candidate, including:

[0222] For each candidate:

[0223] The candidate's answer text is fed into the corresponding feature extraction model to extract the text features of the candidate's answer text;

[0224] The text features of the candidate's answer text are fed into the first regression model to obtain the output first score;

[0225] The first score and the candidate's reasoning score are fed into the second regression model to obtain the final output score.

[0226] In one possible implementation, the process by which the audio recognition unit identifies the answer text corresponding to the answer audio includes:

[0227] Extract the acoustic features of the answer audio;

[0228] The acoustic features are input into a configured speech recognition model, which includes an encoder, a decoder, and a keyword activation module. The keyword activation module encodes the features of each keyword in the reference answer, predicts the sub-word score of each keyword, performs a weighted fusion of the sub-word scores predicted by the decoder and the sub-word scores predicted by the keyword activation module, and determines the recognition result based on the fused sub-word scores, which is used as the answer text.

[0229] In one possible implementation, when there are multiple candidates to be evaluated, a recognition core is configured for each candidate, and an inference core corresponding to the multiple recognition cores is configured. The process by which the audio recognition unit recognizes the answer text corresponding to each candidate's answer audio includes:

[0230] The input features of the speech recognition model are constructed using the recognition core corresponding to each candidate. The inference sequence number of the recognition core is marked and sent to the inference core. The inference result returned by the inference core is received, and the final recognized answer text is output based on the inference result.

[0231] The inference core obtains the input features carrying inference sequence numbers sent by each of the recognition cores, concatenates the input features from multiple channels according to the inference sequence number, sends them into the speech recognition model for inference, and returns the inference result to the corresponding recognition core according to the inference sequence number.

[0232] This application also provides an electronic device in its embodiments. (See reference...) Figure 9 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as tablet computers, learning machines, teaching terminals, laptops, etc. Figure 9 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0233] like Figure 9 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603 to implement the spoken language assessment method of the foregoing embodiments of this application. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0234] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, memory cards, hard drives, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 9 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0235] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the spoken language assessment methods provided in this application.

[0236] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the oral assessment methods provided in this application.

[0237] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0238] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0239] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0240] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0241] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

Claims

1. A method for oral language assessment, characterized in that, include: The system acquires the candidates' answer data, which includes questions, audio recordings of the candidates' answers, and reference answers. Identify the answer text corresponding to the answer audio, and combine the answer text and the answer data to obtain the candidate's reasoning score through the configured reasoning scoring model; Obtain the configured calibration model, which is pre-trained based on the answer texts of the calibration candidates, the reasoning scores of the calibration candidates, and the expert scores. The calibration candidates are a portion of the candidates selected from all candidates participating in this oral exam. Based on each candidate's answer text and reasoning score, the calibration model is used to score each candidate and obtain their final score. The calibration model includes: a feature extraction model for each examinee, and a regression model; The feature extraction model for each candidate is a model trained using the candidate's answer text in an unsupervised training manner. The feature extraction model is used to extract the text features of the corresponding candidate's answer text. The regression model is trained using the textual features and inference scores of the benchmarked candidates' answer texts as input and the expert scores of the benchmarked candidates as output.

2. The method according to claim 1, characterized in that, The reasoning scoring model includes a pronunciation scoring model and a completeness scoring model; The process of obtaining the candidate's reasoning score through a configured reasoning scoring model, combining the answer text and the answer data, includes: The pronunciation scoring model is used to process the answer audio to obtain the pronunciation score output by the model; The completeness scoring model is used to process the question, the reference answer, and the answer text to obtain the completeness score output by the model. The candidate's reasoning score is composed of the pronunciation score and the completeness score.

3. The method according to claim 2, characterized in that, When there are two or more reference answers, the process of using the completeness scoring model to process the question, the reference answers, and the answer text to obtain the completeness score output by the model includes: Calculate the similarity between the answer text and each of the reference answers, and select the target reference answer with the highest similarity. The completeness scoring model is used to process the question, the target reference answer, and the answer text to obtain the completeness score output by the model.

4. The method according to claim 1, characterized in that, The process of selecting qualified candidates from all candidates who participated in this oral exam includes: All candidates who participated in this oral exam were ranked according to their reasoning scores. Select a portion of the qualified candidates from each of the established scoring intervals to form the complete list of qualified candidates.

5. The method according to claim 1, characterized in that, The regression model includes: a first regression model and a second regression model; The first regression model is trained using the text features of the test-takers' answer texts as input and the expert scores of the test-takers as output. The second regression model is trained using the expert ratings predicted by the first regression model for the calibrated candidates and the inference ratings of the calibrated candidates as inputs, and the expert ratings of the calibrated candidates as outputs.

6. The method according to claim 5, characterized in that, The process of obtaining the final score for each candidate by scoring them using the calibration model based on their answer text and reasoning score includes: For each candidate: The candidate's answer text is fed into the corresponding feature extraction model to extract the text features of the candidate's answer text; The text features of the candidate's answer text are fed into the first regression model to obtain the output first score; The first score and the candidate's reasoning score are fed into the second regression model to obtain the final output score.

7. The method according to claim 1, characterized in that, The process of identifying the answer text corresponding to the answer audio includes: Extract the acoustic features of the answer audio; The acoustic features are input into a configured speech recognition model, which includes an encoder, a decoder, and a keyword activation module. The keyword activation module encodes the features of each keyword in the reference answer, predicts the sub-word score of each keyword, performs a weighted fusion of the sub-word scores predicted by the decoder and the sub-word scores predicted by the keyword activation module, and determines the recognition result based on the fused sub-word scores, which is used as the answer text.

8. The method according to claim 1, characterized in that, When there are multiple candidates to be evaluated, a recognition core is configured for each candidate, and a reasoning core corresponding to the multiple recognition cores is configured. The process of identifying the text of each candidate's answer audio includes: The input features of the speech recognition model are constructed using the recognition core corresponding to each candidate. The inference sequence number of the recognition core is marked and sent to the inference core. The inference result returned by the inference core is received, and the final recognized answer text is output based on the inference result. The inference core obtains the input features carrying inference sequence numbers sent by each of the recognition cores, concatenates the input features from multiple channels according to the inference sequence number, sends them into the speech recognition model for inference, and returns the inference result to the corresponding recognition core according to the inference sequence number.

9. A spoken language assessment device, characterized in that, include: The answer data acquisition unit is used to acquire the candidate's answer data, which includes questions, audio recordings of the candidate's answers, and reference answers. An audio recognition unit is used to recognize the answer text corresponding to the answer audio. The reasoning scoring unit is used to combine the answer text and the answer data to obtain the candidate's reasoning score through a configured reasoning scoring model; The calibration model acquisition unit is used to acquire the configured calibration model, which is pre-trained based on the answer text of the calibration candidates, the reasoning scores of the calibration candidates, and the expert scores. The calibration candidates are a portion of the candidates selected from all candidates participating in this oral exam. The final scoring unit is used to score each candidate's answer text and reasoning score using the calibration model to obtain the final score for each candidate. The calibration model includes: a feature extraction model for each examinee, and a regression model; The feature extraction model for each candidate is a model trained using the candidate's answer text in an unsupervised training manner. The feature extraction model is used to extract the text features of the corresponding candidate's answer text. The regression model is trained using the textual features and inference scores of the benchmarked candidates' answer texts as input and the expert scores of the benchmarked candidates as output.

10. An electronic device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the oral assessment method as described in any one of claims 1 to 8.

11. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements each step of the oral assessment method as described in any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the oral assessment method as described in any one of claims 1 to 8.

Images