Reading question evaluation method and related device

Abstract

The invention discloses a reading question evaluation method and a related device, and relates to the technical field of spoken language evaluation, and the method comprises the steps: after obtaining the acoustic features of the audio frequency of a reading question to be evaluated and the phoneme sequence of the reading question, not carrying out the segmentation, and obtaining the phoneme sequence of the reading question; the acoustic features of the reading question audio and the phoneme sequence of the reading question are input into the end-to-end reading question evaluation model to obtain a model evaluation score; and finally, based on the model evaluation score, determining a final evaluation score of the reading question audio. As the task of predicting the position of the phoneme in the audio is added in the training process of the end-to-end reading question evaluation model to relieve the problem that the audio phoneme is difficult to align, the scheme can realize the evaluation of the reading question on the premise of not segmenting the audio data and can improve the accuracy of the evaluation result.

Description

Technical Field

[0001] The present application relates to the technical field of oral language assessment, and in particular to a reading question assessment method and related devices. Background Art

[0002] As an important medium of interpersonal communication, spoken language plays an extremely important role in real life. With the continuous development of social economy and the intensification of globalization, people have put forward higher and higher requirements for the efficiency of language learning and the objectivity, fairness and large-scale testing of language assessment. To this end, the industry has successively developed a number of oral language teaching and oral assessment systems. Reading aloud questions are one of the common question types in oral assessment. Reading aloud questions require users to read preset texts to examine the standard degree of users' pronunciation of basic phonetic units and the fluency of sentence reading. The accuracy of reading aloud scoring is crucial.

[0003] In traditional oral assessment systems, the assessment method for reading aloud questions is usually divided into two stages: the first stage is model feature extraction, and the second stage is to train a regression model based on the extracted features and manual scores, and regress the machine scores from the data to be scored during testing. The whole process is generally as follows: first receive the student audio data and the corresponding test paper text; then extract the acoustic features of the audio data and obtain the phoneme sequence of the test paper text; then, based on the extracted acoustic features and phoneme sequences, segment the audio features and their corresponding phoneme sequences according to the sentences; send the segmented audio features and phoneme sequences to the end-to-end scoring model to obtain the scoring features; combine the obtained scoring features into the scoring features of the entire audio data; finally, use the scoring features and manual scores to train the regression model, and regress the machine scores based on the scoring features and regression model during testing.

[0004] However, when the audio data to be evaluated has a low signal-to-noise ratio, multiple answers, poor pronunciation levels, etc., it is easy to make errors in segmenting the audio data to be evaluated. Errors in audio data segmentation may result in the segmented audio data not matching its corresponding phoneme sequence, and valid audio information may also be discarded, resulting in low accuracy in the evaluation results of the reading questions.

[0005] Therefore, how to provide a reading question evaluation method that can evaluate reading questions without segmenting audio data and improve the accuracy of the evaluation results has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0006] In view of the above problems, the present application provides a reading question evaluation method and related devices to achieve the purpose of evaluating reading questions without segmenting audio data and improving the accuracy of the evaluation results. The specific scheme is as follows:

[0007] The first aspect of the present application provides a reading question evaluation method, comprising:

[0008] Acquire the acoustic features of the audio of the reading question to be evaluated and the phoneme sequence of the reading question title;

[0009] Inputting the acoustic features of the reading question audio and the phoneme sequence of the reading question title into an end-to-end reading question evaluation model to obtain a model evaluation score; the end-to-end reading question evaluation model is obtained by joint training based on a task of predicting a model evaluation score of audio and a task of predicting a phoneme position in audio;

[0010] Based on the model evaluation score, a final evaluation score of the reading question audio is determined.

[0011] In a possible implementation, determining a final evaluation score of the reading question audio based on the model evaluation score includes:

[0012] Determining an evaluation standard for the reading aloud question, wherein the evaluation standard for the reading aloud question includes a correspondence between a model evaluation score and a manual evaluation score for the reading aloud question;

[0013] Based on the scoring criteria for the reading aloud question, the manual evaluation score corresponding to the model evaluation score in the scoring criteria for the reading aloud question is determined as the final evaluation score of the reading aloud question audio.

[0014] In one possible implementation, the end-to-end reading question evaluation model includes: an audio encoder, a phoneme encoder, an audio-phoneme joint encoder, and a model evaluation score output layer; wherein the audio encoder adopts an offset window attention mechanism.

[0015] In a possible implementation, inputting the feature data of the reading question audio and the feature data of the reading question title into an end-to-end reading question evaluation model to obtain a model evaluation score includes:

[0016] The audio encoder receives the acoustic features of the audio of the reading question to be evaluated and encodes it using an offset window attention mechanism to obtain an audio encoding result;

[0017] The phoneme encoder receives and encodes the phoneme sequence of the reading question to obtain a phoneme encoding result;

[0018] The audio-phoneme joint encoder jointly encodes the audio encoding result and the phoneme encoding result to obtain an audio-phoneme joint encoding result;

[0019] The model evaluation score output layer outputs the model evaluation score based on the audio phoneme joint encoding result.

[0020] In a possible implementation, the end-to-end reading question evaluation model is trained as follows:

[0021] Determine training data, wherein the training data includes phoneme recognition training data, reading pronunciation error detection training data, and reading scoring training data;

[0022] Taking the first end-to-end reading aloud question evaluation model as a basic model, pre-training the first end-to-end reading aloud question evaluation model based on the phoneme recognition training data and the reading aloud pronunciation error detection training data to obtain a second end-to-end reading aloud question evaluation model;

[0023] The second end-to-end reading aloud question evaluation model is used as a basic model, and the second end-to-end reading aloud question evaluation model is fine-tuned based on the reading score training data to obtain a trained end-to-end reading aloud question evaluation model.

[0024] In a possible implementation, determining the evaluation criteria for the reading question title includes:

[0025] Obtaining benchmark evaluation standard data corresponding to the reading question, wherein the benchmark evaluation standard data includes a plurality of reading question audio samples, and a manual evaluation score and a model evaluation score of each of the reading question audio samples;

[0026] Based on the difference in manual evaluation scores of the audio samples of the reading questions, the benchmark evaluation standard data is expanded to obtain expanded evaluation standard data;

[0027] Filtering the expanded evaluation standard data to obtain remaining expanded evaluation standard data;

[0028] Based on the baseline evaluation standard data and the remaining expanded evaluation standard data, the evaluation standard of the reading-aloud question is determined.

[0029] A second aspect of the present application provides a reading question evaluation device, comprising:

[0030] An acquisition unit, used to acquire the acoustic features of the audio of the reading question to be evaluated and the phoneme sequence of the reading question title;

[0031] A model evaluation score determination unit is used to input the acoustic features of the reading question audio and the phoneme sequence of the reading question title into an end-to-end reading question evaluation model to obtain a model evaluation score; the end-to-end reading question evaluation model is obtained by joint training based on a task of predicting a model evaluation score of audio and a task of predicting a position of a phoneme in audio;

[0032] The final evaluation score determination unit is used to determine the final evaluation score of the reading question audio based on the model evaluation score.

[0033] In a possible implementation, the final evaluation score determination unit is specifically configured to:

[0034] Determining an evaluation standard for the reading aloud question, wherein the evaluation standard for the reading aloud question includes a correspondence between a model evaluation score and a manual evaluation score for the reading aloud question;

[0035] Based on the scoring criteria for the reading aloud question, the manual evaluation score corresponding to the model evaluation score in the scoring criteria for the reading aloud question is determined as the final evaluation score of the reading aloud question audio.

[0036] In one possible implementation, the end-to-end reading question evaluation model includes: an audio encoder, a phoneme encoder, an audio-phoneme joint encoder, and a model evaluation score output layer; wherein the audio encoder adopts an offset window attention mechanism.

[0037] In a possible implementation, the model evaluation score determination unit is specifically used to:

[0038] The audio encoder receives the acoustic features of the audio of the reading question to be evaluated and encodes it using an offset window attention mechanism to obtain an audio encoding result;

[0039] The phoneme encoder receives and encodes the phoneme sequence of the reading question to obtain a phoneme encoding result;

[0040] The audio-phoneme joint encoder jointly encodes the audio encoding result and the phoneme encoding result to obtain an audio-phoneme joint encoding result;

[0041] The model evaluation score output layer outputs the model evaluation score based on the audio phoneme joint encoding result.

[0042] In a possible implementation, the device further includes an end-to-end reading aloud question evaluation model training unit, and the end-to-end reading aloud question evaluation model training unit is specifically used to:

[0043] Determine training data, wherein the training data includes phoneme recognition training data, reading pronunciation error detection training data, and reading scoring training data;

[0044] Taking the first end-to-end reading aloud question evaluation model as a basic model, pre-training the first end-to-end reading aloud question evaluation model based on the phoneme recognition training data and the reading aloud pronunciation error detection training data to obtain a second end-to-end reading aloud question evaluation model;

[0045] The second end-to-end reading aloud question evaluation model is used as a basic model, and the second end-to-end reading aloud question evaluation model is fine-tuned based on the reading score training data to obtain a trained end-to-end reading aloud question evaluation model.

[0046] In a possible implementation, the device further includes a reading-aloud question title evaluation standard determination unit, and the reading-aloud question title evaluation standard determination unit is specifically configured to:

[0047] Obtaining benchmark evaluation standard data corresponding to the reading question, wherein the benchmark evaluation standard data includes a plurality of reading question audio samples, and a manual evaluation score and a model evaluation score of each of the reading question audio samples;

[0048] Based on the difference in manual evaluation scores of the audio samples of the reading questions, the benchmark evaluation standard data is expanded to obtain expanded evaluation standard data;

[0049] Filtering the expanded evaluation standard data to obtain remaining expanded evaluation standard data;

[0050] Based on the baseline evaluation standard data and the remaining expanded evaluation standard data, the evaluation standard of the reading-aloud question is determined.

[0051] The third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the reading question assessment method of the first aspect or any implementation of the first aspect.

[0052] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0053] The memory is used to store computer programs;

[0054] The processor is used to execute the computer program so that the electronic device can implement the reading question evaluation method of the first aspect or any implementation manner of the first aspect.

[0055] A fifth aspect of the present application provides a computer-readable storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the reading question assessment method of the above-mentioned first aspect or any implementation of the first aspect.

[0056] By means of the above technical scheme, the reading question evaluation method and related devices provided by the present application, after obtaining the acoustic features of the reading question audio to be evaluated and the phoneme sequence of the reading question title, do not perform segmentation, but input the acoustic features of the reading question audio and the phoneme sequence of the reading question title into the end-to-end reading question evaluation model to obtain the model evaluation score; finally, based on the model evaluation score, the final evaluation score of the reading question audio is determined. Since the task of predicting the position of phonemes in the audio is added during the training process of the end-to-end reading question evaluation model to alleviate the problem of difficult audio phoneme alignment, the solution can realize the evaluation of reading questions without segmenting the audio data and can improve the accuracy of the evaluation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0058] Figure 1 A flowchart of a reading question evaluation method provided in an embodiment of the present application;

[0059] Figure 2 A schematic diagram of the structure of an end-to-end reading question evaluation model provided in an embodiment of the present application;

[0060] Figure 3 A flowchart of a training method for an end-to-end reading question evaluation model provided in an embodiment of the present application;

[0061] Figure 4 A flowchart of a method for determining a final evaluation score of a reading question audio based on a model evaluation score provided in an embodiment of the present application;

[0062] Figure 5 A schematic diagram of the structure of a reading question evaluation device provided in an embodiment of the present application;

[0063] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0065] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0066] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0067] As an important medium of interpersonal communication, spoken language plays an extremely important role in real life. With the continuous development of social economy and the intensification of globalization, people have put forward higher and higher requirements for the efficiency of language learning and the objectivity, fairness and large-scale testing of language assessment. To this end, the industry has successively developed a number of spoken language teaching and oral assessment systems.

[0068] Reading aloud questions are one of the common question types in oral assessment. Reading aloud questions require users to read preset texts to test the standard of users' pronunciation of basic phonetic units and the fluency of sentence reading. The accuracy of reading aloud question scoring is crucial.

[0069] In traditional oral assessment systems, the assessment method for reading aloud questions is usually divided into two stages: the first stage is model feature extraction, and the second stage is to train a regression model based on the extracted features and manual scores, and regress the machine scores from the data to be scored during testing. The whole process is generally as follows: first receive the student audio data and the corresponding test paper text; then extract the acoustic features of the audio data and obtain the phoneme sequence of the test paper text; then, based on the extracted acoustic features and phoneme sequences, segment the audio features and their corresponding phoneme sequences according to the sentences; send the segmented audio features and phoneme sequences to the end-to-end scoring model to obtain the scoring features; combine the obtained scoring features into the scoring features of the entire audio data; finally, use the scoring features and manual scores to train the regression model, and regress the machine scores based on the scoring features and regression model during testing.

[0070] The existing reading aloud assessment method requires the audio data to be segmented. The reason for this is that it is limited by video memory. Taking the end-to-end model based on Transformer as an example, its video memory usage is proportional to the square of the audio duration. Directly inputting the entire audio and phoneme sequence will result in excessive video memory usage, thereby reducing computing efficiency and increasing computing costs. On the other hand, there is the difficulty of aligning long audio and phonemes. Taking the end-to-end model as an example, long audio and phoneme alignment is more prone to hallucination problems than short audio, resulting in an overall higher score.

[0071] However, when the audio data to be evaluated has a low signal-to-noise ratio, multiple answers, poor pronunciation levels, etc., it is easy to make errors in segmenting the audio data to be evaluated. Errors in audio data segmentation may result in the segmented audio data not matching its corresponding phoneme sequence, and valid audio information may also be discarded, resulting in low accuracy in the evaluation results of the reading questions.

[0072] In addition, the existing reading aloud question evaluation method relies on the regression model for the final score. However, it is difficult for the regression model to output full marks and 0 points, which can easily lead to low recall rates of full marks and 0 points. Moreover, the existing reading aloud question scoring method does not take into account the differences in evaluation standards in different regions, which will cause the reading aloud question evaluation results to be inconsistent with the evaluation standards of the corresponding regions.

[0073] In order to solve the above problems, an embodiment of the present application provides a reading question evaluation method, which takes into account the differences in evaluation standards in different regions, and avoids the problem of low accuracy of evaluation results caused by audio data segmentation errors by not segmenting the audio data to be evaluated. This can improve the accuracy of the evaluation results, and can also improve the full score recall rate and the 0 score recall rate.

[0074] The following is a detailed introduction to the reading question evaluation method of the embodiment of the present application in conjunction with the accompanying drawings.

[0075] Reference Figure 1 , Figure 1 A flowchart of a reading question evaluation method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, a reading question evaluation method provided in an embodiment of the present application may include the following steps, which are described in detail below.

[0076] S101: Acquire the acoustic features of the audio of the reading question to be evaluated and the phoneme sequence of the reading question title;

[0077] In the present application, the audio of the reading question to be evaluated can be recorded by a recording device in the examination room, and the recording device can be a head-mounted microphone. The title of the reading question can be obtained from an oral evaluation test paper.

[0078] In the present application, the acoustic features of the audio data of the reading question are generally the spectral features of the audio data of the reading question, such as Mel Frequency Cepstrum Coefficient (MFCC) features or Perceptual Linear Predictive (PLP) features, etc. When extracting, it is necessary to first perform frame processing on the audio data of the reading question, then pre-emphasize the framed audio data of the reading question, and finally extract the spectral features of each frame of the audio data of the reading question in turn.

[0079] In the present application, the existing phoneme sequence extraction method can be used to extract the phonemes of the reading question, and the phonemes can be converted into vector representations using one-hot encoding or other encoding methods to finally generate the factor sequence of the reading question. This application will not elaborate on this.

[0080] S102: Input the acoustic features of the reading question audio and the phoneme sequence of the reading question title into an end-to-end reading question evaluation model to obtain a model evaluation score; the end-to-end reading question evaluation model is obtained by joint training based on a task of predicting a model evaluation score of audio and a task of predicting a position of a phoneme in audio;

[0081] Taking into account the difficulty of audio phoneme alignment that will be introduced by directly inputting the acoustic features of the complete reading question audio and the phoneme sequence of the reading question into the end-to-end reading question evaluation model, in the present application, the task of predicting the position of phonemes in the audio can be added during the training process of the end-to-end reading question evaluation model to alleviate the problem of audio phoneme alignment.

[0082] In a possible implementation, the end-to-end reading question evaluation model is obtained by using the acoustic features of the reading question audio samples and the phoneme sequences of the reading question samples as training samples, and the positions of each phoneme in the phoneme sequence of the reading question samples in the audio of the reading question audio samples and the manual evaluation scores of the reading question audio samples as labels for training; since it takes a lot of manpower and time to label the positions of each phoneme in the phoneme sequence of the reading question samples in the audio of the reading question audio samples, in the present application, existing tools can be used to generate the positions of phonemes in the audio as pseudo-labels for training.

[0083] S103: Determine a final evaluation score of the reading question audio based on the model evaluation score.

[0084] In the present application, in a possible implementation, based on the model evaluation score, determining the final evaluation score of the reading question audio includes: using the model evaluation score as the final evaluation score of the reading question audio.

[0085] Considering that it is difficult for the model to output full marks and 0 marks, it is easy to cause the recall rate of full marks and 0 marks to be low. In addition, the difference in evaluation standards in different regions will cause the evaluation results of reading aloud questions to be inconsistent with the evaluation standards of the corresponding regions.

[0086] In another possible implementation, the manual evaluation score corresponding to the model evaluation score may be determined as the final evaluation score of the reading question audio, which will be described in detail in the following embodiments.

[0087] The reading question evaluation method provided in this embodiment does not segment the audio of the reading question to be evaluated and the phoneme sequence of the reading question title after obtaining the acoustic features of the audio of the reading question to be evaluated and the phoneme sequence of the reading question title, but inputs the acoustic features of the audio of the reading question and the phoneme sequence of the reading question title into the end-to-end reading question evaluation model to obtain the model evaluation score; finally, based on the model evaluation score, the final evaluation score of the reading question audio is determined. Since the task of predicting the position of phonemes in the audio is added during the training process of the end-to-end reading question evaluation model to alleviate the problem of difficult audio phoneme alignment, this solution can realize the evaluation of reading questions without segmenting the audio data and can improve the accuracy of the evaluation results.

[0088] In this application, the structure of the end-to-end reading question evaluation model can be an encoder-only structure, wherein the encoder can be an encoder in Transformer. Figure 2 , Figure 2 A schematic diagram of the structure of an end-to-end reading question evaluation model provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the end-to-end reading question evaluation model includes: an audio encoder, a phoneme encoder, an audio-phoneme joint encoder, and a model evaluation score output layer; wherein the audio encoder adopts an offset window attention mechanism.

[0089] In the present application, since the present scheme does not split the audio feature data, in order to alleviate the problem of excessive memory usage when processing long audio feature data, in the present scheme, the Swin Transformer structure can be used as a reference to change the attention mechanism in the audio encoder to a shifted windowed attention mechanism, thereby reducing the complexity of memory usage from the square of the audio feature data to linear.

[0090] In the present application, the audio encoder is used to receive the acoustic features of the audio of the reading question to be evaluated and encode it using an offset window attention mechanism to obtain an audio encoding result;

[0091] The phoneme encoder is used to receive and encode the phoneme sequence of the reading question to obtain a phoneme encoding result;

[0092] The audio-phoneme joint encoder is used to receive the audio encoding result and the phoneme encoding result, and jointly encode the audio encoding result and the phoneme encoding result to obtain the audio-phoneme joint encoding result;

[0093] The model evaluation score output layer is used to receive the audio phoneme joint encoding result and output the model evaluation score based on the audio phoneme joint encoding result.

[0094] Then, inputting the feature data of the reading question audio and the feature data of the reading question title into the end-to-end reading question evaluation model to obtain the model evaluation score includes:

[0095] The audio encoder receives the acoustic features of the audio of the reading question to be evaluated and encodes it using an offset window attention mechanism to obtain an audio encoding result;

[0096] The phoneme encoder receives and encodes the phoneme sequence of the reading question to obtain a phoneme encoding result;

[0097] The audio-phoneme joint encoder jointly encodes the audio encoding result and the phoneme encoding result to obtain an audio-phoneme joint encoding result;

[0098] The model evaluation score output layer outputs the model evaluation score based on the audio phoneme joint encoding result.

[0099] Taking into account the difficulty of obtaining audio samples of reading questions with manual evaluation scores, in order to further reduce the reliance on audio samples of reading questions with manual evaluation scores, further, in this application, a multi-layer pre-training method can be used to train an end-to-end reading question evaluation model.

[0100] In another embodiment of the present application, the training method of the end-to-end reading question evaluation model is described in detail, referring to Figure 3 , Figure 3 A flow chart of a training method for an end-to-end reading question evaluation model provided in an embodiment of the present application. The method may include the following steps:

[0101] S201: Determine training data, wherein the training data includes phoneme recognition training data, reading pronunciation error detection training data, and reading scoring training data;

[0102] In the present application, the phoneme recognition training data includes audio samples and corresponding text samples, the labels are phoneme labels, and the phoneme labels are obtained based on the corresponding text samples. The reading pronunciation error detection training data includes audio samples and corresponding text samples, and the labels are the pronunciation error detection results of each phoneme position. The reading scoring training data includes the acoustic features of the reading question audio samples and the phoneme sequence of the reading question sample, and the labels are the positions of each phoneme in the phoneme sequence of the reading question sample in the audio of the reading question audio sample and the manual evaluation score of the reading question audio sample.

[0103] S202: using the first end-to-end reading aloud question evaluation model as a basic model, pre-training the first end-to-end reading aloud question evaluation model based on the phoneme recognition training data and the reading aloud pronunciation error detection training data to obtain a second end-to-end reading aloud question evaluation model;

[0104] In the present application, the first end-to-end reading question evaluation model may be an initial end-to-end reading question evaluation model.

[0105] Specifically, taking the first end-to-end reading aloud question evaluation model as the basic model, pre-training the first end-to-end reading aloud question evaluation model based on the phoneme recognition training data and the reading aloud pronunciation error detection training data to obtain the second end-to-end reading aloud question evaluation model may include: taking the first end-to-end reading aloud question evaluation model as the basic model, pre-training the first end-to-end reading aloud question evaluation model based on the phoneme recognition training data to obtain the end-to-end reading aloud question evaluation model after phoneme recognition pre-training; taking the end-to-end reading aloud question evaluation model after phoneme recognition pre-training as the basic model, pre-training the end-to-end reading aloud question evaluation model after phoneme recognition pre-training based on the reading aloud pronunciation error detection training data to obtain the end-to-end reading aloud question evaluation model after reading aloud pronunciation error detection pre-training; the end-to-end reading aloud question evaluation model after reading aloud pronunciation error detection pre-training is the second end-to-end reading aloud question evaluation model.

[0106] S203: Using the second end-to-end reading aloud question evaluation model as a basic model, and fine-tuning the second end-to-end reading aloud question evaluation model based on the reading score training data to obtain a trained end-to-end reading aloud question evaluation model.

[0107] Specifically, the end-to-end reading question evaluation model pre-trained with reading pronunciation error detection can be used as the basic model, and the end-to-end reading question evaluation model pre-trained with reading pronunciation error detection can be fine-tuned based on the reading score training data to obtain a trained end-to-end reading question evaluation model.

[0108] It should be noted that the specific implementation method of each training process is a mature method in the existing neural network model training process, and this application will not describe it in detail.

[0109] In the above embodiment, it is mentioned that considering that it is difficult for the model to output full marks and 0 marks, it is easy to cause the full mark recall rate and the 0 mark recall rate to be low. Moreover, the difference in evaluation standards in different regions will cause the evaluation results of the reading question to be inconsistent with the evaluation standards of the corresponding region. To solve this problem, the manual evaluation score corresponding to the model evaluation score can also be determined as the final evaluation score of the reading question audio.

[0110] In another embodiment of the present application, a specific implementation method for determining the final evaluation score of the reading question audio based on the model evaluation score is described.

[0111] Reference Figure 4 , Figure 4 A flowchart of a method for determining a final evaluation score of a reading question audio based on a model evaluation score provided in an embodiment of the present application, the method may include:

[0112] S301: Determine an evaluation standard for the reading-aloud question, wherein the evaluation standard for the reading-aloud question includes a correspondence between a model evaluation score and a manual evaluation score for the reading-aloud question.

[0113] In a possible implementation, determining the evaluation criteria for the reading question title includes:

[0114] S3011: Obtaining benchmark evaluation standard data corresponding to the reading question, wherein the benchmark evaluation standard data includes a plurality of reading question audio samples, and a manual evaluation score and a model evaluation score of each of the reading question audio samples;

[0115] For each reading question audio sample, its manual evaluation score may be manually annotated, and its model evaluation score may be determined based on the end-to-end reading question evaluation model;

[0116] S3012: Expanding the benchmark evaluation standard data based on the difference in manual evaluation scores of the audio samples of the reading questions to obtain expanded evaluation standard data;

[0117] In the present application, for any two audio samples of reading questions, the manual evaluation scores and model evaluation scores of the two audio samples of reading questions are expanded based on the difference in their manual evaluation scores. In one possible implementation, if the difference in their manual evaluation scores is within a certain range, the same linear difference is performed on their manual evaluation scores and model evaluation scores respectively to obtain two expanded evaluation standard data.

[0118] S3013: Filtering the expanded evaluation standard data to obtain remaining expanded evaluation standard data;

[0119] In the present application, a linear regression model can be trained based on the manual evaluation scores and model evaluation scores of each reading question audio sample in the benchmark evaluation standard data, and the linear regression model can be used to filter out data in the expanded evaluation standard data with errors greater than a certain threshold to obtain the remaining expanded evaluation standard data.

[0120] S3014: Determine the evaluation criteria for the reading-aloud question based on the baseline evaluation criteria data and the remaining expanded evaluation criteria data.

[0121] In the present application, the manual evaluation scores and model evaluation scores in the baseline evaluation standard data and the remaining expanded evaluation standard data can be sorted in ascending or descending order to obtain the target evaluation standard, which includes the correspondence between the model evaluation score and the manual evaluation score of the reading question.

[0122] It should be noted that, for different reading aloud questions, the above method can be used to determine the evaluation criteria for the reading aloud questions.

[0123] S302: Based on the scoring criteria for the reading question, determine the manual evaluation score corresponding to the model evaluation score in the scoring criteria for the reading question as the final evaluation score of the reading question audio.

[0124] In summary, the solution of the present application avoids the risk of inaccurate scoring caused by disordered phoneme segmentation by not segmenting the audio, and solves the problem of adapting the current scoring standard based on the calibration and scoring method.

[0125] A reading question evaluation method provided in an embodiment of the present application is introduced above, and a device for executing the above reading question evaluation method will be introduced below.

[0126] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of a reading question evaluation device provided in an embodiment of the present application. Figure 5 As shown, the reading question evaluation device comprises:

[0127] An acquisition unit 11 is used to acquire the acoustic features of the audio of the reading question to be evaluated and the phoneme sequence of the reading question title;

[0128] The model evaluation score determination unit 12 is used to input the acoustic features of the reading question audio and the phoneme sequence of the reading question title into an end-to-end reading question evaluation model to obtain a model evaluation score; the end-to-end reading question evaluation model is obtained by joint training based on the task of predicting the model evaluation score of the audio and the task of predicting the position of the phoneme in the audio;

[0129] The final evaluation score determination unit 13 is used to determine the final evaluation score of the reading question audio based on the model evaluation score.

[0130] In a possible implementation, the final evaluation score determination unit is specifically configured to:

[0131] Determining an evaluation standard for the reading aloud question, wherein the evaluation standard for the reading aloud question includes a correspondence between a model evaluation score and a manual evaluation score for the reading aloud question;

[0132] Based on the scoring criteria for the reading aloud question, the manual evaluation score corresponding to the model evaluation score in the scoring criteria for the reading aloud question is determined as the final evaluation score of the reading aloud question audio.

[0133] In one possible implementation, the end-to-end reading question evaluation model includes: an audio encoder, a phoneme encoder, an audio-phoneme joint encoder, and a model evaluation score output layer; wherein the audio encoder adopts an offset window attention mechanism.

[0134] In a possible implementation, the model evaluation score determination unit is specifically used to:

[0135] The audio encoder receives the acoustic features of the audio of the reading question to be evaluated and encodes it using an offset window attention mechanism to obtain an audio encoding result;

[0136] The phoneme encoder receives and encodes the phoneme sequence of the reading question to obtain a phoneme encoding result;

[0137] The audio-phoneme joint encoder jointly encodes the audio encoding result and the phoneme encoding result to obtain an audio-phoneme joint encoding result;

[0138] The model evaluation score output layer outputs the model evaluation score based on the audio phoneme joint encoding result.

[0139] In a possible implementation, the device further includes an end-to-end reading aloud question evaluation model training unit, and the end-to-end reading aloud question evaluation model training unit is specifically used to:

[0140] Determine training data, wherein the training data includes phoneme recognition training data, reading pronunciation error detection training data, and reading scoring training data;

[0141] Taking the first end-to-end reading aloud question evaluation model as a basic model, pre-training the first end-to-end reading aloud question evaluation model based on the phoneme recognition training data and the reading aloud pronunciation error detection training data to obtain a second end-to-end reading aloud question evaluation model;

[0142] The second end-to-end reading aloud question evaluation model is used as a basic model, and the second end-to-end reading aloud question evaluation model is fine-tuned based on the reading score training data to obtain a trained end-to-end reading aloud question evaluation model.

[0143] In a possible implementation, the device further includes a reading-aloud question title evaluation standard determination unit, and the reading-aloud question title evaluation standard determination unit is specifically configured to:

[0144] Obtaining benchmark evaluation standard data corresponding to the reading question, wherein the benchmark evaluation standard data includes a plurality of reading question audio samples, and a manual evaluation score and a model evaluation score of each of the reading question audio samples;

[0145] Based on the difference in manual evaluation scores of the audio samples of the reading questions, the benchmark evaluation standard data is expanded to obtain expanded evaluation standard data;

[0146] Filtering the expanded evaluation standard data to obtain remaining expanded evaluation standard data;

[0147] Based on the baseline evaluation standard data and the remaining expanded evaluation standard data, the evaluation standard of the reading-aloud question is determined.

[0148] The present application also provides an electronic device in an embodiment. Figure 6 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0149] like Figure 6 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0150] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0151] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the reading question assessment methods provided in the embodiments of the present application.

[0152] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any one of the reading question assessment methods provided in the embodiment of the present application.

[0153] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0154] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0155] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0156] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A reading question evaluation method, characterized in that: include: Acquire the acoustic features of the audio of the reading question to be evaluated and the phoneme sequence of the reading question title; Inputting the acoustic features of the reading question audio and the phoneme sequence of the reading question title into an end-to-end reading question evaluation model to obtain a model evaluation score; the end-to-end reading question evaluation model is obtained by joint training based on a task of predicting a model evaluation score of audio and a task of predicting a phoneme position in audio; Based on the model evaluation score, a final evaluation score of the reading question audio is determined.

2. The method according to claim 1, characterized in that Determining the final evaluation score of the reading question audio based on the model evaluation score includes: Determining an evaluation standard for the reading aloud question, wherein the evaluation standard for the reading aloud question includes a correspondence between a model evaluation score and a manual evaluation score of the reading aloud question; Based on the scoring criteria for the reading aloud question, the manual evaluation score corresponding to the model evaluation score in the scoring criteria for the reading aloud question is determined as the final evaluation score of the reading aloud question audio.

3. The method according to claim 1, characterized in that: The end-to-end reading question evaluation model includes: an audio encoder, a phoneme encoder, an audio-phoneme joint encoder, and a model evaluation score output layer; wherein the audio encoder adopts an offset window attention mechanism.

4. The method according to claim 3, characterized in that The step of inputting the feature data of the reading question audio and the feature data of the reading question title into an end-to-end reading question evaluation model to obtain a model evaluation score includes: The audio encoder receives the acoustic features of the audio of the reading question to be evaluated and encodes it using an offset window attention mechanism to obtain an audio encoding result; The phoneme encoder receives and encodes the phoneme sequence of the reading question to obtain a phoneme encoding result; The audio-phoneme joint encoder jointly encodes the audio encoding result and the phoneme encoding result to obtain an audio-phoneme joint encoding result; The model evaluation score output layer outputs the model evaluation score based on the audio phoneme joint encoding result.

5. The method according to claim 1, characterized in that: The training method of the end-to-end reading question evaluation model is as follows: Determining training data, wherein the training data includes phoneme recognition training data, reading pronunciation error detection training data, and reading scoring training data; Taking the first end-to-end reading aloud question evaluation model as a basic model, pre-training the first end-to-end reading aloud question evaluation model based on the phoneme recognition training data and the reading aloud pronunciation error detection training data to obtain a second end-to-end reading aloud question evaluation model; The second end-to-end reading aloud question evaluation model is used as a basic model, and the second end-to-end reading aloud question evaluation model is fine-tuned based on the reading score training data to obtain a trained end-to-end reading aloud question evaluation model.

6. The method according to claim 2, characterized in that The evaluation criteria for determining the reading question title include: Obtaining benchmark evaluation standard data corresponding to the reading question, wherein the benchmark evaluation standard data includes a plurality of reading question audio samples, and a manual evaluation score and a model evaluation score of each of the reading question audio samples; Based on the difference in manual evaluation scores of the audio samples of the reading questions, the benchmark evaluation standard data is expanded to obtain expanded evaluation standard data; Filtering the expanded evaluation standard data to obtain remaining expanded evaluation standard data; Based on the baseline evaluation standard data and the remaining expanded evaluation standard data, the evaluation standard of the reading-aloud question is determined.

7. A reading question evaluation device, characterized in that: include: An acquisition unit, used to acquire the acoustic features of the audio of the reading question to be evaluated and the phoneme sequence of the reading question title; A model evaluation score determination unit is used to input the acoustic features of the reading question audio and the phoneme sequence of the reading question title into an end-to-end reading question evaluation model to obtain a model evaluation score; the end-to-end reading question evaluation model is obtained by joint training based on a task of predicting a model evaluation score of audio and a task of predicting a position of a phoneme in audio; The final evaluation score determination unit is used to determine the final evaluation score of the reading question audio based on the model evaluation score.

8. A computer program product, characterized in that It includes computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the reading question evaluation method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the reading question evaluation method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the reading question evaluation method as described in any one of claims 1 to 6.

Images