Spoken language evaluation model training method, spoken language evaluation method, device and medium

Abstract

The application provides a spoken language evaluation model training method, a spoken language evaluation method, equipment and a medium, and relates to the technical field of natural language processing. The method comprises: updating parameters of a pre-training language model in a first spoken language evaluation model to obtain a second spoken language evaluation model; constructing a third spoken language evaluation model according to the second spoken language evaluation model and an initialized multi-scale acoustic adaptation module; freezing parameters to be frozen in the third spoken language evaluation model, updating parameters of the initialized multi-scale acoustic adaptation module in the third spoken language evaluation model to obtain a fourth spoken language evaluation model; and training the fourth spoken language evaluation model to obtain a target spoken language evaluation model for predicting a spoken language score according to multi-scale acoustic features and text features. The application improves the objectivity and discrimination of the spoken language evaluation result by fusing the multi-scale acoustic features and the text features, and further improves the accuracy of the spoken language score.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method for training a spoken language assessment model, a spoken language assessment method, equipment, and medium. Background Technology

[0002] Oral communication skills assessment has become an important component of measuring students' overall quality. For example, achieving objective, accurate, and automated assessment for non-reading-aloud question types is a core requirement in current oral communication teaching and examination evaluation.

[0003] To meet assessment requirements, existing technologies typically employ cascaded scoring schemes based on pre-trained language models. In practice, a speech recognition model is first used to convert the candidate's original spoken audio into discrete text recognition results. These text results, along with the question information, are then input into the pre-trained language model. After extracting text features, the final predicted score is output based on these features.

[0004] However, this cascaded scoring scheme relies entirely on the text sequence after speech recognition, which results in a lack of necessary scoring differentiation ability when faced with spoken data with the same recognition results but significant differences in pronunciation, leading to poor scoring accuracy. Summary of the Invention

[0005] This invention provides a method for training a spoken language assessment model, a spoken language assessment method, an apparatus, and a medium to address the shortcomings of existing technologies in the lack of necessary scoring differentiation ability and poor scoring accuracy when faced with spoken language data with the same recognition results but significant differences in pronunciation. This invention aims to improve the objectivity and differentiation of spoken language assessment results, thereby enhancing scoring accuracy.

[0006] This invention provides a method for training an oral assessment model, comprising:

[0007] Obtain a training dataset; the training dataset includes multiple training samples, and the training samples include sample speech, test question information corresponding to the sample speech, and oral score labels;

[0008] Based on the training dataset, the parameters of the pre-trained language model in the first oral assessment model are updated to obtain the second oral assessment model.

[0009] Based on the second oral assessment model and the initial multi-scale acoustic adaptation module, a third oral assessment model is constructed.

[0010] The parameters to be frozen in the third spoken language assessment model are frozen. Based on the training dataset, the parameters of the initial multi-scale acoustic adaptation module in the third spoken language assessment model are updated to obtain the fourth spoken language assessment model. The updated initial multi-scale acoustic adaptation module is used to output multi-scale acoustic features aligned with the text modal space of the pre-trained language model. The parameters to be frozen include at least the parameters of the second spoken language assessment model.

[0011] The fourth oral assessment model is trained based on the training dataset to obtain the target oral assessment model; the target oral assessment model is used to predict oral scores based on multi-scale acoustic features and text features.

[0012] According to the oral assessment model training method provided by the present invention, the initialization multi-scale acoustic adaptation module includes a first initialization acoustic adaptation module and a second initialization acoustic adaptation module.

[0013] The first initialization acoustic adaptation module includes a first initialization acoustic encoder and a first initialization adapter. The first initialization acoustic adaptation module is used to extract word-level acoustic features.

[0014] The second initialization acoustic adaptation module includes a second initialization acoustic encoder and a second initialization adapter. The second initialization acoustic adaptation module is used to extract document-level acoustic features.

[0015] According to the oral assessment model training method provided by the present invention, the parameters to be frozen further include the parameters of the first initial acoustic encoder and the parameters of the second initial acoustic encoder;

[0016] The process involves freezing the parameters to be frozen in the third oral language assessment model, updating the parameters of the initial multi-scale acoustic adaptation module in the third oral language assessment model based on the training dataset, to obtain the fourth oral language assessment model, which includes:

[0017] Freeze the parameters of the second oral assessment model, the parameters of the first initial acoustic encoder, and the parameters of the second initial acoustic encoder. Based on the training dataset, update the parameters of the first initial adapter and the second initial adapter in the third oral assessment model to obtain the fourth oral assessment model.

[0018] According to the oral assessment model training method provided by the present invention, the second oral assessment model includes an updated pre-trained language model, an initialized speech recognition model, and an initialized score prediction model;

[0019] The update steps for the parameters of the first initial adapter in the third oral assessment model include:

[0020] The sample speech and the speech recognition result of the sample speech output by the initial speech recognition model are input into the first initial acoustic adaptation module to obtain the sample word-level acoustic features of the sample speech;

[0021] The sample word-level acoustic features and the test question information are input into the scoring prediction component of the third oral assessment model to obtain the first oral prediction score of the sample speech; the scoring prediction component includes the updated pre-trained language model and the initial scoring prediction model.

[0022] Freeze all parameters in the third oral assessment model except for the parameters of the first initialization adapter, and update the parameters of the first initialization adapter in the third oral assessment model according to the first oral prediction score and the oral score label.

[0023] According to a spoken language assessment model training method provided by the present invention, the step of inputting the sample speech and the speech recognition result of the sample speech output by the initial speech recognition model into the first initial acoustic adaptation module to obtain the sample word-level acoustic features of the sample speech includes:

[0024] The sample speech is input into the first initialized acoustic encoder to obtain the pronunciation features of each word in the sample speech;

[0025] The speech recognition result is used as a query vector, and the pronunciation features are used as value vectors and key vectors. These are then input into the first initialization adapter for attention calculation to obtain the sample word-level acoustic features.

[0026] According to the oral assessment model training method provided by the present invention, the parameter update step of the second initialization adapter in the third oral assessment model includes:

[0027] The sample speech is input into the second initialization acoustic adaptation module to obtain the sample document-level acoustic features of the sample speech;

[0028] The sample text-level acoustic features, the speech recognition results, and the test question information are input into the scoring prediction component to obtain the second spoken language prediction score of the sample speech;

[0029] Freeze all parameters in the third oral assessment model except for the parameters of the second initialization adapter, and update the parameters of the second initialization adapter in the third oral assessment model according to the second oral prediction score and the oral score label.

[0030] According to the oral assessment model training method provided by the present invention, the first initial acoustic encoder is constructed based on the encoding module in the initial speech recognition model, and the second initial acoustic encoder is constructed based on the encoder in the pronunciation level prediction model; the pronunciation level prediction model is trained based on the sample speech and the overall pronunciation level label corresponding to the sample speech.

[0031] According to the oral assessment model training method provided by the present invention, the step of training the fourth oral assessment model based on the training dataset to obtain the target oral assessment model includes:

[0032] The multi-scale acoustic features of the sample speech output by the updated initial multi-scale acoustic adaptation module, the speech recognition result of the sample speech output by the initial speech recognition model in the fourth oral assessment model, and the test question information are input into the updated pre-trained language model in the fourth oral assessment model to obtain the third oral prediction score of the sample speech.

[0033] Freeze the parameters of the updated initial multi-scale acoustic adaptation module and the parameters of the initial speech recognition model in the fourth oral assessment model. Based on the third oral prediction score and the oral score label, jointly train the module to be updated in the fourth oral assessment model to obtain the target oral assessment model.

[0034] The module to be updated includes all other modules in the fourth oral assessment model except for the updated initial multi-scale acoustic adaptation module and the initial speech recognition model.

[0035] This invention also provides a method for oral language assessment, comprising:

[0036] The speech to be tested and the corresponding test question information are input into the target oral assessment model to obtain the oral prediction score of the speech to be tested.

[0037] The target oral assessment model is trained based on the oral assessment model training method described above.

[0038] According to a spoken language assessment method provided by the present invention, the step of inputting the speech to be assessed and the corresponding test question information into a target spoken language assessment model to obtain a spoken language prediction score for the speech to be assessed includes:

[0039] The speech to be evaluated is input into the speech recognition model in the target spoken language assessment model to obtain the speech recognition result of the speech to be evaluated;

[0040] The speech to be evaluated and the speech recognition result of the speech to be evaluated are input into the first acoustic adaptation module of the target spoken language evaluation model to obtain the word-level acoustic features of the speech to be evaluated.

[0041] The speech to be evaluated is input into the second acoustic adaptation module of the target spoken language assessment model to obtain the text-level acoustic features of the speech to be evaluated.

[0042] The speech recognition results of the speech to be evaluated, the word-level acoustic features, the text-level acoustic features, and the test question information corresponding to the speech to be evaluated are input into the language model of the target oral assessment model to obtain global scoring features;

[0043] The global scoring features are input into the scoring prediction model of the target spoken language assessment model to obtain the spoken language prediction score of the speech to be assessed.

[0044] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the oral assessment model training method as described in any of the preceding claims, or implements the oral assessment method as described in any of the preceding claims.

[0045] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the oral assessment model training method as described in any of the preceding claims, or implements the oral assessment method as described in any of the preceding claims.

[0046] The present invention also provides a computer program product, comprising a computer program that, when executed by a processor, implements the oral assessment model training method as described in any of the preceding claims, or implements the oral assessment method as described in any of the preceding claims.

[0047] The oral assessment model training method, oral assessment method, device, and medium provided by this invention construct a target oral assessment model by establishing a semantic scoring benchmark model step by step, implementing spatial alignment from acoustic modality to text modality, and multimodal joint training. This approach retains the powerful semantic modeling capabilities of the pre-trained language model while using multi-dimensional acoustic feature compensation at the physical level to accurately quantify pronunciation accuracy and speech fluency in the examinee's oral quality. This ensures that the target oral assessment model trained accordingly can still output highly discriminative and objectively accurate oral prediction scores when processing oral data with the same text content but significantly different pronunciation quality, thereby effectively improving scoring accuracy. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0049] Figure 1 This is a flowchart illustrating the end-to-end oral assessment method based on pre-trained models provided by existing technologies.

[0050] Figure 2 This is a flowchart illustrating the oral assessment model training method provided by the present invention.

[0051] Figure 3 This is a schematic diagram of the training process of the first oral assessment model provided by the present invention.

[0052] Figure 4 This is a schematic diagram of the pronunciation level prediction model provided by the present invention.

[0053] Figure 5 This is a schematic diagram of the parameter update process for the first initialization adapter provided by the present invention.

[0054] Figure 6 This is a schematic diagram of the structure of the first initialization acoustic adaptation module provided by the present invention.

[0055] Figure 7 This is a schematic diagram of the parameter update process for the second initialization adapter provided by the present invention.

[0056] Figure 8 This is a schematic diagram of the training process for the fourth oral language assessment model provided by the present invention.

[0057] Figure 9 This is one of the flowcharts of the oral assessment method provided by the present invention.

[0058] Figure 10 This is the second flowchart of the oral assessment method provided by the present invention.

[0059] Figure 11 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0061] All actions involving the acquisition of signal information or data in this application are carried out in accordance with the relevant data protection laws and policies of the country where the application is located, and with the authorization granted by the owner of the relevant device.

[0062] Oral assessment tasks typically include two main categories of questions: reading aloud and non-reading aloud. Reading aloud questions involve reading a given text aloud, primarily assessing pronunciation accuracy and fluency, as well as the completeness of the reading. Non-reading aloud questions involve providing background information and requiring candidates to answer questions or elaborate on topics, primarily assessing pronunciation accuracy and fluency, the accuracy of the answer, and the relevance and reasonableness of the topic discussion. The following description uses the non-reading aloud question type as an example to illustrate the method provided in this application.

[0063] For non-reading-aloud question types, early techniques primarily relied on manually extracting relevant features, followed by scoring using regression models. With the development of artificial intelligence, feature extraction has increasingly leveraged deep neural networks, and regression models have evolved into neural networks based on architectures such as Transformers. Furthermore, with the emergence of pre-trained models, such as the Bidirectional Encoder Representations from Transformers (BERT) and Wav2Vec speech vectorization models, end-to-end scoring schemes that fine-tune pre-trained models using scoring data have achieved superior results.

[0064] Figure 1 This is a flowchart illustrating the end-to-end oral assessment method based on pre-trained models provided by existing technologies.

[0065] The current mainstream end-to-end oral assessment solutions based on the BERT model are essentially cascaded solutions. For example... Figure 1As shown, in practice, the user's audio is processed by an Automatic Speech Recognition (ASR) model and converted into text-based language recognition results. These results, along with exam-related information such as questions or topics and answer keys, are then fed into the BERT model. The BERT model's output features corresponding to the Classification (CLS) positions at the beginning of the text sequence are processed through a Full Connection (FC) layer to predict the final score. Because the pre-trained model has undergone extensive self-supervised learning on massive amounts of text corpora and has learned rich semantic knowledge, the overall score predicted by the fine-tuned pre-trained model outperforms traditional manual feature extraction methods.

[0066] However, while existing end-to-end oral assessment schemes based on pre-trained language models have improved scoring performance to some extent, they still have significant drawbacks. These schemes are essentially cascaded solutions, heavily reliant on speech recognition results. During the process of converting audio into recognized text, a large amount of acoustic detail, such as pronunciation accuracy, intonation, overall fluency, and pauses, is inevitably lost. This leads to a critical problem: speech data with identical recognition results but vastly different pronunciation quality often cannot be effectively distinguished. For example, a stuttering, unclear speech and a standard, fluent speech, if their speech-to-text results are identical, would likely receive the same score from existing text-based BERT scoring models. This clearly does not meet the objective standards of oral assessment and fails to accurately reflect the test-taker's oral proficiency.

[0067] To address this issue, this application provides a method for training an oral assessment model. The core concept involves introducing a multi-scale acoustic adaptation module to supplement the input of the pre-trained language model with multi-level acoustic dimensional information. This enables the constructed oral assessment model to perceive subtle pronunciation defects or macroscopic fluency fluctuations. Furthermore, since the pre-trained language model originally operates in a pure text space, to ensure that the hidden representations generated by the multi-scale acoustic adaptation module can be effectively processed by the pre-trained language model, this application proposes a progressive three-stage training strategy. This involves establishing a semantic scoring benchmark step-by-step, implementing spatial alignment from acoustic to text modalities, and finally, multi-modal joint training. This solves the feature mismatch problem when directly fusing heterogeneous modal data. While retaining the powerful semantic modeling capabilities of the pre-trained language model, it achieves precise quantification of pronunciation accuracy and speech fluency in the examinee's oral quality through multi-dimensional acoustic feature compensation at the physical level. This ensures that even when processing oral data with identical text content but significantly different pronunciation quality, it can still output highly discriminative and objectively accurate oral prediction scores, thereby improving scoring accuracy.

[0068] To facilitate a clear description of the technical solution of this invention, it should first be clarified that the oral assessment model training method provided by this invention can be deployed in an electronic device with data processing and computing capabilities. For example, the electronic device can be a cloud computing server, an edge computing node, or a terminal device with corresponding computing power, etc. This invention does not make any specific limitations.

[0069] Figure 2 This is a flowchart illustrating the oral assessment model training method provided by the present invention; as shown below. Figure 2 As shown, the method includes steps 210, 220, 230, 240 and 250.

[0070] Step 210: Obtain the training dataset; the training dataset includes multiple training samples, and the training samples include sample speech, test question information corresponding to the sample speech, and oral score labels.

[0071] Optionally, before training the model, a training dataset for model optimization needs to be obtained. This training dataset is a series of cleaned, aligned, and labeled data sets, with the core basic unit being the training sample. In order for the model to learn the relationship between speech acoustic features and text semantic information simultaneously and accurately map it to the scoring space, each training sample can specifically contain the following three parts: sample speech, the test question information corresponding to the sample speech, and the oral score label.

[0072] The sample speech can be recorded from real exam environments or collected in simulated testing environments. Sample speech can be recorded using standard audio acquisition devices such as headset microphones or array microphones, and in formats such as Pulse Code Modulation (PCM), Waveform Audio File Format (WAV), or Free Lossless Audio Codec (FLAC). To improve the model's generalization ability, the sample speech should cover diverse scenarios, including but not limited to different background noise levels (e.g., quiet classrooms, noisy exam rooms), different types of recording equipment, and different speaker characteristics (e.g., different ages, genders, regional accents, speech rates, etc.). Before storing the training samples in the training dataset, the sample speech in the training samples can be preprocessed, such as removing initial and final silences, normalizing volume, and using data augmentation techniques, such as adding random noise, simulating room reverberation, changing speech rate, or changing pitch, to generate more augmented samples, thereby expanding the dataset size and improving the model's robustness.

[0073] Test question information is an indispensable contextual basis in oral assessment tasks. Test question information typically exists in text form and includes, but is not limited to, question descriptions, topic background information, specific answer requirements, and sample answers. Test question information can be obtained by directly retrieving it from the examination management system database, extracting it from the test papers, or manually entering it and then standardizing and formatting it. When constructing training samples, it is essential to ensure a strict logical correspondence between the test question information and the sample audio; that is, the sample audio in each training sample is the response audio corresponding to its corresponding test question information. The role of test question information is to provide semantic constraints to determine whether the candidate's answer is relevant, rich in content, and logically coherent, rather than simply assessing the accuracy of pronunciation.

[0074] Spoken language scoring labels serve as supervisory signals during model training, representing the true scores of sample speech under specific scoring criteria. Obtaining spoken language scoring labels can rely on professional human scoring or high-precision benchmark model scoring. In the human scoring implementation, rigorously trained and qualified scorers can score sample speech based on a predetermined scoring scale, considering multiple dimensions such as pronunciation accuracy, fluency, completeness, grammatical and lexical richness, and semantic accuracy. A final total score is then used as the label. To reduce subjective error during the aggregation process, a mechanism of multiple blind evaluations and averaging or medianing can be employed. In the automatic labeling implementation, an existing high-precision spoken language assessment model with superior performance to the current model being trained can be used to pre-score a massive amount of unlabeled sample speech, generating pseudo-labels. These pseudo-labels are then corrected through a small amount of manual sampling to obtain the final spoken language scoring labels. The data type of spoken language scoring labels is typically continuous numerical, such as 0-100 or 0-10, or discrete levels, such as A, B, C, and D, depending on the design requirements of the model output.

[0075] The training dataset obtained and constructed using the above methods not only ensures the diversity and authenticity of the data but also guarantees a strong correlation between the input features (i.e., sample speech and its corresponding test question information) and the target variable (i.e., oral score labels). This lays a solid data foundation for the subsequent steps of training the model to learn how to extract key features from multimodal inputs and map them to the scoring space, effectively avoiding problems such as model overfitting or poor generalization ability caused by data bias.

[0076] Step 220: Based on the training dataset, update the parameters of the pre-trained language model in the first oral assessment model to obtain the second oral assessment model.

[0077] Step 230: Construct a third oral assessment model based on the second oral assessment model and the initialized multi-scale acoustic adaptation module.

[0078] Step 240: Freeze the parameters to be frozen in the third spoken language assessment model. Based on the training dataset, update the parameters of the initial multi-scale acoustic adaptation module in the third spoken language assessment model to obtain the fourth spoken language assessment model. The updated initial multi-scale acoustic adaptation module is used to output multi-scale acoustic features aligned with the text modality space of the pre-trained language model. The parameters to be frozen include at least the parameters of the second spoken language assessment model.

[0079] Step 250: Train the fourth oral assessment model based on the training dataset to obtain the target oral assessment model; the target oral assessment model is used to predict oral scores based on multi-scale acoustic features and text features.

[0080] Optionally, after obtaining the training dataset, in order to address the feature mismatch problem when directly fusing heterogeneous modal data, and to achieve accurate quantification of pronunciation accuracy and speech fluency in test takers' spoken language quality through multi-dimensional acoustic feature compensation at the physical level while retaining the powerful semantic modeling capabilities of the pre-trained language model, this embodiment proposes a progressive three-stage training strategy. This strategy involves establishing a semantic scoring benchmark model step by step, implementing spatial alignment from acoustic to text modalities, and finally conducting multi-modal joint training. The final target spoken language assessment model, capable of predicting spoken language scores based on multi-scale acoustic and text features, is obtained by:

[0081] In the first stage, a benchmark scoring model with strong text semantic understanding capabilities, namely the second oral assessment model, is constructed and trained. The core of this stage lies in using the mapping relationship between the text features in the training dataset (i.e., the speech recognition results and test information obtained from the transcription of sample speech) and the oral scoring labels to perform domain-adaptive fine-tuning of the pre-trained language model in the first oral assessment model.

[0082] Figure 3 This is a schematic diagram of the training process of the first oral assessment model provided by the present invention.

[0083] like Figure 3 As shown, a first oral assessment model is constructed. This first oral assessment model includes at least a pre-trained language model pre-trained on a massive general-purpose corpus, such as the existing BERT model, and may also include an initial speech recognition model and an initial score prediction model. The initial speech recognition model here is a pre-trained model with speech recognition capabilities, pre-trained on a large-scale general-purpose speech dataset; the initial score prediction model can be a model whose parameters are randomly initialized and prepared for score prediction through regression or classification, or it can be a pre-trained model with score prediction capabilities, pre-trained on other related scoring tasks, etc. This embodiment does not specifically limit this.

[0084] The initial speech recognition model can be an end-to-end model based on an encoder-decoder architecture. The encoder part can employ a convolution-augmented transform (Conformer) structure to extract deep acoustic features of the speech. The decoder part can use a transformer structure to autoregressively predict the next word, and after decoding, restore the word to its original form, thus obtaining the final speech recognition result. In this training phase, this initial speech recognition model is primarily used to provide text transcription services; its parameters can be frozen to prevent backpropagation of speech recognition error gradients from interfering with the learning of text scoring.

[0085] The initialization of the rating prediction model can employ a fully connected layer (FC) structure. Specifically, since the vector corresponding to the special marker [CLS] in the output of the pre-trained language model contains global semantic information of the entire input sequence, the initialization of the rating prediction model is a fully connected layer following the [CLS] output, used to map this high-dimensional semantic vector into a scalar score, i.e., the spoken language prediction score.

[0086] After the model is built, a parameter update process is performed to obtain the second oral assessment model. Specific steps include:

[0087] First, sample speech is read from the training dataset to initialize the speech recognition model and obtain the speech recognition results (also known as text transcription information). Simultaneously, the corresponding test question information is extracted from the sample speech. The speech recognition results and test question information are then concatenated to form a complete text input sequence. For example, the test question information and speech recognition results can be combined sequentially as input to a pre-trained language model.

[0088] It should be noted that the process of obtaining speech recognition results includes at least the following implementation methods:

[0089] One possible implementation is to directly input the sample speech into the initial speech recognition model to obtain the corresponding speech recognition result. At this time, the encoder inside the initial speech recognition model will automatically complete the extraction process from the original waveform to the deep acoustic features. Subsequently, the decoder inside the initial speech recognition model generates the text sequence of each word based on these internal features to obtain the final speech recognition result.

[0090] Another possible implementation involves first extracting basic acoustic features from the sample speech, and then using a Voice Activity Detection (VAD) model to extract spoken segments based on these basic acoustic features. The extracted spoken segments are then input into an initialized speech recognition model to obtain the corresponding speech recognition results. In this process, the sample speech can be preprocessed, including pre-emphasis, framing, and windowing, before extracting Mel-Frequency Cepstral Coefficients (MFCCs) or Mel-filter Bank Energies (FBanks) as basic acoustic features. These basic acoustic features are then input into the VAD model. Based on the input basic acoustic features, the VAD model identifies and locates the spoken segments that constitute valid human voices in the sample speech, removing initial and final silences, background noise, or non-speech segments. Next, based on the start and end timestamps of the spoken segments output by the VAD model, the corresponding spoken segment speech data (or only the basic acoustic feature sequence of the corresponding spoken segment) is extracted from the original sample speech. Finally, the spoken segment speech data (or its corresponding basic acoustic feature sequence) is input into the encoder of the initial speech recognition model. At this point, the encoder of the initial speech recognition model only needs to perform deep encoding on the valid spoken segment content, capturing long-term temporal dependencies and local spectral details. Finally, the decoder of the initial speech recognition model outputs the text sequence of each word, thus obtaining a high-precision speech recognition result. In this process, by inputting only the valid spoken segments into the initial speech recognition model, the recognition error rate can be significantly reduced and high-quality transcribed text can be provided, laying the foundation for building a highly reliable text semantic scoring benchmark.

[0091] Subsequently, the constructed text input sequence is fed into the pre-trained language model within the first oral assessment model. The pre-trained language model encodes the input sequence through its internal multi-layered self-attention mechanism, outputting a context-related feature vector for each position and extracting the output vector corresponding to the special marker [CLS]. This vector represents the overall semantic representation of the test information and the language recognition result. This [CLS] vector is then input into the initialization scoring prediction model to obtain the oral prediction score of the sample speech through mapping.

[0092] Finally, the loss value between the predicted spoken score of the sample speech and its corresponding spoken score label is calculated. For example, the mean squared error loss function is used to calculate the loss value between the predicted spoken score of the sample speech and its corresponding spoken score label to obtain the loss function of the first spoken language assessment model. Based on freezing the model parameters of other models besides the pre-trained language model (such as the initialized speech recognition model and the initialized score prediction model) in the first spoken language assessment model, the parameters of the pre-trained language model in the first spoken language assessment model are iteratively updated based on the loss function of the first spoken language assessment model to obtain the second spoken language assessment model. The second spoken language assessment model includes the updated pre-trained language model and the frozen original models, such as the initialized speech recognition model and the initialized score prediction model. The second spoken language assessment model trained in this way has fully learned the correlation between text content (including the accuracy of the answer, relevance, vocabulary richness, etc.) and the score in the spoken language assessment field, that is, it has basic spoken language scoring functions.

[0093] After establishing a robust text semantic scoring benchmark model in the first stage, we move on to the second stage. The core objective of this stage is to address the heterogeneous mismatch between the speech modality and the text modality, namely, how to enable the pre-trained language model updated in the first stage to understand and effectively utilize acoustic features.

[0094] Specifically, the third spoken language assessment model is first constructed. In this process, the pre-trained second spoken language assessment model is loaded, and an initialization multi-scale acoustic adaptation module is added on top of it. That is, the third spoken language assessment model includes the pre-trained language model updated in the first stage and the frozen original model (such as the initialization speech recognition model and the initialization score prediction model). In addition, the third spoken language assessment model also includes the initialization multi-scale acoustic adaptation module.

[0095] The initialization multi-scale acoustic adaptation module here can be configured as a multi-branch parallel architecture to capture acoustic information at different scales, specifically based on the dimensions of the acoustic information to be captured. For example, it can be configured as a two-branch parallel architecture, where one branch is used to capture fine-grained word-level acoustic features, that is, acoustic features that represent micro-information such as the pronunciation accuracy and phoneme integrity of each word, and the other branch is used to capture coarse-grained text-level acoustic features, that is, acoustic features that represent macro-information such as the overall fluency, pauses, rhythm, and intonation of the entire speech.

[0096] After constructing the third oral assessment model, a parameter update process is performed to obtain the fourth oral assessment model. The specific update steps include:

[0097] First, sample speech is read from the training dataset to obtain speech recognition results through the initial speech recognition model in the third spoken language assessment model. Then, the multi-scale acoustic features of the sample speech are obtained by initializing the multi-scale acoustic adaptation module. Here, multi-scale acoustic features refer to the acoustic information at different levels of abstraction that the initializing multi-scale acoustic adaptation module can extract to comprehensively characterize speech properties.

[0098] Subsequently, the multi-scale acoustic features, the speech recognition results of the sample speech recognized by the initial speech recognition model, and the test information are input into the updated pre-trained language model. The updated pre-trained language model outputs different spoken language prediction scores for different multi-scale acoustic features. Then, based on the parameters to be frozen in the third spoken language assessment model, the prediction error between the spoken language prediction scores and spoken language score labels corresponding to different multi-scale acoustic features is used to update some or all parameters of the branch structure corresponding to different multi-scale acoustic features in the initial multi-scale acoustic adaptation module. Alternatively, the updated pre-trained language model outputs a unified spoken language prediction score for all multi-scale acoustic features. Then, based on the parameters to be frozen in the third spoken language assessment model, the prediction error between the unified spoken language prediction score and spoken language score labels is used to update some or all parameters of the overall parameter set of the initial multi-scale acoustic adaptation module. When all parameters of the initial multi-scale acoustic adaptation module need to be updated, the parameters to be frozen here only include the parameters of the second spoken language assessment model; when only some parameters of the initial multi-scale acoustic adaptation module need to be updated, the parameters to be frozen here include not only the parameters of the second spoken language assessment model, but also other parameters in the initial multi-scale acoustic adaptation module other than the parameters that need to be updated. Specifically, the parameters to be frozen can be dynamically determined according to the structure and training requirements of the initial multi-scale acoustic adaptation module.

[0099] Finally, after updating the parameters of the initial multi-scale acoustic adaptation module in the third spoken language assessment model, the fourth spoken language assessment model can be constructed. This fourth spoken language assessment model includes the second spoken language assessment model and the updated initial multi-scale acoustic adaptation module. This updated module has the ability to output multi-scale acoustic features highly aligned with the text modality space, enabling effective transfer from acoustic modality to text modality.

[0100] This phase achieves precise alignment between acoustic modalities and text modalities by freezing the backbone of the baseline scoring model and specifically training and initializing the multi-scale acoustic adaptation module. This decoupled training avoids the model oscillation or convergence difficulties commonly encountered during direct joint training of multiple modalities. By introducing a general multi-scale acoustic adaptation mechanism, the pre-trained language model can effectively incorporate speech dimension information without changing the original text scoring logic, thus overcoming the limitation of pure text scoring in distinguishing between speech sounds with the same text but different pronunciations.

[0101] After completing modal alignment, the third stage begins. The core objective of this stage is to perform multimodal joint fine-tuning to uncover high-order interaction information between acoustic and textual features, ultimately obtaining the target spoken language assessment model.

[0102] Specifically, sample speech is read from the training dataset to obtain speech recognition results (i.e., text transcription information) through the initial speech recognition model in the fourth oral language assessment model. Multi-scale acoustic features of the sample speech are then acquired through the updated initial multi-scale acoustic adaptation module in the fourth oral language assessment model. The multi-scale acoustic features, the speech recognition results of the sample speech, and the test question information are input into the updated pre-trained language model in the fourth oral language assessment model. The updated pre-trained language model and the initial scoring prediction model in the fourth oral language assessment model output an overall oral language prediction score based on different multi-scale acoustic features. Then, based on the prediction error between the overall oral language prediction score and the oral language score label, some or all parameters in the fourth oral language assessment model are jointly fine-tuned to obtain the target oral language assessment model. This target oral language assessment model can comprehensively consider the accuracy and richness of the text content, as well as multi-dimensional features at the speech level, to output the final oral language prediction score.

[0103] This stage, through joint fine-tuning, breaks through the limitations of the relatively independent modules in the previous stage, and achieves true multimodal deep fusion. It not only ensures the professionalism of feature extraction of each modality, but also realizes the complementary gain of multimodal information. This makes the final target oral assessment model have extremely high robustness and human-like scoring ability in complex scenarios, which is significantly better than the traditional cascaded scheme. It can truly achieve a comprehensive and accurate quantitative assessment of the candidate's oral pronunciation quality and oral content.

[0104] The method provided in this embodiment constructs a target oral assessment model by establishing a semantic scoring benchmark model step by step, implementing spatial alignment from acoustic modality to text modality, and conducting multimodal joint training. This approach retains the powerful semantic modeling capabilities of the pre-trained language model while using multidimensional acoustic feature compensation at the physical level to accurately quantify the pronunciation accuracy and fluency of the examinee's oral quality. This ensures that the target oral assessment model trained accordingly can still output highly discriminative and objectively accurate oral prediction scores when processing oral data with the same text content but significantly different pronunciation quality, thereby effectively improving scoring accuracy.

[0105] Based on the above embodiments, as an optional embodiment, in order to capture different dimensions of spoken pronunciation performance more precisely, the initialization multi-scale acoustic adaptation module includes a first initialization acoustic adaptation module and a second initialization acoustic adaptation module; the first initialization acoustic adaptation module includes a first initialization acoustic encoder and a first initialization adapter, and the first initialization acoustic adaptation module is used to extract word-level acoustic features; the second initialization acoustic adaptation module includes a second initialization acoustic encoder and a second initialization adapter, and the second initialization acoustic adaptation module is used to extract document-level acoustic features.

[0106] In this embodiment, considering that spoken language assessment focuses not only on microscopic pronunciation details but also on macroscopic fluency performance, this embodiment divides acoustic features into two parallel granularities.

[0107] The first initialization acoustic adaptation module is specifically designed for fine-grained information. The first initialization acoustic encoder, also known as the fine-grained acoustic encoder, is responsible for extracting the low-level acoustic detail features corresponding to each word from the input speech. These detail features reflect the accuracy of pronunciation, such as whether vowels are full, consonants are clear, and whether there are any micro-pronunciation defects like elision, mispronunciation, or unclear pronunciation. The first initialization adapter is a learnable feature mapping network, such as a projection layer with an attention mechanism. Its function is to map and align the word-level pronunciation features output by the first initialization acoustic encoder to the text modal space of the pre-trained language model, thereby generating word-level acoustic features.

[0108] The second initialization acoustic encoder, also known as the initialization coarse-grained acoustic encoder, is responsible for extracting coarse-grained overall pronunciation level information from the entire sample speech. This information represents macroscopic features such as fluency, naturalness of pauses, appropriateness of intonation, rhythm, and overall emotional tone. The second initialization adapter aligns the global, text-level acoustic features output by the second initialization acoustic encoder to the text modal space of the pre-trained language model, enabling effective fusion with the text features input to the pre-trained language model.

[0109] The first initial acoustic encoder here can be a reused encoding module in the initial speech recognition model, or it can be an acoustic encoder that is independently pre-trained based on a general speech dataset; the second initial acoustic encoder can be an encoder specifically trained for the pronunciation level prediction task, or it can be a general speech encoder pre-trained using unsupervised contrastive learning.

[0110] The method provided in this embodiment achieves multi-dimensional decoupled modeling of spoken language quality by refining the initialization multi-scale acoustic adaptation module into two sub-modules targeting the word level and the passage level. The first initialization acoustic adaptation module uses a fine-grained encoder to ensure high sensitivity to single-word pronunciation errors; the second initialization acoustic adaptation module uses a coarse-grained encoder to ensure effective assessment of the overall level. By combining the two, the model can comprehensively cover all dimensions of the human scoring criteria, significantly improving the model's scoring performance in non-reading-based question types (such as topic description or question-and-answer), avoiding information loss caused by single-scale features, and making the scoring results more objective and comprehensive.

[0111] For example, in one possible implementation, the first initial acoustic encoder is constructed based on the encoding module in the initial speech recognition model, and the second initial acoustic encoder is constructed based on the encoder in the articulation level prediction model; the articulation level prediction model is trained based on the sample speech and the overall articulation level label corresponding to the sample speech.

[0112] Specifically, the first initial acoustic encoder directly reuses the encoding module (also called the encoder) in the initial speech recognition model. Its design principle is that the output of the encoding module of the initial speech recognition model already contains deep acoustic feature representations of word pronunciation in the input speech. By comparing these acoustic features with the recognition results output by the decoder (i.e., the standard pronunciation corresponding to the word semantics), it is possible to effectively determine whether there are defects in word pronunciation. Given that this encoding module has undergone large-scale pre-training on massive amounts of general speech data and possesses extremely strong phoneme recognition and pronunciation detail capture capabilities, this embodiment directly utilizes it to construct the first initial acoustic encoder, and the parameters of the first initial acoustic encoder remain fixed throughout the overall training process. This implementation not only avoids the resource consumption caused by repeated training but also ensures that the extracted word-level acoustic features have extremely high signal-to-noise ratio and discriminative power, enabling precise location of micro-pronunciation defects such as elision and mispronunciation, providing reliable fine-grained basis for subsequent scoring.

[0113] The second initialization acoustic encoder is built upon the encoder in the independently trained articulation level prediction model. The articulation level prediction model is specifically designed to extract overall articulation level features of the entire speech, such as the fluency, rhythm, and naturalness of the speech flow.

[0114] Figure 4 This is a schematic diagram of the pronunciation level prediction model provided by the present invention.

[0115] like Figure 4 As shown, the structure of this pronunciation level prediction model includes a Transformer-based encoder (also known as a Transformer Encoder) and a fully connected layer (also known as an FC layer). The specific training steps are as follows:

[0116] First, a training set for overall pronunciation proficiency scoring needs to be constructed. This dataset contains a large number of sample speech, and each sample speech is scored by human experts for its overall pronunciation proficiency, such as fluency and rhythm, thereby obtaining an overall pronunciation proficiency score label.

[0117] Secondly, sample speech is used as input when training this pronunciation level prediction model. To capture global information, a special classification tag, such as the [CLS] tag, is usually added to the beginning of the sample speech.

[0118] Then, the speech with the [CLS] mark added is fed into the Transformer Encoder for deep feature extraction. The vector corresponding to the [CLS] position in the output is regarded as the overall pronunciation representation of the speech segment.

[0119] Next, the output vector at the [CLS] position is input into the subsequent FC layer, and the FC layer predicts the overall pronunciation level score.

[0120] Finally, the pronunciation level prediction model is trained by calculating the error between the overall pronunciation level prediction score and the overall pronunciation level score label.

[0121] Once the pronunciation level prediction model has converged during training, its internal Transformer Encoder possesses a strong ability to extract overall speech fluency and prosodic features. At this point, the parameters of the trained Transformer Encoder can be extracted to initialize a second initial acoustic encoder. During subsequent training, the parameters of this second initial acoustic encoder are typically kept constant to stably output high-quality document-level acoustic features.

[0122] The method provided in this embodiment extracts micro-pronunciation features by reusing a mature speech recognition encoder and extracts macro-prosodic features by using the encoder of a specially trained pronunciation level prediction model. This achieves dedicated coding for features of different granularities, enabling the model to simultaneously possess the keen perception of subtle pronunciation errors and the macro-control of overall expression fluency. This significantly improves the scoring discrimination and objectivity when faced with data of the same text but significantly different pronunciation quality, thereby improving the scoring accuracy.

[0123] Based on the above embodiments, as an optional embodiment, the parameters to be frozen further include the parameters of the first initial acoustic encoder and the parameters of the second initial acoustic encoder;

[0124] The process involves freezing the parameters to be frozen in the third oral language assessment model, updating the parameters of the initial multi-scale acoustic adaptation module in the third oral language assessment model based on the training dataset, to obtain the fourth oral language assessment model, which includes:

[0125] Freeze the parameters of the second oral assessment model, the parameters of the first initial acoustic encoder, and the parameters of the second initial acoustic encoder. Based on the training dataset, update the parameters of the first initial adapter and the second initial adapter in the third oral assessment model to obtain the fourth oral assessment model.

[0126] To reduce the difficulty of multimodal joint training and avoid model training divergence or damage to the existing semantic capabilities of the pre-trained language model due to the huge difference between acoustic feature distribution and text feature distribution, this embodiment adopts a refined hierarchical training and parameter freezing strategy.

[0127] Specifically, the second oral assessment model already possesses strong text-content-based scoring capabilities, and its parameters represent a mature text modality scoring space. Therefore, in the second stage of training, its parameters are listed as parameters to be frozen, meaning that their gradients are not calculated during backpropagation, and their parameters are kept constant.

[0128] The first initial acoustic encoder originates from the encoder of a mature initial speech recognition model; the second initial acoustic encoder originates from the encoder of a specially trained pronunciation level prediction model. These encoders have been thoroughly trained on both speech recognition and pronunciation fluency assessment tasks, possessing the ability to extract high-quality acoustic features. Therefore, to prevent the feature extraction capabilities of these encoders from degrading due to fine-tuning on a relatively small scoring dataset—that is, to avoid catastrophic forgetting—this embodiment also lists their parameters as parameters to be frozen.

[0129] Therefore, the parameters that truly need to be updated and trained in this stage are limited to the parameters of the first and second initialized adapters. These two adapters act as modality conversion layers. By updating only the parameters of these two adapters, the model can focus on learning how to transform the acoustic signal output from the frozen encoder into textual semantic feature vectors that the frozen pre-trained language model can understand and use for scoring. This strategy significantly reduces the number of trainable parameters, accelerates convergence, and ensures the stability of the final feature alignment.

[0130] Furthermore, after completing the above parameter freezing configuration, the parameters of the first initialization adapter and the second initialization adapter in the third oral assessment model are updated in a targeted manner based on the training dataset. In specific implementation, the optimizer is first configured to only pass in the parameter lists of the first initialization adapter (the adapter for fine-grained alignment) and the second initialization adapter (the adapter for coarse-grained alignment), so that it only maintains the momentum and learning rate states of these two parts of the parameters. Subsequently, a forward propagation and backward update process based on the training dataset is executed: sample speech and corresponding text information (i.e., speech recognition results and test question information) are obtained from the training dataset. The sample speech is processed by the frozen first and second initial acoustic encoders to extract multi-scale pronunciation information features, which are then converted into corresponding multi-scale acoustic features by the first and second initial adapters. Simultaneously, the text information and multi-scale acoustic features are input into the updated pre-trained language model in the third oral assessment model. The updated pre-trained language model and the initial score prediction model output different oral prediction scores for different initial adapters. The prediction error between the oral prediction scores and oral score labels of different initial adapters is used to update the models of different initial adapters in the third oral assessment model to obtain the fourth oral assessment model. Alternatively, the updated pre-trained language model and the initial score prediction model in the third oral assessment model output a unified oral prediction score for all initial adapters. The prediction error between the unified oral prediction score and the oral score label is used to update the parameters of all initial adapters in the third oral assessment model to obtain the fourth oral assessment model.

[0131] This embodiment, by precisely controlling the freeze range, fixes the complex acoustic encoder and powerful pre-trained language model, training only the adapter module with a smaller number of parameters. This has multiple beneficial effects: First, it greatly reduces the number of training parameters, lowers the risk of overfitting, and accelerates the convergence speed; second, it forces the adapter to learn pure alignment functions to map the fixed acoustic feature space to a fixed text feature space, avoiding feature space drift or training instability caused by simultaneous updates of multiple complex modules; third, it ensures the accuracy and effectiveness of modal alignment, prevents acoustic noise from interfering with the learned text semantic logic, and provides a high-quality initial state for the joint fine-tuning in the third stage.

[0132] Based on the above embodiments, as an optional embodiment, in order to achieve word-level acoustic feature alignment, the parameter update steps of the first initialization adapter in the third spoken language assessment model include:

[0133] The sample speech and the speech recognition result of the sample speech output by the initial speech recognition model are input into the first initial acoustic adaptation module to obtain the sample word-level acoustic features of the sample speech;

[0134] The sample word-level acoustic features and the test question information are input into the scoring prediction component of the third oral assessment model to obtain the first oral prediction score of the sample speech; the scoring prediction component includes the updated pre-trained language model and the initial scoring prediction model.

[0135] Freeze all parameters in the third oral assessment model except for the parameters of the first initialization adapter, and update the parameters of the first initialization adapter in the third oral assessment model according to the first oral prediction score and the oral score label.

[0136] Optionally, the second spoken language assessment model includes an updated pre-trained language model, an initialized speech recognition model, and an initialized score prediction model. Based on this architecture, the third spoken language assessment model may specifically include an updated pre-trained language model, an initialized speech recognition model, and an initialized score prediction model, as well as a first initialized acoustic adaptation module and a second initialized acoustic adaptation module.

[0137] Figure 5 This is a schematic diagram of the parameter update process for the first initialization adapter provided by the present invention.

[0138] like Figure 5 As shown, the parameter update steps for the first initialization adapter specifically include:

[0139] First, the sample speech is recognized by the initial speech recognition model in the third oral assessment model, and the corresponding text sequence is output, which is the speech recognition result. Then, the sample speech and the speech recognition result are input together into the first initial acoustic adaptation module. The first initial acoustic adaptation module uses the correspondence between acoustic modal features and text modal features to extract the acoustic vector sequence that corresponds one-to-one with each word in the recognition result from the continuous speech stream, which is the sample word-level acoustic feature. Here, the length of the sample word-level acoustic feature is the same as the length of the speech recognition result.

[0140] Subsequently, the sample word-level acoustic features and test item information are input into the scoring prediction component of the third oral assessment model. During this process, the updated pre-trained language model in the scoring prediction component (i.e., the pre-trained language model updated in the first stage) no longer receives the recognition result vector of plain text, but synchronously receives the sample word-level acoustic features generated by the adapter to fuse the sample word-level acoustic features and test item information to output global scoring features. Finally, the global scoring features are mapped to specific oral score values ​​through the initial scoring prediction model in the scoring prediction component to obtain the corresponding oral prediction score, i.e., the first oral prediction score. This process simulates the scenario where the pre-trained language model directly processes acoustic features for scoring.

[0141] Finally, based on freezing all parameters in the third oral assessment model except for the parameters of the first initial adapter, the parameters of the first initial adapter in the third oral assessment model are iteratively updated according to the loss value between the first oral prediction score and the actual oral score label. This allows the updated first initial adapter to map and align the original acoustic features to the text modality space (also known as the text feature space) of the pre-trained language model, thereby giving the pure text model the ability to perceive fine-grained pronunciation quality.

[0142] The method provided in this implementation effectively trains the first initialization adapter through a strategy of feature replacement and supervised training, achieving deep fusion of fine-grained acoustic features and text semantic features. This enables speech segments containing pronunciation defects to be recognized by the pre-trained language model and reflected in the final score prediction, significantly improving the model's accuracy in judging pronunciation details.

[0143] Based on the above embodiments, as an optional embodiment, the sample speech and the speech recognition result of the sample speech output by the initial speech recognition model are input to the first initial acoustic adaptation module to obtain the sample word-level acoustic features of the sample speech, including:

[0144] The sample speech is input into the first initialized acoustic encoder to obtain the pronunciation features of each word in the sample speech;

[0145] The speech recognition result is used as a query vector, and the pronunciation features are used as value vectors and key vectors. These are then input into the first initialization adapter for attention calculation to obtain the sample word-level acoustic features.

[0146] Figure 6 This is a schematic diagram of the structure of the first initialization acoustic adaptation module provided by the present invention.

[0147] like Figure 6 As shown, to achieve accurate alignment between continuous acoustic signals and discrete text words, the first initialization acoustic adaptation module adopts an attention-based architecture. Accordingly, the processing flow for sample word-level acoustic features is as follows:

[0148] First, a fine-grained acoustic coding step is performed. The sample speech is input into a first initial acoustic encoder, such as an initial acoustic encoder constructed by reusing the encoder part of the initial speech recognition model. This encoder performs deep feature extraction on the input speech waveform and outputs a high-density frame-level acoustic feature sequence. These feature sequences implicitly contain the pronunciation details of each word in the speech, i.e., the pronunciation features of each word.

[0149] Subsequently, the attention mechanism alignment step is executed. The speech recognition result, i.e., the text word sequence, decoded from the initial speech recognition model is converted into embedding vectors, which serve as the query vectors (Query, Q) in the attention mechanism. The embedding vector of each word represents the target semantic information that the model wants to find in the acoustic stream. At the same time, the pronunciation features of each word output from the first initial acoustic encoder are used as the key vectors (Key, K) and value vectors (Value, V) in the attention mechanism, respectively. These vectors contain all the acoustic detail information.

[0150] Next, the attention weight calculation and feature aggregation steps are performed. The first initialization adapter calculates the dot product similarity between the query vector and the key vector, thereby automatically deriving the attention distribution weight of each word in the entire acoustic feature sequence. Finally, based on the calculated weights, the value vectors are weighted and summed to obtain the aggregated acoustic feature vector corresponding to the word, which is the sample word-level acoustic feature.

[0151] Through the above calculations, the sequence length of the sample word-level acoustic features output by the first initialization adapter is strictly consistent with the number of words in the speech recognition result, and the vector at each position accurately integrates the pronunciation quality information of the word in the actual speech.

[0152] This embodiment utilizes a cross-attention mechanism, using speech recognition results as query guides to actively retrieve corresponding pronunciation features in a continuous acoustic stream. This achieves forced alignment of acoustics and text at the word level. This mechanism enables the model to accurately locate specific pronunciation errors, such as mispronouncing a word, omitting sounds, or incomplete pronunciation, and maps these micro-defects into feature biases that the pre-trained language model can understand, thereby significantly improving scoring accuracy.

[0153] Based on the above embodiments, as an optional embodiment, in order to achieve text-level acoustic feature alignment, the parameter update steps of the second initialization adapter in the third spoken language assessment model include:

[0154] The sample speech is input into the second initialization acoustic adaptation module to obtain the sample document-level acoustic features of the sample speech;

[0155] The sample text-level acoustic features, the speech recognition results, and the test question information are input into the scoring prediction component to obtain the second spoken language prediction score of the sample speech;

[0156] Freeze all parameters in the third oral assessment model except for the parameters of the second initialization adapter, and update the parameters of the second initialization adapter in the third oral assessment model according to the second oral prediction score and the oral score label.

[0157] Figure 7 This is a schematic diagram of the parameter update process for the second initialization adapter provided by the present invention.

[0158] like Figure 7 As shown, the parameter update of the second initialization adapter follows the logic from overall feature extraction to score feedback.

[0159] First, a macroscopic feature extraction step is performed. The sample speech is input into the second initialization acoustic adaptation module. In this step, the sample speech is first processed by the second initialization acoustic encoder (such as an encoder built based on an articulation level prediction model) to extract overall articulation level features containing macroscopic information such as the overall speech flow, prosody, and emotion of the entire speech. Subsequently, these overall articulation level features are fed into the second initialization adapter (such as a fully connected layer). The second initialization adapter is responsible for projecting these overall articulation level features onto a feature dimension that matches the pre-trained language model, ultimately outputting a fixed-length vector, namely the sample text-level acoustic features, which highly condenses the fluency, intonation, and prosodic expression of the entire speech.

[0160] Secondly, a hybrid feature scoring step is performed. To incorporate this coarse-grained acoustic feature (i.e., sample document-level acoustic feature) into the scoring process, a hybrid input stream is constructed: the sample document-level acoustic feature is converted into global acoustic information, for example, by concatenating the sample document-level acoustic feature after the [CLS] tag, or by directly fusing it with the [CLS] vector, and then fed into the scoring prediction component along with the regular text input (i.e., speech recognition results and test item information). During this process, the updated pre-trained language model in the scoring prediction component simultaneously processes text semantic information and global acoustic information, outputs the fused global scoring feature, and finally maps it to the initial scoring prediction model to obtain the second spoken prediction score.

[0161] Finally, the parameter optimization step is performed. The loss value between the second spoken language predicted score and the actual spoken language score label is calculated. Using the gradient descent algorithm, with all parameters in the third spoken language assessment model frozen except for the parameters of the second initial adapter, only the parameters of the second initial adapter in the third spoken language assessment model are iteratively updated. Through training, the updated second initial adapter learns to convert the overall articulation level features of speech into semantic signals that the pre-trained language model can understand.

[0162] This embodiment maps the overall pronunciation level features into the feature space of the pre-trained language model, enabling the pre-trained language model to perceive macro-prosodic information such as speech rate and pauses. This effectively supplements the lost sense of language information in the speech recognition text, and achieves effective quantification and accurate scoring of the overall oral performance.

[0163] Based on the above embodiments, as an optional embodiment, in order to mine higher-order interaction information between acoustic features and text features, the third stage of training steps includes:

[0164] The multi-scale acoustic features of the sample speech output by the updated initial multi-scale acoustic adaptation module, the speech recognition result of the sample speech output by the initial speech recognition model in the fourth oral assessment model, and the test question information are input into the updated pre-trained language model in the fourth oral assessment model to obtain the third oral prediction score of the sample speech.

[0165] Freeze the parameters of the updated initial multi-scale acoustic adaptation module and the parameters of the initial speech recognition model in the fourth oral assessment model. Based on the third oral prediction score and the oral score label, jointly train the module to be updated in the fourth oral assessment model to obtain the target oral assessment model.

[0166] The modules to be updated include all other modules in the fourth spoken language assessment model except for the updated initial multi-scale acoustic adaptation module and the initial speech recognition model.

[0167] Figure 8 This is a schematic diagram of the training process for the fourth oral language assessment model provided by the present invention.

[0168] like Figure 8 As shown, the third stage of training is no longer a single-modal warm-up or alignment, but a comprehensive joint optimization.

[0169] First, multimodal feature fusion and forward computation steps are performed. The input of the fourth oral assessment model simultaneously receives information from three sources: aligned multi-scale acoustic features (including word-level details and discourse-level prosody) generated by the updated initial multi-scale acoustic adaptation module; speech recognition results (i.e., text content) generated by the initial speech recognition model; and the original test item information. These three parts of information are organically organized into an input sequence and then input into the updated pre-trained language model. The multi-layer Transformer structure within the updated pre-trained language model performs deep cross-attention computation on these heterogeneous information, and finally, the initial scoring prediction model outputs the third oral prediction score.

[0170] Secondly, a selective parameter fine-tuning step is performed. To prevent significant adjustments to the pre-trained language model parameters from disrupting the modal alignment established in the previous stage during the initial joint training phase, this embodiment adopts a protective training strategy: freezing the parameters of the updated initial multi-scale acoustic adaptation module and the initial speech recognition model. This operation means that the mapping relationship from acoustic features to the text feature space remains stable and is not updated with backpropagation. Based on this, according to the loss value between the third spoken language prediction score and the spoken language score label, the backpropagation algorithm is used to focus on jointly training the modules to be updated in the fourth spoken language assessment model, namely, the modules other than the updated initial multi-scale acoustic adaptation module and the initial speech recognition model. Specifically, the modules to be updated mainly include the updated pre-trained language model and the initial score prediction model.

[0171] Through this joint training process, the pre-trained language model learns to simultaneously utilize textual semantic cues and multi-dimensional acoustic cues to make the final scoring judgment. The finally trained model no longer relies solely on the text content of the test taker's answer, but can directly consider multimodal information such as content, pronunciation, and fluency based on the input speech and test questions, outputting a more objective and comprehensive scoring result.

[0172] This embodiment freezes the aligned acoustic adaptation modules during the joint phase, focusing on fine-tuning the fusion capability of the pre-trained language model and the processing capability of other modules. This not only protects the modal alignment results of the previous stage, but also achieves complementary gains of multimodal information, enabling the final trained target oral assessment model to have extremely high robustness and human-like scoring ability in complex scenarios.

[0173] In some embodiments, this application also provides a method for oral language assessment. The method includes:

[0174] The speech to be evaluated and the corresponding test question information are input into the target oral assessment model to obtain the oral prediction score of the speech to be evaluated.

[0175] The target oral assessment model is trained based on the oral assessment model training method.

[0176] In some embodiments, the step of inputting the speech to be evaluated and the corresponding test question information into the target oral assessment model to obtain the oral prediction score of the speech to be evaluated includes:

[0177] The speech to be evaluated is input into the speech recognition model in the target spoken language assessment model to obtain the speech recognition result of the speech to be evaluated;

[0178] The speech to be evaluated and the speech recognition result of the speech to be evaluated are input into the first acoustic adaptation module of the target spoken language evaluation model to obtain the word-level acoustic features of the speech to be evaluated.

[0179] The speech to be evaluated is input into the second acoustic adaptation module of the target spoken language assessment model to obtain the text-level acoustic features of the speech to be evaluated.

[0180] The speech recognition results of the speech to be evaluated, the word-level acoustic features, the text-level acoustic features, and the test question information corresponding to the speech to be evaluated are input into the language model of the target oral assessment model to obtain global scoring features;

[0181] The global scoring features are input into the scoring prediction model of the target spoken language assessment model to obtain the spoken language prediction score of the speech to be assessed.

[0182] Figure 9 This is one of the flowcharts of the oral assessment method provided by the present invention; Figure 10 This is the second flowchart of the oral assessment method provided by the present invention.

[0183] like Figure 9 and Figure 10As shown, the target spoken language assessment model trained through the above embodiments may include at least a speech recognition model, a multi-scale acoustic adaptation module, a language model, and a scoring prediction model. The multi-scale acoustic adaptation module includes a first acoustic adaptation module and a second acoustic adaptation module. The first acoustic adaptation module includes a first adapter and a first acoustic encoder, and the second acoustic adaptation module includes a second adapter and a second acoustic encoder. The speech recognition model is an initialized speech recognition model pre-trained on a large-scale speech dataset. The scoring prediction model is obtained after training the initialized scoring prediction model in the third stage. The first and second acoustic adaptation modules are obtained by correspondingly training the first and second initialized acoustic adaptation modules in the second stage, respectively. The language model is obtained by further fine-tuning the pre-trained language model trained in the first stage in the third stage.

[0184] In practical applications, the system can acquire the audio recordings to be assessed for oral communication tests in real time. Simultaneously, it can acquire the corresponding test paper information to extract test question information. This information may include the question description, topic background, specific answer requirements, and pre-prepared reference answers based on the question or topic.

[0185] After acquiring the speech to be evaluated, its basic acoustic features can be extracted. These features can be spectral features reflecting human hearing characteristics, such as filter bank features. Specifically, the speech is first segmented into frames, then pre-emphasized, and finally the spectral features of each frame are extracted sequentially. Based on this, a speech endpoint detection model is used to extract spoken segments, determining the start and end boundaries of the speech, removing silence and background noise, and extracting the spoken segments containing actual speech content as valid input to the speech recognition model in the target spoken language evaluation model.

[0186] Based on this, in order to achieve accurate extraction and fusion of multimodal information, the following reasoning logic and data flow are specifically executed within the target oral assessment model:

[0187] First, a text transcription process is performed. The extracted spoken segments of the speech to be evaluated are input into the speech recognition model within the target spoken language assessment model to obtain the speech recognition result. In this process, the speech recognition model employs an end-to-end recognition model based on an encoder-decoder architecture. The encoder uses a convolutional enhanced transformer structure to capture the deep acoustic features of the speech, while the decoder uses a transformer structure. During decoding, a beam search algorithm is used to autoregressively predict the next sub-word. After decoding, the sub-words are restored to words, thus obtaining an accurate text transcription sequence as the speech recognition result.

[0188] Secondly, a fine-grained acoustic feature extraction step is performed. The speech to be evaluated and its speech recognition results are input into the first acoustic adaptation module of the target spoken language assessment model to obtain word-level acoustic features of the speech to be evaluated. Specifically, the first acoustic adaptation module uses the first acoustic encoder to encode the pronunciation features of each word, and uses the semantic features of each word in the speech recognition results as query vectors, and the pronunciation features as key vectors and value vectors, and sends them into the first adaptation module. The attention mechanism network inside the first adaptation module is used for calculation, thereby accurately aligning the text features and acoustic features, generating word-level acoustic features that correspond one-to-one with each word and contain details of pronunciation accuracy. These word-level acoustic features are aligned with the text modal space of the language model.

[0189] Next, a coarse-grained acoustic feature extraction step is performed. The speech to be evaluated is input into the second acoustic adaptation module of the target spoken language assessment model to obtain the discourse-level acoustic features of the speech. Specifically, the second acoustic adaptation module adds specific classification labels before the feature sequence of the input speech and then sends it to the second acoustic encoder to extract the overall articulation level features; the size of the overall articulation level features is 1xD, where D is the size of the hidden layer vector. The output of the second acoustic encoder is then processed by a second adapter (such as a fully connected layer) to accurately align the text features with the acoustic features, generating discourse-level acoustic features that characterize the overall speech rate, fluency, and prosody. These discourse-level acoustic features are aligned with the text modal space of the language model.

[0190] Subsequently, a multimodal feature deep fusion step is performed. The speech recognition results of the speech to be evaluated, word-level acoustic features, text-level acoustic features, and the corresponding test question information are input into the language model of the target oral assessment model to obtain global scoring features. In this step, the language model simultaneously receives text semantic information, fine-grained pronunciation features, and coarse-grained prosodic features. Through a multi-layer self-attention mechanism within the model, information is interacted and deeply fused, outputting a global feature vector containing comprehensive evaluation information such as pronunciation quality, fluency, and relevance to the topic.

[0191] Finally, the final score prediction step is performed. The global scoring features are input into the score prediction model of the target oral assessment model to obtain the oral prediction score of the speech to be assessed. The score prediction model here usually adopts a fully connected layer structure, which can accurately map the high-dimensional global scoring features into specific oral scores, thus completing the assessment task.

[0192] The method provided in this embodiment, through an end-to-end inference architecture that integrates high-precision speech recognition, dual-stream multi-scale acoustic feature collaborative extraction, and multimodal deep fusion, not only overcomes the scoring blind spots caused by the loss of pronunciation details and recognition errors in traditional pure text scoring schemes, but also achieves a comprehensive and human-like joint evaluation of the accuracy of candidates' spoken pronunciation, fluency of expression, and relevance of content, significantly improving the discrimination, reliability, and fairness of machine spoken scoring.

[0193] Figure 11 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 11 As shown, the electronic device may include: a processor 1110, a communication interface 1120, a memory 1130, and a communication bus 1140, wherein the processor 1110, the communication interface 1120, and the memory 1130 communicate with each other through the communication bus 1140. The processor 1110 can call logical instructions in the memory 1130 to execute a spoken language assessment model training method. This method includes: acquiring a training dataset; the training dataset includes multiple training samples, the training samples including sample speech, test item information corresponding to the sample speech, and spoken language scoring labels; updating the parameters of a pre-trained language model in a first spoken language assessment model according to the training dataset to obtain a second spoken language assessment model; constructing a third spoken language assessment model based on the second spoken language assessment model and an initialized multi-scale acoustic adaptation module; freezing the parameters to be frozen in the third spoken language assessment model; and, based on the training dataset, updating the parameters of the initialized multi-scale acoustic adaptation module in the third spoken language assessment model. The parameters of the learning adaptation module are updated to obtain a fourth oral assessment model; the updated initial multi-scale acoustic adaptation module is used to output multi-scale acoustic features aligned with the text modality space of the pre-trained language model; the parameters to be frozen include at least the parameters of the second oral assessment model; the fourth oral assessment model is trained according to the training dataset to obtain a target oral assessment model; the target oral assessment model is used to predict oral scores based on the multi-scale acoustic features and text features; or, an oral assessment method, which includes: inputting the speech to be assessed and the test question information corresponding to the speech to be assessed into the target oral assessment model to obtain the oral prediction score of the speech to be assessed.

[0194] Furthermore, the logical instructions in the aforementioned memory 1130 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0195] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the oral assessment model training method provided by the above methods. The method includes: acquiring a training dataset; the training dataset includes multiple training samples, the training samples including sample speech, test question information corresponding to the sample speech, and oral score labels; updating the parameters of the pre-trained language model in the first oral assessment model according to the training dataset to obtain a second oral assessment model; constructing a third oral assessment model according to the second oral assessment model and initializing a multi-scale acoustic adaptation module; and freezing the parameters to be frozen in the third oral assessment model. Based on the training dataset, the parameters of the initial multi-scale acoustic adaptation module in the third oral assessment model are updated to obtain a fourth oral assessment model. The updated initial multi-scale acoustic adaptation module is used to output multi-scale acoustic features aligned with the text modality space of the pre-trained language model. The parameters to be frozen include at least the parameters of the second oral assessment model. Based on the training dataset, the fourth oral assessment model is trained to obtain a target oral assessment model. The target oral assessment model is used to predict oral scores based on the multi-scale acoustic features and text features. Alternatively, an oral assessment method is provided, comprising: inputting the speech to be assessed and the test question information corresponding to the speech to be assessed into the target oral assessment model to obtain the oral prediction score of the speech to be assessed.

[0196] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a method for training a spoken language assessment model provided by the methods described above. This method includes: acquiring a training dataset; the training dataset including multiple training samples, the training samples including sample speech, test item information corresponding to the sample speech, and spoken language scoring labels; updating the parameters of a pre-trained language model in a first spoken language assessment model according to the training dataset to obtain a second spoken language assessment model; constructing a third spoken language assessment model based on the second spoken language assessment model and an initialized multi-scale acoustic adaptation module; freezing the parameters to be frozen in the third spoken language assessment model, and, based on the training dataset, adjusting the parameters of the pre-trained language model in the first spoken language assessment model ... and, based on the training dataset, adjusting the parameters of the pre-trained language model in the first spoken language assessment model to obtain a second spoken language assessment model. The parameters of the initial multi-scale acoustic adaptation module in the third oral assessment model are updated to obtain the fourth oral assessment model; the updated initial multi-scale acoustic adaptation module is used to output multi-scale acoustic features aligned with the text modality space of the pre-trained language model; the parameters to be frozen include at least the parameters of the second oral assessment model; the fourth oral assessment model is trained according to the training dataset to obtain the target oral assessment model; the target oral assessment model is used to predict oral scores based on the multi-scale acoustic features and text features; or, an oral assessment method, which includes: inputting the speech to be assessed and the test question information corresponding to the speech to be assessed into the target oral assessment model to obtain the oral prediction score of the speech to be assessed.

[0197] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0198] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0199] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training an oral assessment model, characterized in that, include: Obtain the training dataset; The training dataset includes multiple training samples, which include sample speech, test question information corresponding to the sample speech, and oral score labels. Based on the training dataset, the parameters of the pre-trained language model in the first oral assessment model are updated to obtain the second oral assessment model; the first oral assessment model includes the pre-trained language model, the initialized speech recognition model, and the initialized score prediction model. Based on the second oral assessment model and the initial multi-scale acoustic adaptation module, a third oral assessment model is constructed. Freeze the parameters to be frozen in the third spoken language assessment model, and update the parameters of the initial multi-scale acoustic adaptation module in the third spoken language assessment model according to the training dataset to obtain the fourth spoken language assessment model; the updated initial multi-scale acoustic adaptation module is used to output multi-scale acoustic features aligned with the text modality space of the pre-trained language model. The parameters to be frozen include at least the parameters of the second oral assessment model; The fourth oral assessment model is trained based on the training dataset to obtain the target oral assessment model. The target oral assessment model includes: a speech recognition model for outputting speech recognition results based on the speech to be assessed; a multi-scale acoustic adaptation module for outputting multi-scale acoustic features based on the speech to be assessed and the speech recognition results; a language model for outputting global scoring features based on the speech recognition results, multi-scale acoustic features, and test item information corresponding to the speech to be assessed; and a scoring prediction model for outputting oral prediction scores based on the global scoring features.

2. The oral assessment model training method according to claim 1, characterized in that, The initialization multi-scale acoustic adaptation module includes a first initialization acoustic adaptation module and a second initialization acoustic adaptation module. The first initialization acoustic adaptation module includes a first initialization acoustic encoder and a first initialization adapter. The first initialization acoustic adaptation module is used to extract word-level acoustic features. The second initialization acoustic adaptation module includes a second initialization acoustic encoder and a second initialization adapter. The second initialization acoustic adaptation module is used to extract document-level acoustic features.

3. The oral assessment model training method according to claim 2, characterized in that, The parameters to be frozen also include the parameters of the first initial acoustic encoder and the parameters of the second initial acoustic encoder; The process involves freezing the parameters to be frozen in the third oral language assessment model, updating the parameters of the initial multi-scale acoustic adaptation module in the third oral language assessment model based on the training dataset, to obtain the fourth oral language assessment model, which includes: Freeze the parameters of the second oral assessment model, the parameters of the first initial acoustic encoder, and the parameters of the second initial acoustic encoder. Based on the training dataset, update the parameters of the first initial adapter and the second initial adapter in the third oral assessment model to obtain the fourth oral assessment model.

4. The oral assessment model training method according to claim 3, characterized in that, The second oral assessment model includes an updated pre-trained language model, an initialized speech recognition model, and an initialized score prediction model; The update steps for the parameters of the first initial adapter in the third oral assessment model include: The sample speech and the speech recognition result of the sample speech output by the initial speech recognition model are input into the first initial acoustic adaptation module to obtain the sample word-level acoustic features of the sample speech; The sample word-level acoustic features and the test question information are input into the scoring prediction component of the third oral assessment model to obtain the first oral prediction score of the sample speech; the scoring prediction component includes the updated pre-trained language model and the initial scoring prediction model. Freeze all parameters in the third oral assessment model except for the parameters of the first initialization adapter, and update the parameters of the first initialization adapter in the third oral assessment model according to the first oral prediction score and the oral score label.

5. The oral assessment model training method according to claim 4, characterized in that, The step of inputting the speech recognition results of the sample speech and the initial speech recognition model into the first initial acoustic adaptation module to obtain the sample word-level acoustic features of the sample speech includes: The sample speech is input into the first initialized acoustic encoder to obtain the pronunciation features of each word in the sample speech; The speech recognition result is used as a query vector, and the pronunciation features are used as value vectors and key vectors. These are then input into the first initialization adapter for attention calculation to obtain the sample word-level acoustic features.

6. The oral assessment model training method according to claim 4, characterized in that, The update steps for the parameters of the second initialization adapter in the third oral assessment model include: The sample speech is input into the second initialization acoustic adaptation module to obtain the sample document-level acoustic features of the sample speech; The sample text-level acoustic features, the speech recognition results, and the test question information are input into the scoring prediction component to obtain the second spoken language prediction score of the sample speech; Freeze all parameters in the third oral assessment model except for the parameters of the second initialization adapter, and update the parameters of the second initialization adapter in the third oral assessment model according to the second oral prediction score and the oral score label.

7. The oral assessment model training method according to claim 4, characterized in that, The first initial acoustic encoder is constructed based on the encoding module in the initial speech recognition model, and the second initial acoustic encoder is constructed based on the encoder in the pronunciation level prediction model; the pronunciation level prediction model is trained based on the sample speech and the overall pronunciation level label corresponding to the sample speech.

8. The oral assessment model training method according to any one of claims 1-7, characterized in that, The step of training the fourth oral assessment model based on the training dataset to obtain the target oral assessment model includes: The multi-scale acoustic features of the sample speech output by the updated initial multi-scale acoustic adaptation module, the speech recognition result of the sample speech output by the initial speech recognition model in the fourth oral assessment model, and the test question information are input into the updated pre-trained language model in the fourth oral assessment model to obtain the third oral prediction score of the sample speech. Freeze the parameters of the updated initial multi-scale acoustic adaptation module and the parameters of the initial speech recognition model in the fourth oral assessment model. Based on the third oral prediction score and the oral score label, jointly train the module to be updated in the fourth oral assessment model to obtain the target oral assessment model. The module to be updated includes all other modules in the fourth oral assessment model except for the updated initial multi-scale acoustic adaptation module and the initial speech recognition model.

9. A method for oral language assessment, characterized in that, include: The speech to be tested and the corresponding test question information are input into the target oral assessment model to obtain the oral prediction score of the speech to be tested. The target oral assessment model is trained based on the oral assessment model training method described in any one of claims 1-8.

10. The oral assessment method according to claim 9, characterized in that, The step of inputting the speech to be evaluated and the corresponding test question information into the target oral assessment model to obtain the oral prediction score of the speech to be evaluated includes: The speech to be evaluated is input into the speech recognition model in the target spoken language assessment model to obtain the speech recognition result of the speech to be evaluated; The speech to be evaluated and the speech recognition result of the speech to be evaluated are input into the first acoustic adaptation module of the target spoken language evaluation model to obtain the word-level acoustic features of the speech to be evaluated. The speech to be evaluated is input into the second acoustic adaptation module of the target spoken language assessment model to obtain the text-level acoustic features of the speech to be evaluated. The speech recognition results of the speech to be evaluated, the word-level acoustic features, the text-level acoustic features, and the test question information corresponding to the speech to be evaluated are input into the language model of the target oral assessment model to obtain global scoring features; The global scoring features are input into the scoring prediction model of the target spoken language assessment model to obtain the spoken language prediction score of the speech to be assessed.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the oral assessment model training method as described in any one of claims 1 to 8, or the oral assessment method as described in any one of claims 9 to 10.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the oral assessment model training method as described in any one of claims 1 to 8, or the oral assessment method as described in any one of claims 9 to 10.

Images