Audio and text based english spoken language scoring method, apparatus, device and medium

Abstract

The application relates to the fields of artificial intelligence and scoring, and discloses an English oral scoring method and device based on audio and text, equipment and a medium. The method comprises the following steps: training a deep learning model by using a training set, and obtaining the loss of the deep learning model on the training set by using a loss function; when the loss of the deep learning model on the training set meets a preset condition, saving the trained deep learning model, wherein the trained deep learning model comprises an audio encoder, a text encoder and a prediction module; determining a fusion feature vector of current English oral audio based on the audio encoder and the text encoder, inputting the fusion feature vector of the current English oral audio into the prediction module, and generating scoring information of the current English oral audio and feedback information of the current English oral audio by using the prediction module. The application can automatically generate the scoring information of the current English oral audio and the feedback information of the current English oral audio.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence technology and scoring technology, and in particular to methods, apparatus, devices and media for English oral scoring based on audio and text. Background Technology

[0002] In language learning and English oral exam scenarios, it is necessary not only to provide quantitative scoring information to objectively reflect the speaker's oral proficiency, but also to provide targeted feedback information to indicate specific areas for improvement in pronunciation, grammar, or fluency, thereby providing effective guidance for the speaker's subsequent learning.

[0003] However, the current scoring and feedback information for spoken English audio relies on manual processing. This manual process requires scorers to repeatedly listen to the audio, meticulously annotating multiple dimensions such as pronunciation accuracy, fluency, and grammatical correctness. They also need to manually compare the audio to preset standard pronunciation templates. This entire process is not only time-consuming and lengthy but also easily influenced by the scorers' subjective judgments, thus hindering the efficiency of obtaining scoring and feedback information for spoken English audio. Therefore, how to generate scoring and feedback information for spoken English audio is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] This application provides an English speaking scoring method, apparatus, device, and medium based on audio and text to solve the aforementioned technical problem of how to generate scoring information and feedback information for the current English speaking audio.

[0005] In a first aspect, embodiments of this application provide an English speaking assessment method based on audio and text, applied to an electronic device, the English speaking assessment method comprising: Obtain a preset English spoken audio, perform speech recognition on the preset English spoken audio, and generate a transcribed text of the preset English spoken audio; A training sample is composed of preset English speaking audio, preset English speaking audio transcribed text, preset English speaking audio scoring information, and preset English speaking audio feedback information. Different training samples are combined into a training set, and the deep learning model is trained using the training set. The loss of the deep learning model on the training set is obtained through the loss function. When the loss of the deep learning model on the training set meets the preset conditions, the trained deep learning model is saved. The trained deep learning model includes an audio encoder, a text encoder, and a prediction module. Based on the audio encoder and text encoder, the fusion feature vector of the current English spoken audio is determined. The fusion feature vector of the current English spoken audio is input into the prediction module, which generates the scoring information and feedback information of the current English spoken audio.

[0006] In one possible implementation of the first aspect, a training sample is composed of preset English spoken audio, the transcribed text of the preset English spoken audio, the scoring information of the preset English spoken audio, and the feedback information of the preset English spoken audio, including: Obtain the labeled data corresponding to the preset English spoken audio. From the labeled data corresponding to the preset English spoken audio, obtain the scoring information and feedback information of the preset English spoken audio. Combine the preset English spoken audio, the transcribed text of the preset English spoken audio, the scoring information of the preset English spoken audio, and the feedback information of the preset English spoken audio into a training sample.

[0007] In one possible implementation of the first aspect, different training samples are combined into a training set, the deep learning model is trained using the training set, and the loss of the deep learning model on the training set is obtained through a loss function, including: Different training samples are combined into a training set, and the deep learning model is trained using the training set. During the iterative training of the deep learning model, the loss of the deep learning model on the training set is obtained through a loss function, which includes one of the following: mean squared error loss function, Huber loss function, and ranking loss function.

[0008] In one possible implementation of the first aspect, when the loss of the deep learning model on the training set meets a preset condition, the trained deep learning model is saved. The trained deep learning model includes an audio encoder, a text encoder, and a prediction module, comprising: When the loss of the deep learning model on the training set is less than a preset value, a stop command is generated. By executing the stop command, the training of the deep learning model is stopped, and the trained deep learning model is saved. The trained deep learning model includes an audio encoder, a text encoder, and a prediction module.

[0009] In one possible implementation of the first aspect, a fusion feature vector of the current spoken English audio is determined based on an audio encoder and a text encoder. This fusion feature vector is then input into a prediction module, which generates scoring information and feedback information for the current spoken English audio, including: The current English spoken audio is obtained, and an audio encoder is used to extract features from the current English spoken audio to obtain the acoustic feature vector, fluency feature vector, and pronunciation feature vector of the current English spoken audio. The current English spoken audio is converted into speech recognition and transcribed into a text. A text encoder is then used to extract features from the transcribed text, generating lexical feature vectors, grammatical feature vectors, and content feature vectors for the current English spoken audio. Using a preset fusion method, the acoustic feature vector, fluency feature vector, pronunciation feature vector, vocabulary feature vector, grammar feature vector, and content feature vector of the current English spoken audio are fused to generate a fused feature vector of the current English spoken audio. The fused feature vector of the current English spoken audio is input into the prediction module, which generates the scoring information and feedback information of the current English spoken audio.

[0010] In one possible implementation of the first aspect, after determining the concatenated feature vector of the current spoken English audio through an audio encoder, determining the feature vector of the transcribed text of the current spoken English audio through a text encoder, fusing the concatenated feature vector of the current spoken English audio and the feature vector of the transcribed text of the current spoken English audio to generate a fused feature vector of the current spoken English audio, inputting the fused feature vector of the current spoken English audio into a prediction module, and generating the scoring information and feedback information of the current spoken English audio through the prediction module, the spoken English scoring method includes: Write the scoring information and feedback information of the current English speaking audio into a preset scoring template to generate a scoring report for the current English speaking audio.

[0011] In one possible implementation of the first aspect, the preset spoken English audio is already evaluated spoken English audio; The current English speaking audio is an English speaking audio to be evaluated; The scoring information for current English speaking audio includes the overall score, fluency score, and pronunciation score. The feedback information for the current English speaking audio includes the reasons for the deduction of points and suggestions for improvement. The scoring information for the preset English speaking audio includes the overall score, the fluency score, and the pronunciation score. Feedback information for the preset English speaking audio includes the reasons for the deduction of points in the preset English speaking audio and suggestions for improvement in the preset English speaking audio; The preset fusion method adopts one of the following: vector dimension splicing, weighted summation, gating dynamic filtering, attention weight allocation, or cross-attention interaction. The prediction module includes at least one of a multilayer perceptron and a regression head.

[0012] Secondly, embodiments of this application provide an English oral assessment device based on audio and text, applied to an electronic device, comprising: The first acquisition module is used to acquire preset English spoken audio, perform speech recognition conversion on the preset English spoken audio, and generate a transcribed text of the preset English spoken audio. The component module is used to combine preset English spoken audio, the transcribed text of preset English spoken audio, the scoring information of preset English spoken audio, and the feedback information of preset English spoken audio into a training sample. The second acquisition module is used to form a training set from different training samples, use the training set to train the deep learning model, and obtain the loss of the deep learning model on the training set through the loss function. The save module is used to save the trained deep learning model when the loss of the deep learning model on the training set meets the preset conditions. The trained deep learning model includes an audio encoder, a text encoder, and a prediction module. The scoring module is used to determine the fusion feature vector of the current English spoken audio based on the audio encoder and the text encoder. The fusion feature vector of the current English spoken audio is input into the prediction module, which generates the scoring information and feedback information of the current English spoken audio.

[0013] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the English oral assessment method described in the first aspect above.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the English oral assessment method described in the first aspect above.

[0015] Fifthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to execute the English oral assessment method described in the first aspect above.

[0016] The beneficial effects of the embodiments of this application are as follows: Firstly, based on the audio encoder and text encoder, the fusion feature vector of the current English spoken audio is determined. The fusion feature vector of the current English spoken audio is input into the prediction module, which generates the scoring information and feedback information of the current English spoken audio. Since no manual operation is required, the time for obtaining the scoring information and feedback information of the current English spoken audio is reduced, which is conducive to improving the efficiency of obtaining the scoring information and feedback information of the current English spoken audio. Secondly, since the scoring information and feedback information of the current English speaking audio are automatically generated, they are not affected by human intervention, which helps to improve the reliability of the scoring information and feedback information of the current English speaking audio. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the English oral assessment method provided in the embodiments of this application; Figure 2 A schematic block diagram of an English oral assessment device provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0020] The English oral assessment method provided in this application can be applied to electronic devices, including but not limited to servers, mobile phones, tablets, and laptops. This application does not impose any restrictions on the specific type of electronic device.

[0021] Please see Figure 1 , Figure 1 This is a flowchart illustrating the English oral proficiency scoring method provided in this application embodiment, which can be applied to electronic devices.

[0022] like Figure 1 As shown in the embodiments of this application, the English oral assessment method includes the following steps, which are detailed below: S101, Obtain a preset English spoken audio, perform speech recognition conversion on the preset English spoken audio, and generate a transcribed text of the preset English spoken audio; The preset English speaking audio is a pre-tested English speaking audio recording. It is recorded by a preset speaker in a standard recording environment according to preset corpus content.

[0023] The pre-selected speakers include any one of the following: English teachers with professional language teaching qualifications, professional voice actors whose native language is English, or language experts.

[0024] S102, a training sample is formed by combining the preset English spoken audio, the transcribed text of the preset English spoken audio, the scoring information of the preset English spoken audio, and the feedback information of the preset English spoken audio. The training sample consists of preset English speaking audio, the transcribed text of the preset English speaking audio, the scoring information of the preset English speaking audio, and the feedback information of the preset English speaking audio, including: Obtain the labeled data corresponding to the preset English spoken audio. From the labeled data corresponding to the preset English spoken audio, obtain the scoring information and feedback information of the preset English spoken audio. Combine the preset English spoken audio, the transcribed text of the preset English spoken audio, the scoring information of the preset English spoken audio, and the feedback information of the preset English spoken audio into a training sample.

[0025] The scoring information for the preset English speaking audio includes the overall score, the fluency score, and the pronunciation score. Feedback information for the preset English speaking audio includes the reasons for the deduction of points and suggestions for improvement.

[0026] The overall score of the preset English speaking audio is a quantitative score obtained by comprehensively evaluating the overall speaking level of the preset English speaking audio, used to reflect the overall speaking ability of the preset speaker. The higher the overall score of the preset English speaking audio, the better the overall speaking level of the preset speaker and the smaller the overall deviation from standard spoken expression; the lower the overall score of the preset English speaking audio, the worse the overall speaking level of the preset speaker and the greater the deviation from standard spoken expression.

[0027] The fluency score of a preset English speaking audio recording is a quantitative score obtained by evaluating the fluency of the preset speaker's pronunciation. It reflects the coherence and smoothness of the preset speaker's spoken expression. The higher the fluency score of the preset English speaking audio recording, the more fluent the preset speaker's spoken expression; the lower the fluency score of the preset English speaking audio recording, the less fluent the preset speaker's spoken expression.

[0028] The pronunciation score of a preset English speaking audio recording is a quantitative score obtained by evaluating the accuracy of phoneme pronunciation in the preset English speaking audio recording. It is used to reflect the accuracy of the preset English speaking audio recording in terms of word pronunciation, phoneme pronunciation, and stress control. The higher the pronunciation score of the preset English speaking audio recording, the more standard the preset speaker's pronunciation is; the lower the pronunciation score of the preset English speaking audio recording, the less standard the preset speaker's pronunciation is.

[0029] S103: Different training samples are combined into a training set, and the deep learning model is trained using the training set. The loss of the deep learning model on the training set is obtained through the loss function. This involves forming a training set from different training samples, using the training set to train the deep learning model, and obtaining the loss of the deep learning model on the training set through a loss function, including: Different training samples are combined into a training set, and the deep learning model is trained using the training set. During the iterative training of the deep learning model, the loss of the deep learning model on the training set is obtained through a loss function, which includes one of the following: mean squared error loss function, Huber loss function, and ranking loss function.

[0030] S104. When the loss of the deep learning model on the training set meets the preset conditions, save the trained deep learning model. The trained deep learning model includes an audio encoder, a text encoder, and a prediction module. Specifically, when the loss of the deep learning model on the training set meets a preset condition, the trained deep learning model is saved. The trained deep learning model includes an audio encoder, a text encoder, and a prediction module, including: When the loss of the deep learning model on the training set is less than a preset value, a stop command is generated. By executing the stop command, the training of the deep learning model is stopped, and the trained deep learning model is saved. The trained deep learning model includes an audio encoder, a text encoder, and a prediction module.

[0031] S105, based on the audio encoder and text encoder, determines the fusion feature vector of the current English spoken audio, inputs the fusion feature vector of the current English spoken audio into the prediction module, and generates the scoring information and feedback information of the current English spoken audio through the prediction module.

[0032] Specifically, based on the audio encoder and text encoder, a fusion feature vector of the current English spoken audio is determined. This fusion feature vector is then input into the prediction module, which generates scoring information and feedback information for the current English spoken audio, including: The current English spoken audio is obtained, and an audio encoder is used to extract features from the current English spoken audio to obtain the acoustic feature vector, fluency feature vector, and pronunciation feature vector of the current English spoken audio. The current English spoken audio is converted into speech recognition and transcribed into a text. A text encoder is then used to extract features from the transcribed text, generating lexical feature vectors, grammatical feature vectors, and content feature vectors for the current English spoken audio. Using a preset fusion method, the acoustic feature vector, fluency feature vector, pronunciation feature vector, vocabulary feature vector, grammar feature vector, and content feature vector of the current English spoken audio are fused to generate a fused feature vector of the current English spoken audio. The fused feature vector of the current English spoken audio is input into the prediction module, which generates the scoring information and feedback information of the current English spoken audio.

[0033] The acoustic feature vector of current spoken English audio includes at least one of the following: energy, fundamental frequency, Mel-frequency cepstral coefficients, zero-crossing rate, silence ratio, and loudness.

[0034] Energy refers to the cumulative average power or squared amplitude of the current spoken English audio signal per unit time. Higher energy indicates a louder pronunciation of the target speaker in spoken English; lower energy indicates a weaker pronunciation of the target speaker in spoken English.

[0035] The fundamental frequency (FFF) is the basic frequency of vocal cord vibration in the current English spoken audio signal. It is used to characterize the pitch of the target speaker and is measured in Hertz (Hz). A higher FFF indicates a higher pitch in spoken English, while a lower FFF indicates a lower pitch.

[0036] Mel-frequency cepstral coefficients are cepstral feature parameters extracted after performing Mel frequency scaling on the current spoken English audio signal. The larger the feature distance between the Mel-frequency cepstral coefficients of the current spoken English audio and the Mel-frequency cepstral coefficients of the preset standard pronunciation template, the greater the difference between the target speaker's spoken English spectral characteristics and the standard pronunciation; the smaller the feature distance between the Mel-frequency cepstral coefficients of the current spoken English audio and the Mel-frequency cepstral coefficients of the preset standard pronunciation template, the closer the target speaker's spoken English spectral characteristics are to the standard pronunciation.

[0037] The zero-crossing rate refers to the number of times the current English spoken audio signal crosses the zero level per unit time, used to characterize the frequency of changes in the audio signal waveform. A higher zero-crossing rate indicates that the target speaker's pronunciation in English spoken language contains more voiceless or fricative components; a lower zero-crossing rate indicates that the target speaker's pronunciation in English spoken language contains more voiced or vowel components.

[0038] The silence ratio refers to the percentage of silent or non-speech segments in a current English spoken audio recording, representing the degree of pauses during the speaker's pronunciation. A higher silence ratio indicates more pauses or a slower speaking speed in the target speaker's English spoken language; a lower silence ratio indicates more concise pronunciation and a faster speaking speed in the target speaker's English spoken language.

[0039] Loudness refers to the subjective volume of the current spoken English audio signal after being weighted by human hearing, and is used to characterize the perceived intensity of the target speaker's pronunciation. The higher the loudness, the louder and clearer the target speaker's pronunciation in spoken English; the lower the loudness, the weaker and more indistinct the target speaker's pronunciation in spoken English.

[0040] The fluency feature vector of current English spoken audio includes at least one of the following: total answer time, effective vocalization time, total number of words, total number of syllables, average speech rate, average pronunciation rate, average pause duration between words, number of pauses, total pause duration, number of hesitant words, and number of repeated words.

[0041] Total response time refers to the total duration of the current English speaking audio. The longer the total response time, the more time the target speaker spends on English speaking; the shorter the total response time, the less time the target speaker spends on English speaking.

[0042] Effective vocalization duration refers to the total duration of speech in a current English spoken audio recording after removing silence segments. The longer the effective vocalization duration, the more actual pronunciation time the target speaker has in English spoken language; the shorter the effective vocalization duration, the less actual pronunciation time the target speaker has in English spoken language.

[0043] The total word count refers to the total number of words spoken by the target speaker in the current English spoken audio. A higher total word count indicates a richer expression in spoken English, while a lower total word count indicates a poorer expression in spoken English.

[0044] The total number of syllables refers to the total number of syllables produced by the target speaker in the current English spoken audio. A higher total number of syllables indicates that the target speaker produces more syllables in spoken English; a lower total number of syllables indicates that the target speaker produces fewer syllables in spoken English.

[0045] Average speaking rate (AP) is the total number of words in a spoken English audio recording divided by the total answering time. A higher average PAP indicates that the target speaker speaks English faster overall; a lower average PAP indicates that the target speaker speaks English slower overall.

[0046] Average rate of speech (APS) is the total number of syllables in a spoken English audio recording divided by the effective duration of speech. A higher APS indicates that the target speaker is speaking English at a faster pace; a lower APS indicates that the target speaker is speaking English at a slower pace.

[0047] Average pause duration refers to the total duration of all pauses between words in the current English spoken audio, divided by the number of pauses. A longer average pause duration indicates that the target speaker spends more time pausing between words on average in spoken English; a shorter average pause duration indicates that the target speaker spends less time pausing between words on average in spoken English. The number of pauses refers to the total number of pauses that occur during the target speaker's pronunciation in the current English spoken audio. A higher number of pauses indicates that the target speaker pauses more frequently in spoken English; a lower number of pauses indicates that the target speaker pauses less frequently in spoken English.

[0048] The total pause duration refers to the sum of the durations of all pauses by the target speaker in the current English spoken audio. The longer the total pause duration, the more time the target speaker spends pausing in English spoken language; the shorter the total pause duration, the less time the target speaker spends pausing in English spoken language.

[0049] The number of hesitation words refers to the total number of times the target speaker uses hesitation words in the current English spoken audio. The more hesitation words there are, the higher the degree of hesitation in the target speaker's expression in English spoken language; the fewer hesitation words there are, the lower the degree of hesitation in the target speaker's expression in English spoken language.

[0050] Hesitation words are filler words or phonemes that the target speaker uses during spoken expression to gain thinking time or fill gaps in the flow of speech. They do not have actual semantic content and are used to characterize the degree of hesitation and frequency of interruptions in the target speaker's speech.

[0051] The number of repeated words refers to the total number of times the target speaker repeats words or phrases in the current English spoken audio. A higher number of repeated words indicates a higher degree of repetition in the target speaker's spoken English; a lower number of repeated words indicates a lower degree of repetition. The pronunciation feature vector of the current English spoken audio includes at least one of the following parameters: phoneme duration parameter, number of missed phonemes, number of inserted phonemes, phoneme prolongation parameter, weak form parameter, and stress parameter.

[0052] Phoneme duration parameters refer to the pronunciation duration of each phoneme of the target speaker in the current spoken English audio. The greater the deviation between the phoneme duration parameters and the corresponding phonemes in the preset standard pronunciation template, the greater the difference between the target speaker's pronunciation duration in spoken English and the standard pronunciation; the smaller the duration deviation, the closer the target speaker's pronunciation duration in spoken English is to the standard pronunciation.

[0053] Phoneme omissions refer to the number of phonemes or syllables that the target speaker does not pronounce or misses in the current English spoken audio. A higher phoneme omission count indicates more severe phoneme omissions in the target speaker's spoken English; a lower count indicates milder phoneme omissions. Phoneme insertions refer to the number of extra, unexpected phonemes or syllables pronounced by the target speaker in the current English spoken audio. A higher phoneme insertion count indicates more severe phoneme insertions in the target speaker's spoken English; a lower count indicates milder phoneme insertions.

[0054] Phoneme prolongation parameter refers to the degree to which the target speaker prolongs a specific phoneme or syllable in the current spoken English audio. The greater the deviation between the phoneme prolongation parameter and the corresponding phoneme in the preset standard pronunciation template, the more pronounced the phoneme prolongation in spoken English by the target speaker; the smaller the deviation, the less pronounced the phoneme prolongation in spoken English by the target speaker.

[0055] The weak form parameter refers to the degree to which the target speaker weakens unstressed syllables in the current spoken English audio. The greater the deviation between the weak form parameter and the weak form pattern of the preset standard pronunciation template, the less standard the target speaker's weak form processing in spoken English; the smaller the deviation, the more standard the target speaker's weak form processing in spoken English.

[0056] Stress parameters refer to the degree to which the target speaker grasps word and sentence stress in the current English spoken audio. The greater the deviation of the stress parameters from the stress pattern of the preset standard pronunciation template, the less stable the target speaker's grasp of stress in spoken English; the smaller the deviation, the more stable the target speaker's grasp of stress in spoken English.

[0057] The English speaking scoring method includes the following steps: First, the concatenated feature vector of the current spoken English audio is determined by an audio encoder. Then, the feature vector of the transcribed text of the current spoken English audio is determined by a text encoder. Next, the concatenated feature vector and the transcribed text feature vector are fused to generate a fused feature vector of the current spoken English audio. Finally, this fused feature vector is input into a prediction module, which generates scoring information and feedback information for the current spoken English audio. Write the scoring information and feedback information of the current English speaking audio into a preset scoring template to generate a scoring report for the current English speaking audio.

[0058] Among them, the current English speaking audio is the English speaking audio to be evaluated; The English spoken audio to be evaluated was recorded in real time by the target speaker in a real-world application scenario using recording equipment. The target speaker includes, but is not limited to, test takers or language learners.

[0059] The preset fusion method adopts one of the following: vector dimension splicing, weighted summation, gating dynamic filtering, attention weight allocation, or cross-attention interaction; the prediction module includes at least one of multilayer perceptron and regression head.

[0060] The scoring information for current English speaking audio includes the overall score, fluency score, and pronunciation score. The feedback information for the current English speaking audio includes the reasons for the deduction of points and suggestions for improvement.

[0061] The scoring and feedback information of the current English speaking audio helps the target speaker quickly understand the evaluation of the current English speaking audio, improving the readability and usability of the evaluation results.

[0062] The overall score of a current English speaking audio recording is a quantitative score obtained by comprehensively evaluating the overall speaking level of the audio recording. It reflects the overall speaking ability of the target speaker. The higher the overall score, the better the overall speaking level of the target speaker and the smaller the overall deviation from standard spoken expression; conversely, the lower the overall score, the worse the overall speaking level of the target speaker and the greater the deviation from standard spoken expression.

[0063] The fluency score of a current English speaking audio recording is a quantitative score obtained by evaluating the fluency of the target speaker's pronunciation. It reflects the coherence and smoothness of the target speaker's spoken expression. The higher the fluency score of the current English speaking audio recording, the more fluent the target speaker's spoken expression; the lower the fluency score, the less fluent the target speaker's spoken expression.

[0064] The pronunciation score of a current English spoken audio recording is a quantitative score obtained by evaluating the accuracy of phoneme pronunciation in the audio. It reflects the accuracy of word pronunciation, phoneme reading, and stress control in the audio. A higher pronunciation score indicates a more standard pronunciation of the target speaker; a lower score indicates a less standard pronunciation of the target speaker.

[0065] The deduction reason for the current English speaking audio refers to the defect description information generated after diagnosing the specific defects of the target speaker in the current English speaking audio. This information indicates the specific reason why points are deducted from the target speaker's English speaking expression. The deduction reason includes a defect type identifier and a defect location identifier. The defect type identifier indicates the specific category of the defect, including but not limited to at least one of the following: phoneme omission, phoneme insertion, phoneme prolongation, improper weak forms, incorrect stress, inappropriate pauses, abnormal speaking rate, repeated words, and hesitant words. For ease of explanation, an example is given below: For example, the current reason for deducting points in English speaking audio is: there are multiple instances of long pauses, which reduce the fluency of spoken expression. For example, the current reason for deducting points in English speaking audio is: there are multiple instances of hesitant words, which reduce the fluency of spoken expression. The improvement suggestions for current English speaking audio refer to corrective guidance information generated based on the aforementioned reasons for score deductions, providing specific operational guidance for the target speaker to correct speaking deficiencies. For example, the improvement suggestion for current English speaking audio might be: the current audio contains a high proportion of silence, resulting in a low fluency score. It is recommended to reduce inappropriate pauses during pronunciation to improve the fluency score. For clarity, an example is provided below: For example, if the frequency of hesitation words in the current spoken audio exceeds the preset threshold, resulting in a low fluency score, it is recommended to reduce the use of filler words to improve the fluency score.

[0066] For the English oral assessment method described in the above embodiments, please refer to [link / reference]. Figure 2 , Figure 2 This is a schematic block diagram of an English speaking assessment device provided in an embodiment of this application. Figure 2 The English speaking assessment device 200 shown can be applied to, for example... Figure 1The application scenario diagram shows electronic devices. The following section uses electronic devices as an example to illustrate this. Figure 2 The English speaking assessment device 200 shown will be described in detail. The English speaking assessment device 200 may include a first acquisition module 201, a composition module 202, a second acquisition module 203, a storage module 204, and a scoring module 205.

[0067] The first acquisition module 201 is used to acquire a preset English spoken audio, perform speech recognition conversion on the preset English spoken audio, and generate a transcribed text of the preset English spoken audio. Module 202 is used to combine preset English spoken audio, the transcribed text of preset English spoken audio, the scoring information of preset English spoken audio, and the feedback information of preset English spoken audio into a training sample. The second acquisition module 203 is used to form a training set from different training samples, train the deep learning model using the training set, and obtain the loss of the deep learning model on the training set through the loss function. The saving module 204 is used to save the trained deep learning model when the loss of the deep learning model on the training set meets the preset conditions. The trained deep learning model includes an audio encoder, a text encoder, and a prediction module. The scoring module 205 is used to determine the fusion feature vector of the current English spoken audio based on the audio encoder and the text encoder. The fusion feature vector of the current English spoken audio is input into the prediction module, which generates the scoring information and feedback information of the current English spoken audio.

[0068] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0069] The beneficial effects of the embodiments of this application are as follows: Firstly, based on the audio encoder and text encoder, the fusion feature vector of the current English spoken audio is determined. The fusion feature vector of the current English spoken audio is input into the prediction module, which generates the scoring information and feedback information of the current English spoken audio. Since no manual operation is required, the time for obtaining the scoring information and feedback information of the current English spoken audio is reduced, which is conducive to improving the efficiency of obtaining the scoring information and feedback information of the current English spoken audio. Secondly, since the scoring information and feedback information of the current English speaking audio are automatically generated, they are not affected by human intervention, which helps to improve the reliability of the scoring information and feedback information of the current English speaking audio.

[0070] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0071] like Figure 3 As shown, Figure 3 The electronic device includes: at least one processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the at least one processor 30, wherein the processor 30 executes the computer program 32 to implement the steps in any of the above method embodiments.

[0072] The electronic device may include, but is not limited to, processor 30 and memory 31. Those skilled in the art will understand that... Figure 3 This is merely an example of an electronic device and does not constitute a limitation on electronic devices. It may include more or fewer components than shown in the illustration, or combinations of certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0073] The processor 30 is used to run a computer program 32 stored in the memory 31, and performs the following steps when executing the computer program 32: Obtain a preset English spoken audio, perform speech recognition on the preset English spoken audio, and generate a transcribed text of the preset English spoken audio; A training sample is composed of preset English speaking audio, preset English speaking audio transcribed text, preset English speaking audio scoring information, and preset English speaking audio feedback information. Different training samples are combined into a training set, and the deep learning model is trained using the training set. The loss of the deep learning model on the training set is obtained through the loss function. When the loss of the deep learning model on the training set meets the preset conditions, the trained deep learning model is saved. The trained deep learning model includes an audio encoder, a text encoder, and a prediction module. Based on the audio encoder and text encoder, the fusion feature vector of the current English spoken audio is determined. The fusion feature vector of the current English spoken audio is input into the prediction module, which generates the scoring information and feedback information of the current English spoken audio.

[0074] The processor 30 may be a central processing unit, or it may be other general-purpose processors, digital signal processors, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0075] In some embodiments, the memory 31 may be an internal storage unit of the electronic device, such as a hard disk or memory of the electronic device.

[0076] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0077] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for scoring spoken English based on audio and text, characterized in that, The English speaking assessment method, applied to electronic devices, includes: Obtain a preset English spoken audio, perform speech recognition on the preset English spoken audio, and generate a transcribed text of the preset English spoken audio; A training sample is composed of preset English speaking audio, preset English speaking audio transcribed text, preset English speaking audio scoring information, and preset English speaking audio feedback information. Different training samples are combined into a training set, and the deep learning model is trained using the training set. The loss of the deep learning model on the training set is obtained through the loss function. When the loss of the deep learning model on the training set meets the preset conditions, the trained deep learning model is saved. The trained deep learning model includes an audio encoder, a text encoder, and a prediction module. Based on the audio encoder and text encoder, the fusion feature vector of the current English spoken audio is determined. The fusion feature vector of the current English spoken audio is input into the prediction module, which generates the scoring information and feedback information of the current English spoken audio.

2. The English oral assessment method according to claim 1, characterized in that, A training sample is composed of preset English speaking audio, the transcribed text of the preset English speaking audio, the scoring information of the preset English speaking audio, and the feedback information of the preset English speaking audio, including: Obtain the labeled data corresponding to the preset English spoken audio. From the labeled data corresponding to the preset English spoken audio, obtain the scoring information and feedback information of the preset English spoken audio. Combine the preset English spoken audio, the transcribed text of the preset English spoken audio, the scoring information of the preset English spoken audio, and the feedback information of the preset English spoken audio into a training sample.

3. The English oral assessment method according to claim 1, characterized in that, A training set is formed by combining different training samples. The deep learning model is trained using the training set. The loss of the deep learning model on the training set is obtained through a loss function, including: Different training samples are combined into a training set, and the deep learning model is trained using the training set. During the iterative training of the deep learning model, the loss of the deep learning model on the training set is obtained through a loss function, which includes one of the following: mean squared error loss function, Huber loss function, and ranking loss function.

4. The English oral assessment method according to claim 1, characterized in that, When the loss of the deep learning model on the training set meets the preset conditions, the trained deep learning model is saved. The trained deep learning model includes an audio encoder, a text encoder, and a prediction module, including: When the loss of the deep learning model on the training set is less than a preset value, a stop command is generated. By executing the stop command, the training of the deep learning model is stopped, and the trained deep learning model is saved. The trained deep learning model includes an audio encoder, a text encoder, and a prediction module.

5. The English oral assessment method according to claim 1, characterized in that, Based on the audio encoder and text encoder, a fusion feature vector of the current English spoken audio is determined. This fusion feature vector is then input into the prediction module, which generates scoring information and feedback information for the current English spoken audio, including: The current English spoken audio is obtained, and an audio encoder is used to extract features from the current English spoken audio to obtain the acoustic feature vector, fluency feature vector, and pronunciation feature vector of the current English spoken audio. The current English spoken audio is converted into speech recognition and transcribed into a text. A text encoder is then used to extract features from the transcribed text, generating lexical feature vectors, grammatical feature vectors, and content feature vectors for the current English spoken audio. Using a preset fusion method, the acoustic feature vector, fluency feature vector, pronunciation feature vector, vocabulary feature vector, grammar feature vector, and content feature vector of the current English spoken audio are fused to generate a fused feature vector of the current English spoken audio. The fused feature vector of the current English spoken audio is input into the prediction module, which generates the scoring information and feedback information of the current English spoken audio.

6. The English oral assessment method according to claim 1, characterized in that, After determining the concatenated feature vector of the current English spoken audio through an audio encoder, determining the feature vector of the transcribed text of the current English spoken audio through a text encoder, fusing the concatenated feature vector of the current English spoken audio and the feature vector of the transcribed text of the current English spoken audio to generate a fused feature vector of the current English spoken audio, inputting the fused feature vector of the current English spoken audio into a prediction module, and generating the scoring information and feedback information of the current English spoken audio through the prediction module, the English spoken audio scoring method includes: Write the scoring information and feedback information of the current English speaking audio into a preset scoring template to generate a scoring report for the current English speaking audio.

7. The English oral assessment method according to claim 1, characterized in that, The preset English speaking audio is already evaluated English speaking audio; The current English speaking audio is an English speaking audio to be evaluated; The scoring information for current English speaking audio includes the overall score, fluency score, and pronunciation score. The feedback information for the current English speaking audio includes the reasons for the deduction of points and suggestions for improvement. The scoring information for the preset English speaking audio includes the overall score, the fluency score, and the pronunciation score. Feedback information for the preset English speaking audio includes the reasons for the deduction of points in the preset English speaking audio and suggestions for improvement in the preset English speaking audio; The preset fusion method adopts one of the following: vector dimension splicing, weighted summation, gating dynamic filtering, attention weight allocation, or cross-attention interaction. The prediction module includes at least one of a multilayer perceptron and a regression head.

8. An English oral proficiency scoring device based on audio and text, characterized in that, Applied to electronic devices, including: The first acquisition module is used to acquire preset English spoken audio, perform speech recognition conversion on the preset English spoken audio, and generate a transcribed text of the preset English spoken audio. The component module is used to combine preset English spoken audio, the transcribed text of preset English spoken audio, the scoring information of preset English spoken audio, and the feedback information of preset English spoken audio into a training sample. The second acquisition module is used to form a training set from different training samples, use the training set to train the deep learning model, and obtain the loss of the deep learning model on the training set through the loss function. The save module is used to save the trained deep learning model when the loss of the deep learning model on the training set meets the preset conditions. The trained deep learning model includes an audio encoder, a text encoder, and a prediction module. The scoring module is used to determine the fusion feature vector of the current English spoken audio based on the audio encoder and the text encoder. The fusion feature vector of the current English spoken audio is input into the prediction module, which generates the scoring information and feedback information of the current English spoken audio.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the English oral assessment method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the English oral assessment method as described in any one of claims 1 to 7.

Images