Spoken language evaluation system and method based on multi-dimensional factors

Abstract

The invention discloses a spoken language evaluation system and method based on multi-dimensional factors, and relates to the technical field of data processing, and the method comprises the steps: obtaining an audio text, a phoneme list and a phoneme time boundary list corresponding to voice data according to a pre-trained text phoneme recognition model; calculating a fluency score according to the phoneme list and the phoneme time boundary list; obtaining a semantic score of the audio text through a pre-trained semantic model; obtaining a grammar score of the audio text through a pre-trained grammar model; matching the audio text and the answer text according to a text matching method based on an editing distance to obtain a text matching score; the pronunciation score of the audio data is finally obtained through the phoneme confidence of the phoneme list; dynamically adjusting the weight of the score of each dimension according to the length of the audio text, and calculating a final spoken language evaluation score; through multi-dimensional evaluation, the spoken language ability is comprehensively evaluated according to dynamic weight adjustment of the scene, and comprehensiveness, fairness, accuracy and high efficiency of spoken language evaluation are achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a spoken language evaluation system and method based on multi-dimensional factors. BACKGROUND

[0002] With the development of education informatization, spoken language evaluation plays an increasingly important role in English learning, teaching and testing, etc. In particular, with the popularization of English listening and speaking tests, the development of spoken language evaluation technology has been accelerated.

[0003] Traditional spoken language evaluation mainly relies on manual scoring, which is not only low in efficiency but also subjective. Existing spoken language evaluation techniques also mostly rely on analysis of speech features, combined with text matching for scoring, which is insufficient in terms of fluency, semantics and grammar, and consideration of different testing scenarios, especially open-ended questions, which are even more difficult to meet the actual evaluation needs.

[0004] Chinese patent application publication No. CN108154735A discloses: after outputting a test question, receiving a speech signal input by a testee; performing speech recognition on the speech signal to obtain corresponding text information; performing text matching on the text information and an answer in an answer set corresponding to the test question, wherein the answer set includes a standard answer to the test question and a derivative answer expanded from the standard answer; obtaining an evaluation result according to at least the text matching result; and outputting the evaluation result. Thus, the semantic and logical nature of the spoken language text can be effectively evaluated, the comprehensiveness and accuracy of the evaluation are improved, and the evaluation result feedback speed is fast. However, it does not consider the cooperative consideration of speech speed and pauses on spoken language fluency, nor does it consider that the evaluation focus is different under different spoken language pronunciation durations.

[0005] Therefore, there is an urgent need for an evaluation method based on multi-dimensional factors such as fluency, semantics, grammar, text matching and spoken language duration, which can more comprehensively and accurately guide students' spoken language. SUMMARY

[0006] In order to overcome the deficiencies in the background art, the present application provides a spoken language evaluation system and method based on multi-dimensional factors.

[0007] To achieve the above-mentioned application purposes, the present application adopts the following technical solutions:

[0008] In a first aspect, the present application provides a spoken language evaluation system based on multi-dimensional factors, comprising the following modules:

[0009] The speech collection module is configured to collect speech data of the user's answers to the questions;

[0010] The scenarios of the questions include oral composition and situational spoken language.

[0011] a speech recognition module configured to extract Fbank features of the speech data, and input the Fbank features into a pre-trained text phoneme recognition model to obtain audio text, a phoneme list and a phoneme time boundary list corresponding to the speech data;

[0012] a fluency analysis module configured to calculate the number of phonemes per minute, the duration of pauses and the number of pauses according to the phoneme list and the phoneme time boundary list, and calculate a fluency score SP according to the number of phonemes per minute, the duration of pauses and the number of pauses;

[0013] a semantic analysis module configured to obtain a semantic score SP2 of the audio text by using a pre-trained semantic model, wherein the pre-trained semantic model is fine-tuned based on a pre-trained model SentenceTransformer model;

[0014] a syntax analysis module configured to obtain a syntax score SP3 of the audio text by using a pre-trained syntax model, wherein the pre-trained syntax model is fine-tuned based on a RoBERTa model;

[0015] a text matching module configured to match the audio text and the answer text according to a text matching method based on an edit distance to obtain a text matching score SP4;

[0016] a pronunciation analysis module configured to calculate a phoneme confidence average value of each word contained in the phoneme list as a word pronunciation score, and then calculate an average value of all word pronunciation scores multiplied by 100 as a final pronunciation score SP5 of the audio data;

[0017] a comprehensive evaluation module configured to dynamically adjust the weights of the fluency score SP, the syntax score SP3, the pronunciation score SP5 and the completeness score SP6 according to the length of the audio text, and calculate a final oral test score Score; the completeness score SP6 is the larger one of the semantic score SP2 and the text matching score SP4.

[0018] Specifically, the text phoneme recognition model adopts a Transformer-CTC architecture, and the specific structure is as follows: the input is the FBank feature of the audio, the FBank feature is down-sampled and local features are extracted through 2 convolutional layers, then the local features are projected to the input dimension of the Transformer through a linear layer, then a plurality of Transformer encoder layers are passed, each Transformer encoder layer includes a multi-head self-attention mechanism and a feedforward neural network, which are used to capture long-time dependencies and output high-level acoustic features, and finally a linear layer is used to map the high-level acoustic features to the phoneme label space, outputting the probability of each phoneme label of each time frame, and finally the CTC decoding outputs the audio text, the phoneme list and the phoneme time boundary list composed of the time boundary of each phoneme; wherein each element of the phoneme list includes a specific phoneme and its corresponding confidence.

[0019] Specifically, the fluency score SP is as follows:

[0020] SP=Sx0.85-Px0.15;

[0021] Wherein, SP is the final fluency score; S is the basic fluency score; P is the pause penalty score;

[0022] The basic fluency score S is calculated as follows:

[0023] S=100x[1-|(PPM-PPM’) / PPM’|^a]

[0024] Wherein, PPM' is the optimal number of phonemes per minute; a is a speed-sensitive adjustment coefficient, and the value of a is different according to the different scenes of the question; the value range of S is [0, 100], and the unit is point;

[0025] The calculation formula of the pause penalty score P is as follows:

[0026] P=β1xN2+β2xN3+β3xT p

[0027] Wherein, N2 is the number of 2-second pauses; N3 is the number of 3-second pauses; T p is the total pause time; β1 is the 2-second pause penalty coefficient; β2 is the 3-second pause penalty coefficient; and β3 is the total pause time penalty coefficient.

[0028] Specifically, in the oral composition scene, a is taken in the range of [1.1, 1.3]; and in the situational oral scene, a is taken in the range of [1.5, 1.7].

[0029] Specifically, the input of the pre-trained semantic model is an audio text and an answer text. The semantic model is used to perform word segmentation and embedding on the two texts respectively, and finally obtain two sentence vector representations. Then, the cosine similarity of the two sentence vectors is calculated, and the cosine similarity is multiplied by 100 to obtain a semantic score SP2 of the audio text. The answer text is obtained in advance according to the question.

[0030] Specifically, the input of the pre-trained semantic model is an audio text and an answer text. The semantic model is used to perform word segmentation and embedding on the two texts respectively, and finally obtain two sentence vector representations. Then, the cosine similarity of the two sentence vectors is calculated, and the cosine similarity is multiplied by 100 to obtain a semantic score SP2 of the audio text. The answer text is obtained in advance according to the question.

[0031] Specifically, the text matching method based on the edit distance is as follows:

[0032]

[0033] wherein, SP4 is a text matching score; wd1 i represents the i-th word in the audio text; wd2 i represents the i-th word in the answer text; N is the total number of matched word pairs; ES(wd1 i , wd2 i ) is an edit distance similarity function, which is used to represent the edit distance similarity between wd1 i and wd2 i , and the value range is [0-1], 0 represents complete difference, and 1 represents complete same; w i is the high-frequency word weight of wd2 i .

[0034] Specifically, the edit distance similarity function is as follows:

[0035]

[0036] wherein, EditDistance() is an edit distance algorithm; len() is a length function; max() is a maximum value function.

[0037] Specifically, the calculation formula of the final spoken language evaluation score Score is as follows:

[0038] Score=a1(L)×SP+a2(L)×SP3+a3(L)×SP5+a4(L)×SP6

[0039] Wherein, a1(L) is the fluency score dynamic weight; a2(L) is the syntax score dynamic weight; a3(L) is the pronunciation score dynamic weight; a4(L) is the completeness score dynamic weight; L is the audio text length, that is, the number of words in the audio text L.

[0040] Specifically, the dynamic weight distribution formula is realized based on the dynamic weight distribution formula by dynamically adjusting the weight of each score according to the length of the audio text through the fluency score SP, the syntax score SP3, the pronunciation score SP5 and the completeness score SP6.

[0041]

[0042] In a second aspect, the present application provides a spoken language evaluation method based on multi-dimensional factors, based on the system for evaluating spoken language based on multi-dimensional factors in the first aspect, comprising the following steps:

[0043] S1, collecting voice data of the user's answers to the questions; the scene of the questions includes oral composition and situational spoken language;

[0044] S2, extracting the Fbank features of the voice data, and inputting the Fbank features into a pre-trained text phoneme recognition model to obtain the audio text, phoneme list and phoneme time boundary list corresponding to the voice data;

[0045] S3, calculating the number of phonemes per minute, the duration of pauses and the number of pauses according to the phoneme list and the phoneme time boundary list, and calculating the fluency score SP according to the number of phonemes per minute, the duration of pauses and the number of pauses;

[0046] S4, obtaining the semantic score SP2 of the audio text through a pre-trained semantic model; the pre-trained semantic model is obtained by fine-tuning based on the pre-trained model SentenceTransformer model;

[0047] S5, obtaining the syntax score SP3 of the audio text through a pre-trained syntax model; the pre-trained syntax model is obtained by fine-tuning based on the RoBERTa model;

[0048] S6, matching the audio text and the answer text according to the text matching method based on the edit distance to obtain the text matching score SP4;

[0049] S7, calculate the phoneme confidence average value contained in each word by the phoneme confidence of the phoneme list as the word pronunciation score, and then calculate the average value of all word pronunciation scores multiplied by 100 as the final pronunciation score SP5 of the audio data;

[0050] S8, dynamically adjust the weight of each score according to the length of the audio text by the fluency score SP, the grammar score SP3, the pronunciation score SP5 and the completeness score SP6, and calculate the final oral test score Score; the completeness score SP6 is the larger value of the semantic score SP2 and the text matching score SP4;

[0051] Specifically, the method further comprises;

[0052] S9, display the answer text and the dimension scores on the application end; the dimension scores include the fluency score SP, the semantic score, the grammar score SP3, the pronunciation score SP5, the completeness score SP6 and the final oral test score Score.

[0053] In a third aspect, the present application also provides an electronic device, comprising a processor, a memory, a communication interface, and one or more programs stored in the memory and configured to be executed by the processor, the program comprising functions for executing the system described in the first aspect.

[0054] The present application provides a multi-dimensional factor-based oral test system and method, comprising: collecting voice data of a user answering questions; obtaining audio text, phoneme list and phoneme time boundary list corresponding to the voice data according to a pre-trained text phoneme recognition model; calculating the number of phonemes per minute, the duration of pause and the number of pauses according to the phoneme list and the phoneme time boundary list, and calculating the fluency score; obtaining the semantic score of the audio text by a pre-trained semantic model; obtaining the grammar score of the audio text by a pre-trained grammar model; matching the audio text and the answer text according to an editing distance-based text matching method to obtain the text matching score; calculating the pronunciation score of the final audio data by the phoneme confidence of the phoneme list; and a comprehensive evaluation module for dynamically adjusting the weight of each dimension score according to the length of the audio text and calculating the final oral test score; the method of the present embodiment comprehensively evaluates oral ability from five dimensions of fluency, semantics, grammar, text matching and pronunciation, avoiding the limitation of a single indicator;

[0055] Further, the fluency analysis module of the present application avoids the limitations of a single indicator by using multi-dimensional quantitative evaluation, dynamic parameter configuration, reasonable weight distribution, and precise punishment mechanism through phoneme-level analysis, ensuring the coordinated consideration of speech speed and pauses, improving the fairness and practicality of the evaluation, conforming to authoritative language ability standards, enhancing the authority and reliability of the evaluation results, and realizing scientific, scenario-based, and personalized evaluation of oral fluency, suitable for different language levels and scenarios.

[0056] Further, the comprehensive evaluation module of the present application adopts different scoring weights for different audio text length answers, ensuring that the evaluation results are closer to actual oral language ability.

[0057] Further, the present application captures the deep intention of the answer through pre-training semantic models, avoids misjudgment due to different expressions, and quickly filters completely consistent answers based on an edit distance matching method, while combining semantic and text matching to judge completeness, improving the accuracy and robustness of the evaluation.

[0058] The present application realizes the comprehensiveness, fairness, accuracy, and efficiency of oral language evaluation through multi-dimensional evaluation, dynamic weight adjustment, semantic and text matching combination, and phoneme-level analysis, and is particularly suitable for education, examination, language learning, and other scenarios, and can more scientifically reflect the oral language ability of students. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0060] Figure 1 is a schematic diagram of an oral language evaluation system based on multi-dimensional factors according to an embodiment of the present application;

[0061] Figure 2 is a schematic diagram of an oral language evaluation method based on multi-dimensional factors according to an embodiment of the present application;

[0062] Figure 3 is a schematic diagram of an oral language evaluation device based on multi-dimensional factors according to an embodiment of the present application. DETAILED DESCRIPTION

[0063] The application can be explained in detail by the following examples, and the purpose of the application is to protect all technical improvements within the scope of the application. In the description of the application, it should be understood that the orientations or positional relationships indicated by terms such as “up”, “down”, “front”, “back”, “left”, “right” and the like are only corresponding to the drawings of the present application, and are for the convenience of describing the application, and do not indicate or imply that the devices or elements referred to must have a particular orientation.

[0064] Embodiment one

[0065] Reference Figure 1 The embodiment provides a spoken language evaluation system based on multi-dimensional factors, comprising the following modules:

[0066] A voice collection module is configured to collect voice data of a user answering a question; the question scenario includes oral composition and situational spoken language.

[0067] The voice data of the user is collected through the recording function of an application end; the application end is an App program including a mobile phone or a tablet, a computer desktop program, and a web browser; the mobile phone, the tablet and the computer all have a microphone recording function, and the application end calls the microphone to collect the audio, which is prior art and will not be described here.

[0068] The audio format of the voice data supports single-channel, 16KHz, WAV or MP3 format, and the audio duration unit of the voice data is second;

[0069] A voice recognition module is configured to extract Fbank features of the voice data, and input the Fbank features into a pre-trained text phoneme recognition model to obtain audio text, a phoneme list and a phoneme time boundary list corresponding to the voice data.

[0070] The Fbank features of the voice data include the following steps: pre-emphasis, framing, windowing, Fourier transform, spectral line energy calculation, Mel filtering, and logarithm taking are performed on the voice data to obtain the Fbank features.

[0071] Fbank (Filter Bank) is a feature obtained by taking logarithm after Mel filtering, which is commonly used in voice signal processing, and has a shape of [T, D], wherein T is the number of time frames, and D is the feature dimension.

[0072] Specifically, the Fbank features of the voice data are extracted by the librosa library.

[0073] Further, the audio data is loaded by the librosa.load() function, the Mel-spectrogram energy without taking logarithm is obtained by the librosa.feature.melspectrogram() function, and then the Mel-spectrogram energy without taking logarithm is converted into logarithmic scale to obtain the Fbank features by the librosa.power_to_db() function.

[0074] Librosa is a Python library for audio and music analysis, focusing on Music Information Retrieval (MIR) tasks. It provides a powerful set of tools for processing audio signals, extracting features, visualizing, and analyzing music content.

[0075] The text phoneme recognition model adopts the Transformer-CTC architecture, and the specific structure is as follows: the input is the FBank features of the audio (shape [time frame number, feature dimension]), the FBank features are reduced in sampling and local features are extracted by 2 convolutional layers, then the local features are projected to the input dimension of the Transformer through a linear layer, then multiple Transformer encoder layers are used, each Transformer encoder layer contains a multi-head self-attention mechanism and a feedforward neural network, which is used to capture long-time dependencies and output high-level acoustic features, and finally a linear layer is used to map the high-level acoustic features to the phoneme label space, outputting the probability of each phoneme label for each time frame, and finally the CTC decoding outputs the audio text, phoneme list, and phoneme time boundary list composed of the time boundary of each phoneme; wherein each element of the phoneme list includes a specific phoneme and its corresponding confidence.

[0076] The text phoneme recognition model can be trained in advance by pre-collected audio data, and how to train is prior art, which will not be repeated here.

[0077] Phoneme is the smallest unit of speech divided according to the natural properties of speech, analyzed according to the pronunciation actions in syllables, and one action constitutes a phoneme. Phonemes are divided into vowels and consonants. Phonemes are generally marked with the International Phonetic Alphabet (IPA), and the English International Phonetic Alphabet has 48 phonemes, including 20 vowel phonemes and 28 consonant phonemes.

[0078] It is worth noting that the phoneme label of the embodiment also contains a silence label sil.

[0079] For example, the embodiment provides an example to illustrate the above scheme:

[0080] Record the audio data of students' answers to the question "what's your age" through a mobile app. Assume that the audio duration is 4.2 seconds. Use librosa to extract the Fbank feature fb of the audio data audio. The specific code is as follows:

[0081]

[0082]

[0083] The input of fb.T to the text phoneme recognition model is as follows:

[0084] {"text":"what's your age",

[0085] "phonemeList":[{"phoneme":"sil","confidence":0.9},{"phoneme":"w","confidence":0.87},{"phoneme":"ah","confidence":0.9 1},{"phoneme":"t","confidence":0.85},{"phoneme":"s","confidence":0.93},{"phoneme":"sil","confidence":0.9},{"phoneme": "y","confidence":0.88},{"phoneme":"ao","confidence":0.90},{"phoneme":"r","confidence":0.92},{"phoneme":"sil","confid ence":0.9},{"phoneme":"ey","confidence":0.89},{"phoneme":"jh","confidence":0.94},{"phoneme":"sil","confidence":0.9}],

[0086] "phonemeTs":[[0,180],[180,300],[300,450],[450,600],[600,720],[720,2820],[2820, 2970],[2970,3120],[3120,3320],[3320,3620],[3620,3820],[3820,4070],[4070,4200]]}

[0087] Wherein, text represents audio text, phonemeList represents a phoneme list, each element includes a phoneme and a confidence corresponding to the phoneme, the confidence is in the range of [0, 1], and is a floating point type; sil is a silence mark, indicating a pause, phonemeTs represents a phoneme time boundary list, the number of which corresponds to phonemeList one by one, each element represents the time range of the phoneme with the same index, start is the start time, and end is the end time, both in ms;

[0088] The design of the boundary of the time range of the phoneme in the phoneme time boundary list is the same as that of the adjacent phonemes (i.e., the end time of the previous phoneme is equal to the start time of the next phoneme), which indicates a seamless transition between phonemes, conforms to the smooth conversion of acoustic characteristics, simplifies the alignment algorithm, balances acoustic authenticity and computational feasibility, and provides key information such as the number of phonemes per minute, the duration of the pause and the number of pauses for subsequent fluency calculation.

[0089] The fluency analysis module is configured to calculate the number of phonemes per minute, the duration of the pause and the number of pauses according to the phoneme list and the phoneme time boundary list, and calculate the fluency score SP according to the number of phonemes per minute, the duration of the pause and the number of pauses.

[0090] Fluency is mainly calculated according to the speed of pronunciation, i.e., the number of phonemes per minute, and the duration of the pause and the number of pauses are comprehensively calculated. Specifically, the total number of phonemes of the final recognized text of the audio data (excluding the silence mark sil) is divided by the total duration of the audio, so that the basic score of pronunciation is obtained, and the basic fluency score is obtained according to the punishment of the basic score according to the speech rate; the lower the speech rate, the less skilled the learner, and the lower the corresponding score; if the speech rate is too fast, the pronunciation will be blurred, which will also affect the score, so the basic fluency score is obtained according to the punishment of the basic score according to the speech rate.

[0091] First, the number of phonemes per minute PPM = PN num / (T total / 60) is calculated, wherein T total is the audio duration, in seconds, and PN num is the number of phonemes contained in the audio, which is obtained according to the phoneme list phonemeList obtained by the speech recognition module and removing the silence mark.

[0092] The basic fluency score S is calculated according to the following formula:

[0093] S = 100 x [1 - |(PPM-PPM') / PPM'|] a]

[0094] Wherein, PPM' is the optimal phoneme quantity per minute; a is a speech rate sensitive adjustment coefficient, which punishes the base score according to the speech rate, and the greater the value, the more obvious the punishment for deviating from the standard speech rate, and the value of the speech rate sensitive adjustment coefficient a is also different according to the different scenes of the questions;

[0095] The value of the speech rate sensitive adjustment coefficient a is also different according to the different scenes of the questions, and in the embodiment, the scenes of the questions include oral composition and situational oral English; in the oral composition scene, a lower a value helps to maintain the stability of the speech rate and ensure the effective transmission of information, and a is taken in [1.1, 1.3]; while in the situational oral English scene, a higher a value encourages the speaker to adjust the speech rate according to the situation, and the increase of a may reduce the punishment for slight speeding, so as to adapt to different communication needs, and a is taken in [1.5, 1.7].

[0096] It can be understood that the scenes of the questions described in the text are not limited by the present application, and the scenes of the questions can be expanded according to the actual situation, and the value of a can be adjusted according to the different expanded scenes.

[0097] According to the definition of speech rate in the Chinese English Proficiency Scale, the speech rate is as follows:

[0098] Fast speech rate: about 140-180 words per minute;

[0099] Normal speech rate: about 100-140 words per minute;

[0100] Slow speech rate: about 80-100 words per minute;

[0101] Very slow speech rate: about 60-80 words per minute.

[0102] Each word in English contains about 4-5 phonemes on average, and calculated at 100-180 words per minute, it is converted into 400-900 phonemes per minute; therefore, a general speech rate of 400-900 phonemes per minute is relatively fluent. Therefore, the embodiment takes [600-700] phonemes per minute as the value range of the optimal phoneme quantity per minute;

[0103] Preferably, the embodiment takes 650 phonemes per minute as the optimal phoneme quantity per minute;

[0104] It is worth noting that the value range of the optimal phoneme quantity per minute and the optimal phoneme quantity per minute are not specifically limited in the embodiment, and they can be adjusted according to different examination levels.

[0105] For example, in the graded Examination in Spoken English (GESE) jointly launched by Beijing Education Examination Authority and Trinity College London, the test is divided into 4 sections and 12 levels, namely: Preparatory 1-3, Elementary 4-6, Intermediate 7-9, Advanced 10-12.

[0106] For example, at the preparatory level, considering the reason for beginners, the learner's speed may be slower, so the value range of the optimal number of phonemes per minute can be appropriately adjusted to [550-650] phonemes per minute as the value range of the optimal number of phonemes per minute; the optimal number of phonemes per minute is 600 phonemes per minute.

[0107] At the elementary or intermediate level, to fluently and skillfully answer the questions, take [600-700] phonemes per minute as the value range of the optimal number of phonemes per minute; the optimal number of phonemes per minute is 650 phonemes per minute.

[0108] At the advanced level, to freely use various sentence structures and vocabulary, to fluently and skillfully answer the questions, the value range of the optimal number of phonemes per minute needs to be appropriately adjusted, so take [650-800] phonemes per minute as the value range of the optimal number of phonemes per minute; the optimal number of phonemes per minute is 750 phonemes per minute.

[0109] Calculate the duration of the pause between words and words through the phoneme list and the phoneme time boundary list, and then count the number of pauses exceeding 2 seconds, 3 seconds and above. The more and longer the pauses, the more points will be deducted. The final fluency score is obtained by penalizing the basic score for pauses;

[0110] Pause duration T p Determined according to the time boundary of the phoneme time boundary list corresponding to the silence mark sil in the phoneme list phonemeList;

[0111] The final fluency score formula is as follows:

[0112] SP = S x 0.85 - P x 0.15;

[0113] Where SP is the final fluency score; S is the basic fluency score; P is the pause penalty; 0.85 in the final fluency score formula is the speed weight, reflecting the dominant influence of speed on fluency, and 0.15 is the pause weight, which is an important correction factor to avoid extreme pauses affecting the evaluation; The speed weight + the pause weight = 1, and the speed weight and the pause weight can also be adjusted according to actual conditions.

[0114] The calculation formula of the pause penalty P is as follows:

[0115] P = β1 x N2 + β2 x N3 + β3 x T p

[0116] Wherein, N2 is the number of 2-second pauses, the initial value of N2 is 0, when the time boundary corresponding to the silence mark sil is greater than or equal to 2 seconds and less than 3 seconds, the value of N2 is added by 1; N3 is the number of 3-second pauses, the initial value of N3 is 0, when the time boundary corresponding to the silence mark sil is greater than or equal to 3 seconds, the value of N3 is added by 1; T p is the total pause duration, which is determined according to the time boundary list of the phoneme time boundary list corresponding to the silence mark in the phoneme list; β1 is the 2-second pause penalty coefficient, which is 5 points / second in the embodiment; β2 is the 3-second pause penalty coefficient, which is 9 points / second in the embodiment; β3 is the total pause duration penalty coefficient, which is 0.5 points / second in the embodiment;

[0117] The final fluency score formula realizes multi-dimensional quantitative evaluation of oral fluency through the nonlinear speech speed score and the composite pause penalty mechanism, which not only guarantees the dominant position of the core speech speed index, but also avoids the one-sidedness of single index evaluation through multi-dimensional penalty. The pause penalty score adopts a double penalty mechanism, which not only penalizes high-frequency long pauses, but also penalizes the total pause duration; when evaluating language fluency, humans will pay attention to the frequency and duration of pauses. For example, high-frequency short pauses (such as 1 second pause after each sentence) will destroy coherence and appear to be rigid in expression; low-frequency long pauses (such as 3-second single pause) may cause the listener's attention to be distracted, and even be misunderstood as content forgetting. The double penalty mechanism simultaneously restricts N2, N3 (long pause frequency) and T p (total pause duration), which more comprehensively covers two typical modes of destroying fluency.

[0118] For example, the time boundaries corresponding to "sil" in phonemeList are [0, 180], [720, 2820], [3320, 3620], [4070, 4200], then the pause duration T p = 180 + 2100 + 300 + 170 = 2750 ms = 2.75 s; wherein the silence duration corresponding to [720, 2820] is 2.1 seconds, which is obviously that the examinee hesitated, so the long waiting time of the silence duration is penalized;

[0119] The semantic analysis module is configured to obtain a semantic score SP2 of the audio text by using a pre-trained semantic model; the pre-trained semantic model takes the audio text and an answer text as input, performs word segmentation and embedding on the two texts by using the semantic model, finally obtains two sentence vector representations, and then calculates the cosine similarity of the two sentence vectors, multiplies the cosine similarity by 100 to obtain the semantic score SP2 of the audio text; the answer text is obtained in advance according to the question;

[0120] The pre-trained semantic model is fine-tuned on the basis of a pre-trained model SentenceTransformer model, and is fine-tuned and trained by collecting a large number of standard similarity sentence pairs. The principle is to perform word segmentation and embedding on the text, finally obtain a vector representation, and then calculate the cosine similarity of the two sentence vectors. The cosine similarity is valued between 0 and 1. If the value is closer to 1, it means that the semantics of the two sentences are closer.

[0121] The Sentence Transformer model is a deep learning model based on the Transformer architecture, which is specially used to convert sentences or text passages into high-dimensional vector representations, and is widely used in semantic search, text similarity calculation, information retrieval and other scenarios. Its core value lies in capturing the deep semantic features of text through vectorization expression, so that computers can more accurately understand natural language.

[0122] The syntax analysis module is configured to obtain a syntax score SP3 of the audio text by using a pre-trained syntax model.

[0123] The RoBERTa model (Robustly Optimized BERT Pretraining Approach) was released by Meta AI in July 2019, and is based on the BERT model. The model learns and predicts the masked text part, builds its language model on the BERT language masking strategy, and uses a larger batch size and learning rate for training. Compared with BERT, this allows RoBERTa to improve the masked language modeling objective and produce better downstream task performance. RoBERTa is a pre-trained model that performs well on GLUE or General Language Understanding Evaluation.

[0124] The pre-trained syntax model is fine-tuned based on the RoBERTa model and fine-tuned on a large number of syntax correction data sets. The model outputs information on whether each word is correct or needs to be operated (such as insertion, replacement, deletion), and then uses the proportion of correct words as the syntax score.

[0125] Specifically, the input of the pre-trained grammar model is a word sequence [wd1, wd2,..., wdi] corresponding to the audio text, where i is the number of words in the audio text, and the output of the pre-trained grammar model is a predicted label sequence [l1, l2,..., li] equal in length to the word sequence, each label li belongs to a set of predefined operation labels, such as {CORRECT, DELETE, REPLACE, INSERT_AFTER,...}, indicating the model's grammar judgment and suggested operation for the word wdi;

[0126] The number of words marked as CORRECT in the predicted label sequence is counted to obtain N2, and N2 is divided by the length of the word sequence corresponding to the audio text and multiplied by 100 to obtain the grammar score of the audio text;

[0127] The text matching module is configured to match the audio text and the answer text according to an edit distance-based text matching method to obtain a text matching score SP4; the edit distance-based text matching method is as follows:

[0128]

[0129] where SP4 is the text matching score; wd1 i represents the i-th word in the audio text; wd2 i represents the i-th word in the answer text; N is the total number of matched word pairs; ES(wd1 i , wd2 i ) is an edit distance similarity function for representing the edit distance similarity between wd1 i and wd2 i , with a value range of [0-1], 0 indicating complete difference and 1 indicating complete identity; w i is the high-frequency word weight of wd2 i ;

[0130] The edit distance-based text matching method includes the following steps:

[0131] 1) Tokenize the audio text and the answer text to obtain corresponding token lists, and perform global alignment of the token lists by using the Nederman-Wunsch algorithm;

[0132] The Needleman-Wunsch Algorithm is an algorithm that matches protein sequences or DNA sequences based on knowledge of bioinformatics. It is one of the earliest examples of applying dynamic programming to the comparison of biological sequences. The algorithm was invented by two scientists, Saul B. Needleman and Christian D. Wunsch, in 1970. This algorithm efficiently solves the process of how to break down a huge mathematical problem into a series of small problems, and reconstruct the solution of the big problem from the solutions of a series of small problems. This algorithm is also known as the optimal matching algorithm and the global sequence comparison method. The Needleman-Wunsch algorithm is still widely used in the optimization of global sequence comparison.

[0133] For example, the audio text word segmentation list wd1: ["the", "cat", "sits"];

[0134] The answer text word segmentation list wd2: ["a", "cat", "sat"];

[0135] The aligned word segmentation list: ["the / a", "cat / cat", "sits / sat"]

[0136] 2) Calculate the edit distance similarity of each pair of words by the edit distance algorithm EditDistance, the calculation formula is as follows:

[0137]

[0138] Where EditDistance() is the edit distance algorithm; len() is the length function; max() is the maximum value function; ES(wd1 i , wd2 i ) is the edit distance similarity function, used to represent the edit distance similarity between the i-th word wd1 i in the audio text word segmentation list wd1 and the i-th word wd2 i in the answer text word segmentation list, the similarity value range is [0-1], 0 means completely different, 1 means completely the same; i=0,...,M, M is the length of the aligned word segmentation list;

[0139] Edit distance, also known as Levenshtein distance, is a method for measuring the similarity between two strings. It was proposed by Russian scientist Vladimir Levenshtein in 1965. It is defined as the minimum number of single-character edit operations required to transform one string into another. These operations include inserting, deleting, or replacing a character. This is prior art, so the detailed implementation of the EditDistance() algorithm will not be detailed here.

[0140] 3)w i wd2 i The high-frequency word weight, judge wd2 i Whether it is a high-frequency word to determine its corresponding w i The value of this embodiment uses the binary weight determination: if wd2 i is a high-frequency word, then w i is 0.5; if wd2 i It is not a high frequency word, then w i is 1;

[0141] The determination of high-frequency words can be done by determining a high-frequency word list in advance through a large amount of text statistics. This is an existing technology and will not be described in detail here.

[0142] 4) Aggregate the edit distance similarities and weights of all words in the word list and then take the average value and multiply it by 100 to obtain the final text matching score SP4.

[0143] The pronunciation analysis module is used to calculate the average phoneme confidence of each word as the word pronunciation score through the phoneme confidence of the phoneme list, and then calculate the average of all word pronunciation scores and multiply it by 100 as the final pronunciation score SP5 of the audio data;

[0144] Specifically, the audio text is segmented to obtain a word list, and the phoneme confidence of each word is calculated based on the phoneme list to obtain the average pronunciation score of each word. Then, the pronunciation scores of all words are added together and divided by the length of the word list and multiplied by 100 to obtain the final pronunciation score SP5 of the audio data;

[0145] The comprehensive evaluation module is used to dynamically adjust the weights of the fluency score SP, grammar score SP3, pronunciation score SP5 and completeness score SP6 according to the length of the audio text, and calculate the final oral evaluation score Score; the completeness score SP6 is the larger value of the semantic score SP2 and the text matching score SP4;

[0146] Score=a1(L)×SP+a2(L)×SP3+a3(L)×SP5+a4(L)×SP6

[0147] Wherein, a1(L) is the fluency score dynamic weight; a2(L) is the grammar score dynamic weight; a3(L) is the pronunciation score dynamic weight; a4(L) is the completeness score dynamic weight; L is the audio text length, that is, the number of words in the audio text L;

[0148] The dynamic adjustment of each score weight according to the audio text length by the fluency score SP, the grammar score SP3, the pronunciation score SP5 and the completeness score SP6 specifically includes:

[0149] When the number of words L is greater than or equal to 1 and less than or equal to 5, it is determined to be a short sentence; at this time, the pronunciation score (SP5) and the completeness score (SP6) have higher weights, because the short sentence needs to quickly convey the core information, and the pronunciation clarity and semantic accuracy are the key. The grammar score (SP3) has a lower weight, because the short sentence has a simple grammar structure (such as “I’m eight”), and the grammar error has less impact on understanding.

[0150] When the number of words L is greater than 5 and less than or equal to 10, it is determined to be a normal sentence;

[0151] The grammar score (SP3) and the fluency score (SP) have higher weights, because the sentence becomes longer and needs more complex grammar structure and coherent expression; the pronunciation score (SP5) has a moderate decrease in weight, but still needs to ensure understandability.

[0152] When the number of words L is greater than 10, it is determined to be a long sentence;

[0153] The fluency score (SP) and the grammar score (SP3) have the highest weights, because the long sentence needs to avoid pauses and grammar errors to ensure clear logic; the completeness score (SP6) has a lower weight, because the long sentence itself contains more information, and the semantic matching degree is weakened; finally, the following dynamic weight distribution formula is obtained:

[0154]

[0155] The completeness score SP6 is used to measure the matching degree of the student’s answer to the requirements of the question, and its core goal is to evaluate whether the answer completely and accurately conveys the core intention of the question. In order to avoid the limitations of single dependence on text matching (such as matching failure caused by synonym rewriting of the question), the completeness score adopts the following rules:

[0156] SP6 = max (SP4, SP2)

[0157] Wherein, SP4 is the text matching score, and SP2 is the semantic score.

[0158] The fluency analysis module uses multi-dimensional quantitative evaluation, dynamic parameter configuration, reasonable weight distribution, and precise punishment mechanism through phoneme-level analysis to avoid the limitations of a single indicator, ensure the coordinated consideration of speech rate and pauses, improve the fairness and practicality of the evaluation, meet the authoritative language ability standards, enhance the authority and credibility of the evaluation results, and realize scientific, scenario-based, and personalized evaluation of oral fluency, which is suitable for different language levels and scenarios.

[0159] The comprehensive evaluation module focuses on different evaluation points for different audio text length answers and uses different scoring weights to ensure that the evaluation results are closer to the actual oral language ability.

[0160] The pre-trained semantic model captures the deep intention of the answer to avoid misjudgment due to different expressions. The matching method based on the edit distance quickly filters the completely consistent answers, and the combination of semantic and text matching judges the completeness, which improves the accuracy and robustness of the evaluation.

[0161] Embodiment Two

[0162] Reference Figure 2 The embodiment provides a spoken language evaluation method based on multiple factors, based on the above-mentioned spoken language evaluation system based on multiple factors, comprising the following steps:

[0163] S1, collecting voice data of user answers to questions; the scene of the question includes oral composition and situational oral language;

[0164] S2, extracting Fbank features of the voice data, and inputting the Fbank features into a pre-trained text phoneme recognition model to obtain audio text, phoneme list and phoneme time boundary list corresponding to the voice data;

[0165] S3, calculating the number of phonemes per minute, pause duration and pause frequency according to the phoneme list and phoneme time boundary list, and calculating the fluency score SP according to the number of phonemes per minute, pause duration and pause frequency. The final fluency score formula is as follows:

[0166] SP=S*0.85-P*0.15;

[0167] Wherein, SP is the final fluency score; S is the basic fluency score; P is the pause penalty; the 0.85 in the final fluency score formula is the speech rate weight, and the 0.15 is the pause weight; the speech rate weight + the pause weight = 1,

[0168] S4, obtaining a semantic score of the audio text by a pre-trained semantic model; an input of the pre-trained semantic model is the audio text and the answer text, the two texts are respectively segmented and embedded by the semantic model, finally two sentence vector representations are obtained, and a cosine similarity of the two sentence vectors is calculated, and the cosine similarity is multiplied by 100 to obtain the semantic score SP2 of the audio text;

[0169] S5, obtaining a grammar score SP3 of the audio text by a pre-trained grammar model; the pre-trained grammar model is obtained by fine-tuning based on a RoBERTa model, and is obtained by fine-tuning on a large grammar correction data set;

[0170] S6, matching the audio text and the answer text according to an edit distance-based text matching method to obtain a text matching score SP4;

[0171] S7, calculating an average value of phoneme confidence of each word contained in a phoneme list as a word pronunciation score, and then calculating an average value of all word pronunciation scores multiplied by 100 as a final pronunciation score SP5 of the audio data;

[0172] S8, taking a larger value of the semantic score SP2 and the text matching score SP4 as a completeness score SP6, dynamically adjusting the weight of each score according to the length of the audio text through the fluency score SP, the grammar score SP3, the pronunciation score SP5 and the completeness score SP6, and calculating a final oral test score Score:

[0173] Score = a1(L) × SP + a2(L) × SP3 + a3(L) × SP5 + a4(L) × SP6

[0174] Wherein, a1(L) is a dynamic weight of the fluency score; a2(L) is a dynamic weight of the grammar score; a3(L) is a dynamic weight of the pronunciation score; a4(L) is a dynamic weight of the completeness score; L is the length of the audio text, that is, the number of words L in the audio text.

[0175] S9, displaying the answer text and the scores of each dimension on the application end; the scores of each dimension include the fluency score SP, the semantic score, the grammar score SP3, the pronunciation score SP5, the completeness score SP6 and the final oral test score Score.

[0176] The application end displays the test results on the interactive interface, that is, the answer text and the scores of each dimension, so that the user can clearly see the test scores of each dimension, and can improve his oral expression and enhance his oral expression ability accordingly.

[0177] The present embodiment provides a method for oral evaluation based on multi-dimensional factors, including: collecting voice data of users answering questions; obtaining audio text, phoneme list and phoneme time boundary list corresponding to the voice data according to a pre-trained text phoneme recognition model; calculating the number of phonemes pronounced per minute, pause length and number of pauses according to the phoneme list and phoneme time boundary list, and calculating a fluency score; obtaining a semantic score of the audio text through a pre-trained semantic model; obtaining a grammatical score of the audio text through a pre-trained grammatical model; matching the audio text and the answer text according to a text matching method based on edit distance to obtain a text matching score; obtaining a final pronunciation score of the audio data through the phoneme confidence of the phoneme list; a comprehensive evaluation module dynamically adjusts the weight of each dimension score according to the length of the audio text, and calculates the final oral evaluation score; the method of the present embodiment comprehensively evaluates oral ability through five dimensions of fluency, semantics, grammar, text matching and pronunciation, avoiding the limitations of a single indicator;

[0178] The method of this embodiment achieves comprehensiveness, fairness, accuracy and efficiency in oral assessment through technical means such as multi-dimensional evaluation, dynamic weight adjustment, combination of semantic and text matching, and phoneme-level analysis. It is particularly suitable for scenarios such as education, examinations, and language learning, and can more scientifically reflect students' oral ability.

[0179] Example 3

[0180] refer to Figure 3 , Figure 3 is a step in the above-described method embodiment. Alternatively, when the processor 21 executes the computer program, a schematic diagram of the structure of a multi-dimensional oral assessment device is implemented in this embodiment. The multi-dimensional oral assessment device 20 in this embodiment includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, the functions of each module / unit in the above-described device embodiments are implemented.

[0181] Exemplarily, the computer program can be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to implement the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the device 20 for oral language assessment based on multi-dimensional factors. For example, the computer program can be divided into the modules described in Example 2. The specific functions of each module can be referred to the operating process of the device described in the above embodiment, and will not be repeated here.

[0182] The device 20 for oral test evaluation based on multi-dimension factors can include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art can understand that the schematic diagram is only an example of the device 20 for oral test evaluation based on multi-dimension factors, and does not constitute a limitation on the device 20 for oral test evaluation based on multi-dimension factors, which can include more or less components than the diagram, or combine certain components, or different components, for example, the device 20 for oral test evaluation based on multi-dimension factors can also include an input / output device, a network access device, a bus, etc.

[0183] The processor 21 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The processor 21 is the control center of the device 20 for oral test evaluation based on multi-dimension factors, and connects various parts of the entire device 20 for oral test evaluation based on multi-dimension factors through various interfaces and lines.

[0184] The memory 22 can be used to store computer programs and / or modules, and the processor 21 realizes various functions of the device 20 for oral test evaluation based on multi-dimension factors by running or executing computer programs and / or modules stored in the memory 22, and calling data stored in the memory 22. The memory 22 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required for a function (such as a sound playing function, an image playing function, etc.), etc.; and the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0185] The modules / units of the device 20 for oral test evaluation based on multi-dimensional factors are stored in a computer readable storage medium if they are realized in the form of software function units and sold or used as independent products. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0186] It should be noted that the above-described device embodiments are only illustrative, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. In addition, the connection relationship between the modules in the device embodiment provided by the present application indicates that there is a communication connection between them, which can be realized as one or more communication buses or signal lines. Those skilled in the art can understand and implement it without creative labor.

[0187] In this embodiment, the device can be a mobile phone, a tablet or a computer.

[0188] The part of the present application not described in detail is the prior art, and it is obvious to those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and the present application can be realized in other specific forms without departing from the spirit or essential characteristics of the present application; therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and all changes falling within the meaning and scope of the equivalent elements are intended to be included in the present application.

Claims

1. A spoken language assessment system based on multi-dimensional factors, characterized by: Specifically including the following modules: The voice collection module is used to collect the voice data of users answering questions; the scenarios of the questions include oral composition and situational speaking; The speech recognition module is used to extract the Fbank features of the speech data and input the Fbank features into the pre-trained text phoneme recognition model to obtain the audio text, phoneme list and phoneme time boundary list corresponding to the speech data; A fluency analysis module is used to calculate the number of phonemes, pause duration, and number of pauses per minute of pronunciation based on the phoneme list and the phoneme time boundary list, and to calculate a fluency score SP based on the number of phonemes, pause duration, and number of pauses per minute of pronunciation; A semantic analysis module is used to obtain a semantic score SP2 of the audio text through a pre-trained semantic model; the pre-trained semantic model is fine-tuned based on the pre-trained SentenceTransformer model; A grammar analysis module is used to obtain a grammar score SP3 of the audio text through a pre-trained grammar model; the pre-trained grammar model is fine-tuned based on the RoBERTa model; A text matching module is used to match the audio text and the answer text according to a text matching method based on edit distance to obtain a text matching score SP4; The pronunciation analysis module is used to calculate the average phoneme confidence of each word as the word pronunciation score through the phoneme confidence of the phoneme list, and then calculate the average of all word pronunciation scores and multiply it by 100 as the final pronunciation score SP5 of the audio data; The comprehensive evaluation module dynamically adjusts the weights of each score according to the length of the audio text through the fluency score SP, grammar score SP3, pronunciation score SP5 and completeness score SP6, and calculates the final oral evaluation score Score; the completeness score SP6 is the larger value of the semantic score SP2 and the text matching score SP4.

2. The method according to claim 1, characterized in that The text phoneme recognition model adopts the Transformer-CTC architecture, and its specific structure is as follows: the input is the FBank feature of the audio, which is downsampled and local features are extracted through two layers of convolutional layers. The local features are then projected to the input dimension of the Transformer through a linear layer. Then, multiple Transformer encoder layers are passed through. Each Transformer encoder layer contains a multi-head self-attention mechanism and a feedforward neural network to capture long-term dependencies and output high-level acoustic features. Finally, the high-level acoustic features are mapped to the phoneme label space through a linear layer, and the probability of each phoneme label in each time frame is output. Finally, the CTC decoder outputs a phoneme time boundary list consisting of audio text, a phoneme list, and the time boundary of each phoneme; wherein each element of the phoneme list includes a specific phoneme and its corresponding confidence.

3. The method according to claim 1, characterized in that The fluency score SP is shown in the following formula: SP = S × 0.85 - P × 0.15; Among them, SP is the final fluency score; S is the basic fluency score; P is the pause penalty score; The calculation formula for the basic fluency score S is as follows: S=100×[1-|(PPM-PPM') / PPM'|^α]; Where PPM' is the optimal number of phonemes pronounced per minute; α is the speech rate sensitivity adjustment coefficient, and the value of α varies depending on the scenario of the question; the value range of S is [0,100], and the unit is minute; The calculation formula for the pause penalty point P is as follows: P=β1×N2+β2×N3+β3×T p Among them, N2 is the number of 2-second pauses; N3 is the number of 3-second pauses; T p is the total pause duration; β1 is the 2-second pause penalty coefficient; β2 is the 3-second pause penalty coefficient; β3 is the total pause duration penalty coefficient.

4. The method according to claim 1, wherein The input of the pre-trained semantic model is the audio text and the answer text. The two texts are segmented and embedded respectively through the semantic model, and finally two sentence vector representations are obtained. The cosine similarity of the two sentence vectors is then calculated, and the cosine similarity is multiplied by 100 to obtain the semantic score SP2 of the audio text; the answer text is obtained in advance based on the question.

5. The method according to claim 1, wherein The input of the pre-trained grammar model is the word sequence [wd1, wd2, ..., wdi] corresponding to the audio text, where i is the number of words in the audio text. The output of the pre-trained grammar model is a predicted label sequence [l1, l2, ..., li] of the same length as the word sequence. Each label li belongs to a predefined operation label set, including CORRECT, DELETE, REPLACE, and INSERT_AFTER, indicating the model's grammatical judgment and recommended operation on the word wdi. The predicted label sequence is traversed, and the number N2 of words marked as CORRECT is obtained by counting. N2 is divided by the length of the word sequence corresponding to the audio text and multiplied by 100 to obtain the grammatical score of the audio text.

6. The method according to claim 1, characterized in that The text matching method based on edit distance is shown in the following formula: Among them, SP4 is the text matching score; wd1 i Represents the i-th word in the audio text; wd2 i Represents the i-th word in the answer text; N is the total number of matching word pairs; ES(wd1 i , wd2 i ) is the edit distance similarity function, used to represent wd1 i with wd2 i The edit distance similarity between them ranges from [0-1], where 0 means completely different and 1 means completely identical; i wd2 i The weight of high-frequency words.

7. The method according to claim 6, characterized in that The edit distance similarity function is as follows: Among them, EditDistance() is the edit distance algorithm; len() is the length function; max() is the maximum value function.

8. The method according to claim 1, characterized in that The final oral assessment score is calculated as follows: Score=a1(L)×SP+a2(L)×SP3+a3(L)×SP5+a4(L)×SP6 Among them, a1(L) is the dynamic weight of the fluency score; a2(L) is the dynamic weight of the grammar score; a3(L) is the dynamic weight of the pronunciation score; a4(L) is the dynamic weight of the completeness score; L is the length of the audio text, that is, the number of words L in the audio text.

9. The method according to claim 8, characterized in that The dynamic weight distribution formula is implemented by dynamically adjusting the weights of the fluency score SP, grammar score SP3, pronunciation score SP5, and completeness score SP6 according to the length of the audio text; the dynamic weight distribution formula is as follows:

10. A spoken language assessment method based on multi-dimensional factors, based on the spoken language assessment system based on multi-dimensional factors according to any one of claims 1 to 9, characterized in that: The following steps are involved: S1. Collecting voice data of users answering questions; the scenarios of the questions include oral composition and situational speaking; S2. Extract the Fbank features of the speech data and input the Fbank features into the pre-trained text phoneme recognition model to obtain the audio text, phoneme list and phoneme time boundary list corresponding to the speech data; S3. Calculate the number of phonemes, pause duration, and number of pauses per minute of pronunciation based on the phoneme list and the phoneme time boundary list, and calculate a fluency score SP based on the number of phonemes, pause duration, and number of pauses per minute of pronunciation; S4. Obtaining a semantic score SP2 of the audio text through a pre-trained semantic model; the pre-trained semantic model is fine-tuned based on the pre-trained SentenceTransformer model; S5. Obtaining a grammar score SP3 of the audio text through a pre-trained grammar model; the pre-trained grammar model is fine-tuned based on the RoBERTa model; S6. Match the audio text and the answer text according to a text matching method based on edit distance to obtain a text matching score SP4; S7, calculating the average phoneme confidence of each word as the word pronunciation score through the phoneme confidence of the phoneme list, and then finding the average of all word pronunciation scores and multiplying it by 100 as the final pronunciation score SP5 of the audio data; S8. Dynamically adjust the weights of each score according to the length of the audio text through the fluency score SP, grammar score SP3, pronunciation score SP5 and completeness score SP6, and calculate the final oral evaluation score Score; the completeness score SP6 is the larger value of the semantic score SP2 and the text matching score SP4.

Images