Machine learning feature data processing method and related equipment

Abstract

The invention provides a machine learning feature data processing method and related equipment, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring original voice data input by a plurality of users for an open oral evaluation question type; extracting first voice feature data containing a first number of dimension voice features from the original voice data; according to the relevancy between the speech features of each dimension and the spoken language evaluation result, second speech feature data containing speech features of a second number dimension are screened out from the first speech feature data, and the second number dimension is smaller than the first number dimension; and constructing a plurality of voice feature sets according to the second voice feature data, and screening out a voice feature set containing a third number of dimensions of voice features from the plurality of voice feature sets by adopting a wrapping method as a self-defined voice feature set so as to perform machine learning on the spoken language evaluation model. According to the method, the feature data for training the spoken language evaluation model can be optimized, so that the performance and efficiency of the trained model are greatly improved.

Description

Background Art

[0002] Computer-assisted language testing is designed to measure the language proficiency of the test-taker based on his / her oral responses. Replacing manual testing with computers has the advantages of stability, fairness, and cost savings. In recent years, computer-assisted language testing has been increasingly widely used in various exams, mock tests, teaching, practice, and other scenarios.

[0003] In the oral assessment, open-ended questions refer to those that require free expression, such as story retelling and topic summary. Compared with text-related follow-up reading questions and question-and-answer questions with relatively fixed answers, open-ended questions have diverse answers and give candidates more room to express themselves.

[0004] There are also regional differences in retelling questions. For example, the retelling question of a certain province's college entrance examination quoted the college entrance examination syllabus of Province A, and the retelling question was defined as follows: "Section C story retelling requires candidates to first listen to a monologue of about two minutes, and the recording is played twice. After one minute of preparation, candidates begin to retell what they heard. Candidates are required to retell in their own language as much as possible, and the retelling should cover as many information points of the original text as possible. The monologue genres selected are mainly narrative and argumentative essays." For another example, the open-ended topic summary requirements for the oral English test in City B are as follows: "You have 2 minutes to read the key points on the screen and prepare for the answer. After hearing the prompt to start recording, please express the topic in English. Your expression must include the following key points and no less than 7 sentences, and the time is 1 minute."

[0005] It can be seen that in the design of an open-ended question system, it is necessary to take into account the differences in assessment requirements for different regions, different stages of education, and different question types, and to use massive data for regional training to obtain an assessment model that can accurately evaluate open-ended oral assessment questions.

[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0007] The present disclosure provides a method for processing feature data of machine learning and related equipment, which at least to a certain extent overcomes the technical problem that the evaluation results of oral evaluation models in related technologies for open-ended oral evaluation questions are not accurate enough.

[0008] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0009] According to one aspect of the present disclosure, a method for processing feature data for machine learning is provided, comprising: obtaining original speech data input by multiple users for open-ended oral assessment questions; extracting first speech feature data from the original speech data, wherein the first speech feature data is used to characterize the spectral information of the original speech data, and the first speech feature data comprises: speech features of a first quantity dimension; filtering out second speech feature data from the first speech feature data according to the correlation between the speech features of each dimension and the oral assessment result, wherein the second speech feature data comprises: speech features of a second quantity dimension, and the second quantity dimension is smaller than the first quantity dimension; constructing a plurality of speech feature sets according to the second speech feature data, and using a wrapping method to filter out a custom speech feature set for machine learning of an oral assessment model from the plurality of speech feature sets, wherein the custom speech feature set comprises: speech features of a third quantity dimension, and wherein the third quantity dimension is smaller than the second quantity dimension.

[0010] In some embodiments, after constructing multiple speech feature sets based on the second speech feature data, and using the wrapping method to filter out a custom speech feature set from the multiple speech feature sets for machine learning of the oral evaluation model, the method further includes: training a lightweight gradient boosting machine LightGBM model based on the custom speech feature set to obtain an oral evaluation model for evaluating open-ended oral evaluation questions.

[0011] In some embodiments, after training the lightweight gradient boosting machine LightGBM model according to the custom speech feature set to obtain a spoken language evaluation model for an open-ended spoken language evaluation question type, the method further includes: obtaining speech data to be evaluated input by a target user for an open-ended spoken language evaluation question type; extracting corresponding custom speech features from the speech data to be evaluated according to the custom speech feature set; inputting the extracted custom speech feature set into a pre-trained spoken language evaluation model, and outputting the spoken language evaluation result of the target user for the open-ended spoken language evaluation question type.

[0012] In some embodiments, after obtaining the voice data to be evaluated input by the target user for an open-ended oral assessment question type, the method further includes: performing voice recognition on the voice data to be evaluated based on a predefined voice recognition system to obtain a voice recognition result, a pronunciation evaluation result, and a fluency evaluation result, wherein the voice recognition result is used to extract the custom voice feature.

[0013] In some embodiments, the speech recognition system includes: an acoustic model module, a pronunciation dictionary module, a topic language model module and a decoder module.

[0014] In some embodiments, the custom speech feature set includes the following types of speech features: vocabulary features, text similarity features, word vector semantic similarity features, and pronunciation features.

[0015] According to another aspect of the present disclosure, a machine learning feature data processing device is also provided, including: a speech data acquisition module, used to acquire original speech data input by multiple users for open-ended oral assessment questions; a speech feature extraction module, used to extract first speech feature data from the original speech data, wherein the first speech feature data is used to characterize the spectral information of the original speech data, and the first speech feature data includes: speech features of a first quantitative dimension; a speech feature preliminary screening module, used to screen out second speech feature data from the first speech feature data according to the correlation between the speech features of each dimension and the oral assessment results, wherein the second speech feature data includes: speech features of a second quantitative dimension, and the second quantitative dimension is smaller than the first quantitative dimension; a speech feature fine screening module, used to construct multiple speech feature sets based on the second speech feature data, and use a wrapping method to screen out a custom speech feature set from the multiple speech feature sets for machine learning of an oral assessment model, and the custom speech feature set includes: speech features of a third quantitative dimension, wherein the third quantitative dimension is smaller than the second quantitative dimension.

[0016] According to another aspect of the present disclosure, an electronic device is also provided, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the machine learning feature data processing methods described by Shangshanhu by executing the executable instructions.

[0017] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the machine learning feature data processing method described in any one of the above is implemented.

[0018] According to another aspect of the present disclosure, a computer program product is also provided, including: a computer program or instructions, which, when executed by a processor, implements any one of the above-mentioned machine learning feature data processing methods.

[0019] The machine learning feature data processing method, apparatus and related equipment provided in the embodiments of the present disclosure, after collecting the original voice data input by multiple users for open-ended oral assessment questions, extract voice features of a first quantity dimension from the original voice data, and then filter out voice features of a second quantity dimension from the voice features of the first quantity dimension based on the correlation between the voice features of each dimension and the oral assessment results, finally construct multiple voice feature sets based on the voice features of the second quantity dimension, and use the wrapping method to filter out a custom voice feature set from the multiple voice feature sets for machine learning of the oral assessment model.

[0020] Through the embodiments of the present disclosure, speech feature data suitable for training a spoken language evaluation model can be screened out, so that the trained spoken language evaluation model can be greatly improved in terms of performance and efficiency.

[0021] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0023] Figure 1 A schematic diagram of an application system architecture in an embodiment of the present disclosure is shown;

[0024] Figure 2 A flow chart of a method for processing feature data of machine learning in an embodiment of the present disclosure is shown;

[0025] Figure 3 A flow chart of another method for processing feature data of machine learning in an embodiment of the present disclosure is shown;

[0026] Figure 4 A flow chart of oral evaluation based on an oral evaluation model in an embodiment of the present disclosure is shown;

[0027] Figure 5 A schematic diagram of an open oral evaluation system architecture in an embodiment of the present disclosure is shown;

[0028] Figure 6 A speech recognition flow chart according to an embodiment of the present disclosure is shown;

[0029] Figure 7 A block diagram of a speech recognition system based on FBANK features in an embodiment of the present disclosure is shown;

[0030] Figure 8 A flow chart of speech feature extraction in an embodiment of the present disclosure is shown;

[0031] Fig. 9 A flowchart of acoustic model training in an embodiment of the present disclosure is shown;

[0032] Fig.10 A language model training flow chart according to an embodiment of the present disclosure is shown;

[0033] Fig.11 A flowchart of an algorithm implementation of a speech recognition decoder in an embodiment of the present disclosure is shown;

[0034] Fig.12 A score prediction flow chart of a scoring engine in an embodiment of the present disclosure is shown;

[0035] Fig.13 A scoring model training flow chart according to an embodiment of the present disclosure is shown;

[0036] Fig.14 A schematic diagram of a machine learning feature data processing device in an embodiment of the present disclosure is shown;

[0037] Fig.15 A structural block diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0038] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0039] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0040] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0041] Figure 1FIG. 1 is a schematic diagram showing an exemplary application system architecture to which the machine learning feature data processing method in the embodiment of the present disclosure can be applied. Figure 1 As shown, the system architecture may include a terminal device 101 , a network 102 and a server 103 .

[0042] The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103, and can be a wired network or a wireless network.

[0043] Optionally, the wireless network or wired network described above uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, the data exchanged through the network is represented by technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPSec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.

[0044] The terminal device 101 can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, wearable devices, augmented reality devices, virtual reality devices, etc.

[0045] Optionally, the client of the application installed in different terminal devices 101 is the same, or the client of the same type of application based on different operating systems. Based on the different terminal platforms, the specific form of the client of the application can also be different, for example, the application client can be a mobile client, a PC client, etc.

[0046] The server 103 may be a server that provides various services, such as a background management server that provides support for the device operated by the user using the terminal device 101. The background management server may analyze and process the received request and other data, and feed back the processing results to the terminal device.

[0047] Optionally, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0048] Those skilled in the art will know that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration, and any number of terminal devices, networks and servers may be provided according to actual needs, and the embodiments of the present disclosure do not limit this.

[0049] Under the above system architecture, a method for processing feature data of machine learning is provided in an embodiment of the present disclosure, and the method can be executed by any electronic device with computing and processing capabilities.

[0050] In some embodiments, the machine learning feature data processing method provided in the embodiments of the present disclosure may be executed by a terminal device of the above-mentioned system architecture; in other embodiments, the machine learning feature data processing method provided in the embodiments of the present disclosure may be executed by a server in the above-mentioned system architecture; in other embodiments, the machine learning feature data processing method provided in the embodiments of the present disclosure may be implemented by the terminal device and server in the above-mentioned system architecture through interaction.

[0051] Figure 2 A flow chart of a method for processing feature data of machine learning in an embodiment of the present disclosure is shown as follows: Figure 2 As shown, the machine learning feature data processing method provided in the embodiment of the present disclosure includes the following steps:

[0052] S202, obtaining original speech data input by multiple users for open-ended oral assessment questions.

[0053] It should be noted that an open-ended oral assessment question is a method for evaluating a user's oral ability, which allows the assessor to answer questions or express in an unstructured manner, and usually does not restrict the content and form of the answer, so that the user's language application ability can be more comprehensively evaluated. The open-ended oral assessment question type in the disclosed embodiment refers to a question type that is mainly based on free play in the oral assessment (such as story retelling, talking about topics, etc.). Compared with the text-related follow-up reading question type and the question-and-answer question type with relatively fixed answers, the open-ended oral assessment question type has the characteristics of diversified answers and large room for play.

[0054] The original voice data obtained in the above S202 may be audio data collected by a voice collection device such as a microphone and input by the user when answering an open-ended oral assessment question.

[0055] S204, extracting first speech feature data from the original speech data, wherein the first speech feature data is used to characterize frequency spectrum information of the original speech data, and the first speech feature data includes: speech features of a first quantitative dimension.

[0056] It should be noted that the above-mentioned first voice feature data can be that original voice data is carried out voice recognition to obtain voice feature data, and the voice recognition method adopted is different, and the voice feature data obtained are also different. In one embodiment, it can be 72-dimensional FBANK (Filter Bank, filter bank) feature data. In speech signal processing, Filter Bank is a kind of tool that voice signal is converted into a series of different frequency bands. These frequency bands simulate the auditory filter in the human ear, and the energy distribution of voice signal in different frequency ranges can be captured. When specifically implemented, corresponding FBANK features can be extracted from the frequency spectrum of voice signal by a series of triangular filters (Mel filter bank), and each filter corresponds to a specific frequency range.

[0057] S206, filtering out second speech feature data from the first speech feature data according to the correlation between the speech feature of each dimension and the oral evaluation result, wherein the second speech feature data includes: speech features of a second number dimension, and the second number dimension is smaller than the first number dimension.

[0058] It should be noted that the speech feature data extracted from the original speech data usually contains speech features of many dimensions, and each dimension contains multiple features. When the number of features is too large, the model may learn the noise in the feature data rather than the actual signal, especially when the sample size is relatively small, resulting in poor performance of the model on unseen data. Directly using the first speech feature data containing the speech features of the first number dimension may cause overfitting problems. At the same time, the more features there are, the greater the computing and storage resources required. In the application scenario where the score is immediately given after the answer is given, too high a feature dimension will affect the real-time scoring and the user's experience. Therefore, in the embodiment of the present disclosure, after extracting the first speech feature data containing the speech features of the first number dimension (such as 72-dimensional FBANK feature data) from the original speech data, the second speech feature data containing the speech features of the second number dimension (such as 70-dimensional feature data) is screened out from the first speech feature data according to the correlation between the speech features of each dimension and the oral evaluation results.

[0059] S208, constructing multiple speech feature sets based on the second speech feature data, and using the wrapping method to filter out a custom speech feature set from the multiple speech feature sets for machine learning of the spoken language evaluation model, wherein the custom speech feature set includes: speech features of a third quantitative dimension, wherein the third quantitative dimension is smaller than the second quantitative dimension.

[0060] In the embodiment of the present formula, the process of filtering out the second voice feature data from the first voice feature data is called the feature initial screening stage; the process of filtering out the custom voice feature set from the second voice feature data is called the feature fine screening stage. In the feature initial screening stage, only the correlation between the feature itself and the manual score is considered. It can be understood as using the features of each dimension to perform linear regression with the manual score separately, and calculating the correlation coefficient of the linear regression. This process does not take into account the selection and training process of the regression model. In the fine screening stage, the wrapping method is used to further screen the features. Different from the method of directly selecting features based on statistical tests in the feature initial screening stage, the wrapping method evaluates the importance of features through specific prediction models. The selection of feature sets is based on the improvement of model performance and can be completed through a search algorithm, such as the RFE (Recursive Feature Elimination) algorithm. The RFE algorithm is a greedy algorithm for selecting feature subsets, which is commonly used in the fields of machine learning and data mining. The basic idea of ​​the RFE algorithm is to reduce the number of features by recursion, and retain the features that have the greatest impact on the model in each iteration.

[0061] From the above, it can be seen that the machine learning feature data processing method provided in the embodiment of the present disclosure, after collecting the original voice data input by multiple users for open-ended oral assessment questions, extracts voice features of a first quantity dimension from the original voice data, and then filters out voice features of a second quantity dimension from the voice features of the first quantity dimension based on the correlation between the voice features of each dimension and the oral assessment results, and finally constructs multiple voice feature sets based on the voice features of the second quantity dimension, and uses the wrapping method to filter out a custom voice feature set from the multiple voice feature sets for machine learning of the oral assessment model.

[0062] Through the machine learning feature data processing method provided in the embodiments of the present disclosure, speech feature data suitable for training a spoken language evaluation model can be screened out, so that the performance and efficiency of the trained spoken language evaluation model are greatly improved.

[0063] In some embodiments, after constructing multiple voice feature sets based on the second voice feature data, and using the wrapping method to select a custom voice feature set for machine learning of the spoken language evaluation model from the multiple voice feature sets, such as Figure 3 As shown, the machine learning feature data processing method provided in the embodiment of the present disclosure may also include the following steps:

[0064] S210, according to the custom speech feature set, train the lightweight gradient boosting machine LightGBM model to obtain a spoken language evaluation model for evaluating open-ended spoken language evaluation questions.

[0065] In some embodiments, after training the lightweight gradient boosting machine LightGBM model according to the custom speech feature set to obtain a spoken language evaluation model for evaluating open-ended spoken language evaluation questions, Figure 4 As shown, the machine learning feature data processing method provided in the embodiment of the present disclosure may also include the following steps:

[0066] S402, obtaining speech data to be evaluated input by the target user for an open-ended oral assessment question type;

[0067] S404, extracting corresponding custom speech features from the speech data to be evaluated according to the custom speech feature set;

[0068] S406, inputting the extracted custom speech feature set into a pre-trained oral evaluation model, and outputting the oral evaluation results of the target user for the open-ended oral evaluation question type.

[0069] Further, in some embodiments, after obtaining the speech data to be evaluated input by the target user for the open-ended oral assessment question type, such as Figure 4As shown, the machine learning feature data processing method provided in the embodiment of the present disclosure may also include the following steps:

[0070] S408, performing speech recognition on the speech data to be evaluated based on a predefined speech recognition system to obtain a speech recognition result, a pronunciation evaluation result, and a fluency evaluation result, wherein the speech recognition result is used to extract a custom speech feature.

[0071] In some embodiments, the speech recognition system in the embodiments of the present disclosure may include: an acoustic model module, a pronunciation dictionary module, a topic language model module and a decoder module.

[0072] In some embodiments, the custom speech feature set in the embodiments of the present disclosure may include but is not limited to the following types of speech features: vocabulary features, text similarity features, word vector semantic similarity features, and pronunciation features.

[0073] It should be noted that feature engineering is the basis for building effective machine learning models, which covers the process of extracting, selecting and converting information from raw data. This process is key to developing a model that can accurately predict or classify. Especially in open oral assessments, the purpose of feature engineering is to extract key information that helps evaluate oral ability from complex and multidimensional data, which includes extracting features that reflect students' language ability from voice and text data. Feature engineering is mainly divided into two steps: feature design and feature screening. Feature design focuses on how to define and calculate meaningful indicators from raw data, while feature screening focuses on how to optimize the selection and use of these features to improve the performance and efficiency of the model. The feature engineering used in the present disclosure embodiment includes the following:

[0074] 1) Feature design:

[0075] In the automatic scoring system, feature design is a key step to construct features that can effectively represent students' oral ability from raw data. These data mainly include recordings of students' oral responses and their text transcriptions. Based on the business scenario of middle school English retelling questions, feature design covers the following key categories:

[0076] 1.1) Lexical features: Lexical features can assess students’ performance in the breadth and depth of vocabulary use, including indicators such as the number of words and vocabulary diversity. For example, by comparing the degree of vocabulary matching between students’ answers and standard answers, we can assess students’ language richness and accuracy.

[0077] 1.2) Text similarity feature: The text similarity feature evaluates the similarity between the student's answer and the standard answer in terms of structure and content through n-gram analysis, edit distance, longest common substring and other techniques. This type of feature helps to judge the accuracy and completeness of the student's retelling.

[0078] 1.3) Word embedding semantic similarity features: Using pre-trained word embedding models, these features measure the semantic proximity between student answers and standard answers. Through word embedding technology, the model can capture the deep semantic information in the answers, thereby assessing the depth of students’ understanding of the listening materials.

[0079] 1.4) Pronunciation features: Pronunciation features use acoustic models to analyze students’ pronunciation accuracy and fluency. This includes the probability of correct pronunciation of phonemes and overall language fluency, which is an important indicator for evaluating the naturalness of spoken language.

[0080] 2) Feature screening: Feature screening is a key step in optimizing the performance of machine learning models, especially when dealing with a large number of raw features. When there are too many features, the model may learn the noise in the data rather than the actual signal, especially when the sample size is relatively small, causing the model to perform poorly on unseen data. Directly using the 72-dimensional raw feature set may lead to the above-mentioned overfitting problem. At the same time, the more features there are, the more computing and storage resources are required. In application scenarios where the score is released immediately after the answer is given, the high feature dimension affects the real-time scoring and the user experience.

[0081] After obtaining the initial feature set, the following two steps are used to perform preliminary and fine screening of the features, which can effectively reduce the complexity of the model and improve prediction accuracy and operating efficiency:

[0082] 2.1) Initial screening of feature importance: This step preliminarily screens the features that have the greatest impact on the scoring results by calculating the correlation between the features and the final score. This helps focus on the most critical information and narrows the scope of feature selection.

[0083] 2.2) Feature fine screening: For the features retained in the initial screening stage, the wrapping method is used to recursively select feature subsets, further screen the features that are highly correlated with the score, and eliminate those with low contribution or irrelevant features. This not only reduces the complexity of the model, but also reduces the risk of overfitting.

[0084] The technical details of feature design and feature post-processing are introduced in detail below.

[0085] (I) Feature design: Based on manual analysis of audio and text of students’ answers in different score ranges of the middle school English test, the original 72-dimensional features were designed. The design ideas and calculation methods for each set of features are as follows:

[0086] 1) Lexical features: feat_1, feat_2, ..., feat_6, a total of 6 dimensions:

[0087] The breadth and depth of vocabulary use are key indicators for evaluating students' oral ability. By calculating the vocabulary overlap and uniqueness between students' answers and reference answers, these features aim to capture students' ability in vocabulary use and assess their specific performance in vocabulary use.

[0088] 1.1)feat_1: Number of words in student answer recognition text:

[0089] Business background and design reason: Vocabulary richness is one of the important indicators to measure students' oral expression ability. feat_1 evaluates the richness of students' expressions by comparing the total number of words in the students' answers with the number of words in the listening original text and reference answers.

[0090] Calculation method: Let N be the number of words in the student's answer, N1 be the number of words in the listening text, and N2 be the number of words in the reference answer, then feat_1=2N / (N1+N2).

[0091] 1.2)feat_2: Recognition text vocabulary of student answers:

[0092] Business background and design rationale: Vocabulary proximity can show the consistency between the language used by students and the standard answers. This feature is designed to assess whether students can effectively use core vocabulary to express their ideas.

[0093] Calculation method: Let V, V1 and V2 represent the deduplicated vocabulary of the student's answer, the listening text and the reference answer respectively, then feat_2 = 2V / (V1+V2); the higher the ratio, the closer the vocabulary used by the student is to the standard answer.

[0094] 1.3)feat_3: The number of words in the first-level vocabulary removed from the recognized text of the student's answer:

[0095] Business background and design reason: The first-level vocabulary refers to the vocabulary composed of English function words (such as articles a / an, prepositions of, etc.). This feature is designed to exclude the interference of function words when calculating the number of words. By excluding common function words in English (such as conjunctions, prepositions, etc.), this feature focuses more on evaluating students' ability to use content words, which can better reflect students' language application level.

[0096] Calculation method: Let M, M1, and M2 be the number of words in the student's answer, the listening text, and the reference answer after removing the first-level vocabulary list, then feat_3 = 2M / (M1+M2).

[0097] 1.4)feat_4: The recognized text of the student's answer removes the vocabulary of the first-level word list:

[0098] Business background and design rationale: This feature further refines the concept of feat_3 and accurately assesses the breadth and depth of students' use of core vocabulary by measuring the vocabulary diversity of non-function words.

[0099] Calculation method: Let V1, V1_1, and V2_2 be the vocabulary of the student's answer, the listening text, and the reference answer after removing the first-level vocabulary list, then feat_4 = 2V1 / (V1_1+V2_2).

[0100] 1.5)feat_5: The recognized text of the student's answer removes the vocabulary of the secondary word list:

[0101] Business background and design reason: The secondary vocabulary refers to the common words in English corpus (such as better, begin), also known as the stop word list. This feature is designed to further exclude common stop words when calculating vocabulary. This feature aims to reveal students' ability to use more professional or complex vocabulary, which is usually related to advanced language expression ability.

[0102] Calculation method: Let V2, V1_2, and V2_2 represent the vocabulary after removing the secondary vocabulary, then feat_5 = 2*V2 / (V1_2+V2_2).

[0103] 1.6)feat_6: The number of words in the recognized text of the student's answer that overlap with the original listening text and the reference answer, excluding the primary vocabulary:

[0104] Business background and design reason: This feature focuses on evaluating the number of core words in the student's answer that overlap with the listening text and reference answers, and is an important indicator for measuring whether the student understands and accurately repeats key information. This feature is designed based on the consideration that the greater the intersection of the student's answer recognition text and the listening text and reference answer words, the more similar the student's answer is to the original text.

[0105] Calculation method: Assume that P1 and P2 represent the number of words in the student's answer that appear in the listening text and the reference answer respectively, then feat_6 = max(P1 / (M+M1), P2 / (M+M2)).

[0106] 2) Text similarity features: feat_7, ..., feat_17, a total of 11 dimensions:

[0107] In the oral retelling task, students need to express themselves as close to the content and structure of the original text as possible. Through methods such as n-gram analysis and edit distance, this type of feature evaluates the structural similarity between the student's answer and the listening original text or reference answer. The text similarity feature provides an objective standard to evaluate the accuracy and completeness of the student's retelling, which is crucial to improving students' language comprehension and expression skills.

[0108] 2.1) feat_7, feat_8, feat_9: n-gram features, n=1, n=2, n=3, a total of 3 dimensions:

[0109] Business background and design reason: For a sentence S, its n-gram combination is defined as the whole string of words of length n in the original sentence, recorded as ngram(S), where 1-gram corresponds to a single word, 2-gram corresponds to a phrase, collocation, vocabulary or the connection between the previous and next words of length 2, and so on. In oral scoring, the overlap of phrases between students' answers and reference answers can significantly indicate the accuracy and fluency of language use. By analyzing 1-gram, 2-gram, and 3-gram matches, the word-for-word accuracy of students' retelling can be carefully evaluated, thereby quantifying the similarity of vocabulary and phrases used between the recognized text and the reference material.

[0110] Calculation method: Given two sentences S1 and S2, with lengths L1 and L2 respectively, the number of n-gram combinations contained in the sentences are L1-n+1 and L2-n+1 respectively.

[0111] Suppose the number of n-gram combinations that appear in ngram(S1) but not in ngram(S2) is N1, the number of n-gram combinations that appear in ngram(S2) but not in ngram(S1) is N2, and the number of n-gram combinations that appear in both ngram(S1) and ngram(S2) is N. Then the n-gram features of sentences S1 and S2 are:

[0112] ngram_feat(S1, S2, n)=1-(N1+N2+N) / (L1+L2-2n+2);

[0113] Let the student's answer recognition text be W, the listening text be T, and the reference answer be R, then:

[0114] feat_7=max(ngram_feat(W, T, 1), ngram_feat(W, R, 1));

[0115] feat_8=max(ngram_feat(W, T, 2), ngram_feat(W, R, 2));

[0116] feat_9=max(ngram_feat(W, T, 3), ngram_feat(W, R, 3));

[0117] 2.2) feat_10: Difference in sentence length:

[0118] Business background and design reason: Sentence length can reflect the information richness of students' answers and the degree of matching with the reference answers. Close sentence length usually means that students can repeat what they heard more completely. This feature is designed to consider the closeness between the length of the identified text sentences and the original listening text or reference answers.

[0119] Calculation method: Assume that the number of words in the listening original text is N1, the number of words in the reference answer is N2, and the number of words in the recognized text is N, then feat_10=1-min(|N-N1| / max(N, N1), |N-N2| / max(N, N2)); This calculation formula takes into account the length difference between the student's answer and the listening original text or reference answer, and improves the fairness and accuracy of the scoring by minimizing this difference.

[0120] 2.3) feat_11, feat_12: Edit distance related features:

[0121] Business background and design reason: Edit distance is a method to measure the difference between two strings, which can effectively reflect the degree of difference between the student's oral answer and the standard answer. A smaller edit distance indicates that the student's answer is closer to the standard answer in terms of literal meaning. This type of feature is designed to use edit distance to measure the degree of match between the recognition text and the listening original text or reference answer. The smaller the edit distance, the more similar the two sentences are.

[0122] In order to analyze various feature calculation strategies, two methods are used to calculate the edit distance feature in the feature design stage: feat_11 considers the edit distance between the recognition text and the listening original text and the reference answer at the same time, and feat_12 only considers the edit distance between the recognition text and the listening original text. In the subsequent feature screening stage, these two features will be further analyzed and screened.

[0123] Calculation method: Assume that the number of words in the listening text is N1, the number of words in the reference answer is N2, and the number of words in the recognized text is N. Use the dynamic programming algorithm to find the edit distance between the recognized text and the listening text as edit1, and the edit distance between the recognized text and the reference answer as edit2. Then:

[0124] feat_11=1-min(edit1 / max(N, N1), edit2 / max(N, N2));

[0125] feat_12=1.0-edit1 / ((N1+N2) / 2);

[0126] In the above formula, since the absolute value of the edit distance is positively correlated with the text length, and the length of the original listening texts of different questions is different, in order to eliminate the influence of the text length, the edit distance is divided by the text length for normalization.

[0127] 2.4)feat_13: Common substring length:

[0128] Business background and design reason: The length of the common substring is an effective indicator for evaluating the degree of matching between two texts in terms of continuous vocabulary. Longer common substrings usually indicate that students can repeat information more accurately. This feature is designed to use the length of the common substring to measure the matching degree between the recognition text and the listening original text or reference answer. The longer the common substring length, the more similar the two sentences are.

[0129] Calculation method: Assume that the number of words in the listening original text is N1, the number of words in the reference answer is N2, and the number of words in the recognized text is N. Use the dynamic programming algorithm to obtain the longest common substring length between the recognized text and the listening original text as Lcs1, and the longest common substring length between the recognized text and the reference answer as Lcs2. Then: feat_13 = max(Lcs1 / max(N, N1), Lcs2 / max(N, N2)).

[0130] 2.5)feat_14: Common subsequence length:

[0131] Business background and design reasons: Common subsequences are similar to common substrings, with the following differences: Common substrings need to be continuous texts, i.e. local fragments such as words, phrases, and language blocks used in English texts; Common subsequences do not require continuity, and can eliminate the influence of modifying elements such as parentheses, adjectives, and adverbs on similarity calculations, thereby providing a more flexible way to measure text similarity. From the perspective of language usage characteristics, it measures the degree of match between the recognized text and the listening original text or reference answers.

[0132] Calculation method: Let the number of words in the listening text be N1, the number of words in the reference answer be N2, and the number of words in the recognition text be N. Using the dynamic programming algorithm, the longest common subsequence length between the recognition text and the listening text is Lcseq1, and the longest common subsequence length between the recognition text and the reference answer is Lcseq2, then: feat_14 = max(Lcseq1 / max(N, N1), Lcseq2 / max(N, N2)).

[0133] 2.6) feat_15, feat_16: intersection of two sentence vocabularies, totaling 2 dimensions:

[0134] Business background and design reason: In oral scoring, the consistency of the vocabulary of the student's answer and the listening text is a key indicator for evaluating the student's understanding. By quantifying the vocabulary intersection between the student's answer and the listening text or reference answer, this set of features effectively evaluates whether the student can accurately and completely repeat what he heard.

[0135] Calculation formula: Assume that after the original listening text has been deduplicated, the number of words in the vocabulary is V1, the number of words in the vocabulary of the recognized text is V2, and the number of words in the intersection of the original listening text and the vocabulary of the recognized text is V, then:

[0136] feat_15 = V / max(V1, V2);

[0137] feat_16 = V / min(V1, V2);

[0138] Among them, feat_15 takes into account the breadth of overall vocabulary use. This feature measures the proportion of intersectional vocabulary in the larger vocabulary of the listening original text or recognition text; feat_16 pays more attention to extreme cases and measures the proportion of intersectional vocabulary in the smaller vocabulary to emphasize the closeness in vocabulary use.

[0139] 2.7) feat_17: Identify the average length of text words:

[0140] Business background and design rationale: The average length of vocabulary can indirectly reflect the complexity and maturity of language use. Longer words usually mean that students use more advanced vocabulary, which may be associated with high-scoring answers. This feature is designed based on the consideration that, on average, the longer the length of the words used in the recognized text, the more complex the vocabulary, and the more likely it is to get a high score.

[0141] Calculation formula: Assume there are N words in the recognized text, and the number of characters in word W_n is len(W_n), then feat_17 is the average value of the word length len(W_n) in the recognized text:

[0142] feat_17=(len(W_1)+len(W_2)+…+len(W_N)) / N.

[0143] 3) Word vector semantic similarity features: feat_18, ..., feat_70, a total of 53 dimensions:

[0144] Retelling questions require students to accurately retell the story content of the listening text. This type of feature takes into account the similarity at the semantic level and uses word vector technology to evaluate the semantic proximity between students' answers and the listening text and reference answers, measuring students' ability to understand the listening text and accurately retell the story.

[0145] 3.1) feat_18: Cosine distance of word vectors at the chapter level:

[0146] Business background and design reason: The cosine distance of word vectors at the chapter level measures the degree of proximity of the overall semantic content. This feature is designed to measure students' ability to understand and express the overall story content. In order to calculate this feature, it is necessary to train word vectors with a large amount of English text in advance, store them in the scoring system as model resources, and count the word frequencies of all words in the vocabulary as the word vector weights of each word.

[0147] Calculation formula: First, query the pre-trained word vector model and word frequency, and obtain the word frequency of each word w in the text as f(w) and the word vector as vec(w); secondly, calculate the sentence vector of the word vector of the listening original text weighted by word frequency as vec(T), the sentence vector of the reference answer weighted by word frequency as vec(R), and the sentence vector of the recognition text weighted by word frequency as vec(W); finally, calculate the cosine distance dist1=cos(vec(W), vec(T)) between the recognition text sentence vector and the listening original text sentence vector, and the cosine distance dist2=cos(vec(W), vec(R)) between the recognition text sentence vector and the reference answer sentence vector.

[0148] Take the maximum value of the above two cosine distances, that is, between the listening original text and the reference answer, use the text content close to the student's answer to calculate the semantic similarity, and get:

[0149] feat_18=max(dist1, dist2).

[0150] 3.2) feat_19: Average cosine distance of word vectors at the sentence level:

[0151] Business background and design reason: By calculating the average cosine distance of word vectors at each sentence level, this feature provides a more fine-grained semantic similarity assessment, so that the score can more carefully reflect the sentence-by-sentence matching degree between the student's expression and the standard answer. To calculate this feature, it is necessary to prepare word vectors and word frequency files in advance, and split the recognition results into the same number of sentences as the listening original text / reference answer.

[0152] Calculation formula: Split the listening original text into P sentences, and the sentence vector obtained by weighted summing up the word vectors of the p-th sentence according to word frequency is vec(Tp). Split the recognition text into P sentences on average, and the sentence vector obtained by weighted summing up the word vectors of the p-th sentence according to word frequency is vec(Wp).

[0153] The cosine distance between the sentence vectors of the recognized text and the corresponding sentences of the listening original text is calculated. The average value of all cosine distances is D(W, T). Similarly, the average value of the cosine distance between the recognized text and the reference answer sentence is D(W, R).

[0154] Taking the maximum value of the above two cosine distances, we obtain: feat_19=max(D(W,T),D(W,R)).

[0155] 3.3) feat_20: Cosine distance feature between the identified text and the original listening text and the reference answer text vector

[0156] Business background and design reason: This feature measures the similarity between the student's answer and the original text or reference answer from the overall semantic level. A higher cosine distance value indicates that the student can retell the story he heard well, showing a higher language comprehension and expression ability. This feature is designed to measure the student's ability to semantically retell the content of the listening story. To calculate this feature, it is necessary to pre-count the word frequency and ngram weight of each word in the retelling question answer corpus.

[0157] Calculation formula: First, calculate the weight of each word based on word frequency and ngram weight:

[0158] Word weight = word frequency / 10 (1-gram weight) * number of words in the vocabulary;

[0159] The word vector of each word in the text is weighted and summed according to the word weight to obtain the text vector:

[0160] Text vector = sum(word weight * word vector);

[0161] According to the above formula, the text vectors of the recognized text, the listening original text, and the reference answer are calculated.

[0162] Calculate the maximum value of the cosine distance between the recognized text and the original listening text and the reference answer text vector:

[0163] feat_20=max(cos(recognition text vector, listening original text vector), cos(recognition text vector, reference answer text vector)).

[0164] 3.4) feat_21–feat_70: original text vector features, totaling 50 dimensions:

[0165] Business background and design rationale: When evaluating the language quality of students' oral retelling, high-scoring answers usually show semantic depth and complexity. This set of features is designed based on the fact that there are certain semantic features in the recognition text of students' answers that can distinguish high-scoring answers from low-scoring answers. For example, high-scoring answers use more advanced vocabulary, while low-scoring answers are simple repetitions of some common words. Word vectors can describe this feature from a semantic perspective. This series of features uses word vector technology to capture the semantic attributes in the answers, distinguishing high-scoring answers that use advanced vocabulary and semantically rich expressions from low-scoring answers that repeat simple, common vocabulary.

[0166] Calculation formula: When calculating this feature, a pre-trained 50-dimensional word vector table is used. This model is stored as a resource in the scoring system. Suppose the word vectors Vec(w) corresponding to all words w in the recognized text W are averaged, and the resulting 50-dimensional text vector is Vec(W). Taking each dimension of the text vector as a feature, we get features feat_21, feat_22, ..., feat_70.

[0167] 4) feat_71, feat_72: pronunciation features, a total of 2 dimensions:

[0168] Good pronunciation is the basis of fluent speaking. The pronunciation feature analyzes the students' pronunciation quality and fluency through an acoustic model. The pronunciation score evaluates the students' pronunciation accuracy as a whole, and the fluency feature reflects the students' fluency in language expression, which is extremely important for evaluating students' ease in using language.

[0169] 4.1)feat_71: Pronunciation points:

[0170] Business background and design reason: The accuracy of pronunciation directly affects the results of oral English scoring. This feature is designed based on the consideration that the better the student's pronunciation, the more likely they are to get a high score, and the degree of good pronunciation can be characterized by the posterior probability of phonemes. By accurately measuring the posterior probability of phonemes, this feature can quantitatively evaluate the accuracy of students' pronunciation. High-quality pronunciation is usually associated with high-scoring answers.

[0171] Calculation formula: The acoustic model, pronunciation dictionary and the recognized text of the student's answer are forced to align, and the phoneme string corresponding to the student's answer voice is obtained as P = p_1, p_2, ... p_N. For the phoneme p_n, the acoustic model is used to calculate the pronunciation score of the phoneme p_n, that is, the posterior probability is Score(p_n) = posterior(p_n), then feat_71 is the average of the posterior probabilities of all phonemes p in the phoneme string P corresponding to the student's answer audio:

[0172] feat_71=(Score(p_1)+Score(p_2)+...+Score(p_N)) / N.

[0173] 4.2) feat_72: Fluency:

[0174] Business background and design reasons: Fluency is another key dimension of oral assessment. Good fluency not only shows the proficiency of language mastery, but also reflects the coherence of thinking and the natural fluency of language. This feature is designed based on the consideration that the more fluent the student's pronunciation, the more likely they are to get a high score. When calculating fluency, the measurement indicators involved in fluency, such as speaking speed, pause duration, and pause frequency, are quantified to obtain four fluency measurement indicators. The manually scored data is used as the annotation set, and the final fluency score is obtained through regression as the feature of the final score.

[0175] Calculation formula: Assume that the number of frames of the student's answering speech is F, the number of speech frames is F_speech, the number of silent frames is F_silence, the number of recognized text words is N, and after the speech and the recognized text are forcibly aligned through the acoustic model, the number of speech phonemes is M_speech, and the number of silent phonemes sil is M_sil. First, calculate the following four fluency measurement indicators:

[0176] Percentage of effective speech duration: ft1 = F_speech / F;

[0177] The number of valid phonemes per unit time: ft2 = 3M_speech / (100*F);

[0178] How many words does a pause occur on average: ft3 = M_silence / N;

[0179] Total number of words divided by number of pauses: ft4 = N / M_silence;

[0180] Then, we regress the manually scored data to obtain the fluency features as follows:

[0181] feat_72=4.97-7.85*ft1+26.4*ft2+0.44*ft3+13.47*ft4.

[0182] (ii) Feature screening: After obtaining the above 72-dimensional original features, it is necessary to manually score the audio of students’ answers collected from the middle school English oral exam and practice scenarios, and use this as the labeled data to analyze and screen the features of each dimension. First, calculate the correlation between each dimensional feature and the final score as a measure of feature importance to perform a preliminary screening of the features; then use the wrapping method to further fine-tune the remaining features to reduce the feature dimension and improve the relevance of the score fitting. The detailed technical implementation of each step is as follows:

[0183] 1) Initial screening of feature importance: In the automatic oral scoring system, different features (such as pronunciation accuracy, fluency, grammatical correctness) have different degrees of influence on the score. Feature importance extracts features from the answer data of the annotated set and calculates the correlation between each dimension and the final score. The analysis can help determine which features have a significant impact on the scoring results. This step ensures that the scoring model focuses on the most influential features, thereby identifying the most critical features and improving the accuracy of the scoring.

[0184] When measuring the correlation between each feature and the manual score, the correlation coefficient is used as the measurement standard:

[0185] The n-th dimension feature feat_n and the manual scoring annotation set<feat_n_i,Label_i> The correlation coefficient between them can be calculated by the Pearson correlation coefficient formula, where feat_n_i is the n-th dimension feature of the i-th data in the manual scoring dataset, and Label_i is the numerical value of the manual scoring of the i-th data:

[0186]

[0187] Among them, X i is the value of the feature feat_n_i, Y i is the corresponding manual score Label_i, and are the average of the feature value and the score value respectively.

[0188] Formula (1) evaluates the covariance between the feature value and the score value by quantifying the product of their deviation from their mean. The value of the correlation coefficient shows the magnitude of the correlation between the feature and the score, and the positive and negative signs indicate whether the score increases or decreases as the feature value increases.

[0189] By calculating the correlation coefficient between each feature and the manual score, we can identify the features that have the most significant impact on the score. For example, if the "pronunciation accuracy" feature is found to be highly positively correlated with the score, then this feature should be given a higher weight in the scoring model. On the contrary, if the correlation between a feature and the score is close to zero, it means that it contributes little to predicting the score, and you can consider removing it from the model to simplify the model and reduce the risk of overfitting.

[0190] For the 72-dimensional original features feat_1, feat_2, ... feat_72, we analyzed the correlation coefficient of the manually scored data set, removed the features with low correlation, retained the features with high correlation, and preliminarily screened out the following 70-dimensional features:

[0191] 1.1) All vocabulary features are retained: feat_1, feat_2, feat_3, feat_4, feat_5, feat_6, a total of 6 dimensions.

[0192] 1.2) Text similarity features:

[0193] For the original 11-dimensional text similarity features, the 3-gram feature feat_9 with low relevance and the feature feat_12 that only uses the original listening text to calculate the edit distance are eliminated in the initial screening stage, and the remaining features feat_7, feat_8, feat_10, feat_11, feat_13, feat_14, feat_15, feat_16, and feat_17 with high relevance are retained, totaling 9 dimensions. The analysis of the screening results of these two features is as follows:

[0194] 1.2.1) Among the n-gram hit rate features, the 3-gram feature feat_9 has a low correlation with the manual score and is eliminated in the initial screening, while the 1-gram and 2-gram features are retained.

[0195] This feature selection result reflects the limitations of n-gram features with a large context length n. According to a detailed analysis of the answer data, 3-gram phrases or language blocks are easily inserted into the separated parts, thereby reducing the proportion of 3-gram hits. For example, the original listening text contains the music was simply unforgettable. The student did not remember the adverb simply, and the answer he repeated was the music was unforgettable. In this example, the 3-gram calculation formula will consider that the music was unforgettable in the student's answer did not appear in the original listening text and will not be included in the calculation of the 3-gram hit rate; however, from the perspective of teaching and research experts, simply only modifies the adverb unforgettable and plays a role in emphasizing the tone. It is not key information. The student's retelling answer already contains the main content of the original listening text. Music was unforgettable hits the main information of music was simply unforgettable, which should have a benefit in the calculation of text similarity. Therefore, the similarity calculated based on 3-gram is not highly correlated with the manual scoring of teaching and research experts.

[0196] 1.2.2) Among the edit distance features, feature feat_12, which only uses the listening original text to calculate the edit distance, has a low correlation with manual scoring and is removed. At the same time, feature feat_11, which takes into account the edit distance between the recognition text and the listening original text and the reference answer, is retained.

[0197] This screening result illustrates the necessity of the feature screening step. In the feature design stage, it is not necessarily possible to ensure that the feature calculation method is completely correct, so it is necessary to consider various situations and design different feature calculation methods. In the edit distance feature design, two dimensions of features are designed to determine whether the edit distance between the recognized text and the reference answer should be included in the calculation: the calculation process of feat_11 uses the normalized edit distance between the recognized text and the listening original text and the reference answer, and the minimum value of the two is taken, which is equivalent to extracting features between the listening text and the reference answer using an edit distance that is closer to the recognized text; feat_12 only uses the listening original text to calculate the edit distance. In the subsequent feature analysis and screening stage, a data-driven approach is used to further screen these two features based on the correlation coefficient between the features and the manual scores.

[0198] 1.3) All word vector semantic similarity features are retained: feat_18, ..., feat_70, a total of 53 dimensions.

[0199] 1.4) All pronunciation features are retained: pronunciation is divided into feat_71 and fluency feat_72, a total of 2 dimensions.

[0200] In the initial feature screening stage, only the correlation between the feature itself and the manual score is considered. It can be understood as using the features of each dimension to perform linear regression with the manual score separately and calculating the correlation coefficient of the linear regression. This process does not take into account the selection and training process of the regression model.

[0201] 2) Feature fine-screening: In the fine-screening stage, the wrapping method is used to further screen the features. Unlike the initial screening, which directly selects features based on statistical tests, the wrapping method evaluates the importance of features through specific prediction models. The selection of feature sets is based on the improvement of model performance, which is usually done through a search algorithm, such as recursive feature elimination (RFE). The implementation of RFE in the fine-screening stage uses lightGBM as the regression model, recursively repeating the following steps:

[0202] Step 1: Train the lightGBM model with the selected feature set;

[0203] Step 2: Calculate the gain (Gain) and split frequency (SplitFrequency) of each feature during lightGBM training as a measure of feature importance;

[0204] Step 3: Remove the least important features;

[0205] Step 4: Repeat steps 1 and 2 until the desired number of features is reached;

[0206] Step 2 uses two feature importance metrics of the lightGBM model.

[0207] 2.1) Gain: Gain refers to the information gain brought by a feature when it is used to split a node during model training. This usually refers to the total loss reduced by the feature during the splitting process (for example, based on Gini impurity, entropy, or mean squared error in regression tasks). This gain is calculated every time the feature is used to split the tree, and all these gains are accumulated throughout the training process to characterize the overall importance of the feature. The higher the gain, the more important the feature is in the model because it plays a significant role in reducing the total loss of the model.

[0208] Calculation method: At each split, the gain is calculated by how much the feature can reduce the loss function. This value is determined by comparing the difference in the loss function before and after the split. For example, in a regression problem using mean squared error as the loss function, the gain can be the reduction in the sum of squared residuals before and after the split.

[0209] 2.2) Split Frequency: Split frequency refers to the number of times a feature is used as a split node in all trees of the model. This indicator directly reflects the frequency of use of a feature in the model building process. If a feature is frequently used for splitting, it indicates that it has high practical value and influence in the model because it helps the model effectively distinguish different data points.

[0210] Calculation method: Simply count the number of times each feature is used to make decision splits in the process of building all decision trees. Each time this feature is used to create a new tree node, its count increases by one.

[0211] 3) 32-dimensional feature list: After the fine screening stage, the final 32-dimensional feature list is as follows:

[0212] 3.1) Lexical features: feat_1, feat_2, feat_3, feat_4, feat_5, feat_6, a total of 6 dimensions.

[0213] 3.1.1)feat_1: The number of words in the student answer recognition text.

[0214] 3.1.2)feat_2: The recognition text vocabulary of the student’s answer.

[0215] 3.1.3)feat_3: The number of words in the first-level vocabulary removed from the recognized text of the student’s answer.

[0216] 3.1.4)feat_4: The recognized text of the student’s answer removes the vocabulary of the first-level word list.

[0217] 3.1.5)feat_5: The recognized text of the student’s answer removes the vocabulary of the secondary word list.

[0218] 3.1.6)feat_6: The recognized text of the student’s answer removes the first-level vocabulary and the number of words that overlap with the listening original text and reference answers.

[0219] 3.2) Text similarity features: feat_7, feat_8, feat_10, feat_11, feat_13, feat_14, feat_15, feat_16, feat_17, a total of 9 dimensions.

[0220] 3.2.1) feat_7, feat_8: n-gram features, n=1, n=2, a total of 2 dimensions.

[0221] 3.2.2)feat_10: Difference in sentence length.

[0222] 3.2.3) feat_11: Normalized edit distance feature for identifying text, original listening text, and reference answers.

[0223] 3.2.4)feat_13: common substring length.

[0224] 3.2.5)feat_14: common subsequence length.

[0225] 3.2.6) feat_15, feat_16: intersection of two sentence vocabularies: a total of 2 dimensions.

[0226] 3.2.7)feat_17: Identify the average length of text words.

[0227] 3.3) Word embedding semantic similarity features: feat_18, feat_19, feat_21, feat_22, feat_24, feat_29, feat_30, feat_32, feat_38, feat_44, feat_47, feat_53, feat_54, feat_57, feat_60, feat_66, feat_67, a total of 15 dimensions.

[0228] 3.3.1) feat_18: cosine distance of word vectors at the paragraph level.

[0229] 3.3.2) feat_19: Average cosine distance of word vectors at the sentence level.

[0230] 3.3.3) feat_21, feat_22, feat_24, feat_29, feat_30, feat_32, feat_38, feat_44, feat_47, feat_53, feat_54, feat_57, feat_60, feat_66, feat_67: 13-dimensional features selected from the 50-dimensional text vector features through feature engineering.

[0231] 3.4) Pronunciation features: feat_71, feat_72, a total of 2 dimensions.

[0232] 3.4.1)feat_71: pronunciation points.

[0233] 3.4.2)feat_72: Fluency.

[0234] When the machine learning feature data processing method provided in the embodiments of the present disclosure is applied to open-ended oral assessment questions, the audio quality analysis of the speech is performed to detect problems with the recording or whether the volume is too loud / too soft; then, the speech is input into the speech recognition engine, and the text recognition result is decoded based on the offline trained deep neural network acoustic model and language model; next, feature extraction is performed on the recognition result to obtain features of answer similarity, pronunciation score, and fluency; then, the scoring engine regresses a model score based on the input features and the scoring model as the final score output.

[0235] Figure 5 A schematic diagram of an open oral evaluation system architecture in an embodiment of the present disclosure is shown. Figure 5 As shown in the figure, each module is introduced as follows:

[0236] 1) Voice input module: Voice input is divided into streaming and non-streaming types. Streaming input supports decoding while inputting, which is suitable for mobile practice mock exams and other services; non-streaming input supports offline decoding after inputting all audio, which is suitable for offline scoring services such as computer room mock exams. For input voice, whether it is streaming input or non-streaming input, it must undergo audio quality analysis. The module itself supports both streaming and non-streaming input methods. Afterwards, online decoding is performed under streaming input, and offline decoding is performed under non-streaming input. After decoding, the decoding result is output. The speech recognition process executed by the voice input module is as follows: Figure 6 shown.

[0237] 2) Audio quality analysis module: The acoustic environment of the examination room is complex and changeable, including recording problems, noise, reverberation, truncation, low signal-to-noise ratio, etc., and the training data cannot exhaust all possibilities. Therefore, it is necessary to perform audio quality detection on the input speech.

[0238] The audio quality analysis module first judges the recording file of each candidate and outputs 10 situations: 0: the volume is too loud, 1: the volume is normal, 2: the volume is too low, 3: empty audio, 4: bad audio that cannot be read, 5: the audio length is close to 0 seconds, 6: the current sound is too loud, 7: the signal-to-noise ratio is too low, 8: cheating in the recorded original text.

[0239] In addition, the audio quality analysis module will also detect whether the candidate's answers are all in English, and output three situations: 0: all in English, 1: partly in English, 2: no English.

[0240] In practice and exam modes, only audio with normal volume and spoken entirely in English will be scored normally. In other cases, corresponding prompts will be given in practice mode; in exam mode, problematic audio will be marked for manual marking.

[0241] 3) Speech recognition module: The function of this module is to recognize the input speech and output the corresponding text content. To achieve this goal, the speech recognition engine needs to load the offline trained model. According to the current mainstream Hidden Markov Model HMM (Hidden Markov Models) speech recognition system architecture, the speech recognition module is divided into five parts: speech feature extraction, acoustic model, language model, pronunciation dictionary and decoder.

[0242] The features use 72-dimensional FBANK (Filter Bank) features. The acoustic model is the latest LFMMI+TDNN-F (Lattice-Free Maximum Mutual Information+Factored Time Delayed Neural Network) deep learning architecture. This technical framework can learn the optimal pronunciation distribution from massive voice data; the language model is a domain-related trigram model (Trigram), which is trained from text related to the topic; the pronunciation dictionary corrects the phonetic notation of common words in middle school based on cmudict.0.7a. These three modules are loaded as resources of the decoder after offline training. After the engine is started, the decoder loads the three offline trained models to recognize the input speech. Figure 7 A block diagram of a speech recognition system based on FBANK features is shown.

[0243] 4) Speech feature extraction module: The process of extracting FBANK features from speech is as follows: First, pre-emphasize the input speech in the time domain to increase high-frequency energy; then, perform frame division and windowing, with a frame length of 25ms and a frame shift of 10ms, add a Hamming window to suppress side lobes, and perform a fast Fourier transform from the time domain to the frequency domain; next, filter according to the pre-designed Mel-scale triangular filter group to obtain the energy value of each filter output, take the logarithm to obtain a 24-dimensional FBANK feature; finally, calculate the first-order difference and second-order difference of the adjacent feature values ​​to obtain a 72-dimensional FBANK feature. The logic block diagram of speech feature extraction is as follows: Figure 8 shown.

[0244] 5) Acoustic model submodule: This module is used to calculate the posterior probability of the input speech. The acoustic model structure used in this system is TDNN-F. The acoustic model needs to be trained offline with a large amount of annotated speech data, which includes two parts: one is the public native speaker data, and the other is the specially annotated business data. Due to the uniqueness of the English speaking scene of Chinese middle school students, adding business data to the acoustic model training data can significantly improve the recognition rate and noise robustness.

[0245] The acoustic model training process is as follows: first, the speech data and speech annotations are forced to be aligned, and the expectation-maximization algorithm is used to train the monophone acoustic model; second, based on the monophone acoustic model and training data, statistics are calculated, decision tree clustering is performed, and a decision tree for triphone state clustering is generated; third, the three-factor model is trained using the expectation-maximization algorithm; fourth, in view of the differences between channels and speakers, LDA+MLLT (Linear Discriminative Analysis+Maximum Likelihood Linear Transform) transformation is added to perform speaker / channel adaptation; fifth, based on LDA+MLLT, SAT (Speaker Adaptive Training) speaker adaptation training is added to obtain the HMM-GMM (Hidden Markov Models–Gaussian Mixture Models) acoustic model for generating state labels; sixth, the training speech and annotations are forced to be aligned to obtain state-level annotations; finally, the TDNN-f network is supervised trained using speech features and labels to obtain the TDNN-f model. The acoustic model training process is as follows Fig. 9 shown.

[0246] 6) Pronunciation dictionary submodule: The pronunciation dictionary is used to give the pronunciation of each word, and adopts the ARPAbet phonetic transcription system of cmudict. Based on cmudict.0.7a, the pronunciation dictionary of this invention has been revised for the pronunciation of commonly used words in middle school, and finally included 134,000 pronunciation entries.

[0247] Some words may have multiple pronunciations, such as:

[0248] CONTRARY K AA1 NTR EH0 R IY0;

[0249] CONTRARY(2)K AH0 NTR EH1 R IY0;

[0250] In addition to multiple pronunciations, homophones introduce confusion when constructing the decoding space and need to be handled with disambiguation symbols (such as #1 and #2):

[0251] SEA S IY1=> SEA S IY1#1;

[0252] SEE S IY1=>SEE S IY1#2;

[0253] 7) Language model submodule: During speech recognition, the language model is used to calculate the probability of candidate hypotheses appearing. The basis of N-gram is Markov property, which assumes that the conditional distribution of the current word is only related to the previous n-1 words. The present invention adopts the trigram language model, and the training process is as follows: first, use the question recording and the reference answer, use these two texts to train the language model, and obtain a small trigram language model related to the question, and then use this language model as a seed to perform corpus screening in the English learner corpus based on the perplexity, screen out the question-related corpus, train to obtain the question-related English learner trigram, and then perform language model interpolation to obtain the question language model. The language model training process is as follows Fig.10 shown.

[0254] 8) Decoder: The decoder is the core of the speech recognition system. The decoder uses a token passing algorithm. It calculates the posterior probability of the phoneme state based on the acoustic model, and the probability of the candidate sequence appearing based on the language model, and uses dynamic programming beam search to find the optimal path. The specific steps are as follows: First, before the first frame of speech features comes in, initialize the token; thereafter, for each frame of features added, determine whether it is a transmitting state for each active state: for the transmitting state, generate a new token based on the decoding graph, and according to the idea of ​​beam search, cut off the suboptimal path whose cost is greater than a certain value from the optimal path, perform a round of token passing, and enter the next node; for the non-transmitting state, record the current optimal path prefix node and enter the next node. This cycle continues until the termination node is encountered. If there is a path to the termination node, backtrack to the optimal path to obtain the recognition result. The algorithm implementation process of the speech recognition decoder is as follows: Fig.11 shown.

[0255] 9) Scoring feature extraction module: For open-ended questions, the present invention designs 32-dimensional features, denoted as feat_1, feat_2, ... feat_32. The calculation of these 32-dimensional features, except for pronunciation score and fluency which are related to the original input audio, other features can be calculated based on the recognition text, the original listening recording and the reference answer. The feature design and calculation method are as described above and will not be repeated here. As follows:

[0256] 1) Lexical features: feat_1, feat_2, ..., feat_6, a total of 6 dimensions:

[0257] 1.1)feat_1: Number of words in student answer recognition text:

[0258] This feature is designed based on the consideration that the closer the number of words in the recognized text is to the reference answer, the more likely it is to get a high score. Let the number of words in the original listening text be N1, the number of words in the reference answer be N2, and the number of words in the recognized text be N. Then the number of words in the recognized text of the student's answer feat_1 = 2N / (N1+N2).

[0259] 1.2) feat_2: The vocabulary of the recognized text of the student's answer: This feature is designed to take into account that the closer the vocabulary of the recognized text is to the reference answer, the more likely it is to get a high score. Suppose the vocabulary of the original listening text after removing duplicate words is V1, the vocabulary of the reference answer is V2, and the vocabulary of the recognized text is V. Then feat_2 = 2V / (V1+V2).

[0260] 1.3)feat_3: The recognized text of the student's answer removes the number of words in the primary vocabulary: The primary vocabulary refers to the vocabulary composed of English function words (such as a / an). This feature is designed to eliminate the interference of function words when calculating the number of words.

[0261] Suppose that after removing the first-level vocabulary from the listening original text, the number of words contained is M1, after removing the first-level vocabulary from the reference answer, the number of words contained is M2, and after removing the first-level vocabulary from the recognition text, the number of words contained is M, then feat_3=2M / (M1+M2).

[0262] 1.4) feat_4: The vocabulary of the recognized text of the student's answer after removing the first-level vocabulary: This feature is designed to exclude the interference of function words when calculating the vocabulary. After removing the first-level vocabulary from the listening original text, the vocabulary is V1_1, after removing the first-level vocabulary from the reference answer, the vocabulary is V2_1, and after removing the first-level vocabulary from the recognized text, the vocabulary is V_1, then feat_4 = 2V_1 / (V1_1+V2_2).

[0263] 1.5) feat_5: The recognized text of the student’s answer removes the vocabulary of the secondary word list: The secondary word list refers to the stop word list in English (such as better, begin), which lists common words in English corpus. This feature is designed to eliminate the interference of stop words when calculating the vocabulary.

[0264] After removing the secondary vocabulary from the listening original text, the vocabulary is V1_2, after removing the secondary vocabulary from the reference answer, the vocabulary is V2_2, and after removing the secondary vocabulary from the recognized text, the vocabulary is V_2; then feat_5=2V_2 / (V1_2+V2_2).

[0265] 1.6) feat_6: The number of words in the recognized text of the student's answer that overlap with the original listening text and the reference answer after removing the first-level vocabulary: This feature is designed based on the consideration that the greater the intersection of the recognized text of the student's answer and the words in the original listening text and the reference answer, the more similar the student's answer is to the original text.

[0266] Suppose the number of words in the listening text after removing the first-level vocabulary is M1, the number of words in the reference answer after removing the first-level vocabulary is M2, the number of words in the recognized text after removing the first-level vocabulary is M, the number of words appearing in the listening text after removing the first-level vocabulary is P1, and the number of words appearing in the reference answer after removing the first-level vocabulary is P2, then:

[0267] feat_6=max(P1 / (M+M1), P2 / (M+M2)).

[0268] 2) Text similarity features: feat_7, ..., feat_17, a total of 11 dimensions:

[0269] 2.1) feat_7: Cosine distance of word vectors at the paragraph level: This feature is designed to use the cosine distance of word vectors at the paragraph level to measure the overall semantic similarity between the student's answer recognition text and the listening original text and reference answers at the paragraph level. In order to calculate this feature, it is necessary to train the word vectors with a large amount of English text in advance, and to count the word frequencies of all words in the vocabulary as the word vector weights of each word.

[0270] Suppose the word frequency of word w in the vocabulary is f(w), and the word vector is vec(w); suppose the sentence vector of the word vector of the listening text weighted by word frequency is vec(T), the sentence vector of the reference answer weighted by word frequency is vec(R), and the sentence vector of the recognition text weighted by word frequency is vec(W). Calculate the cosine distance between the recognition text sentence vector and the listening text sentence vector dist1=cos(vec(W), vec(T)), and the cosine distance between the recognition text sentence vector and the reference answer sentence vector dist2=cos(vec(W), vec(R)). Then feat_7 is the maximum value of dist1 and dist2.

[0271] 2.2) feat_8: Average cosine distance of word vectors at the sentence level: This feature is designed to measure the sentence-by-sentence semantic similarity between the student's answer recognition text and the listening original text and reference answers. To calculate this feature, it is necessary to prepare the word vector and word frequency files in advance, and split the recognition results into the same number of sentences as the listening original text / reference answers.

[0272] The listening original text is divided into P sentences, and the sentence vector obtained by weighted summing up the word vectors of the p-th sentence according to word frequency is vec(Tp). The recognition text is evenly divided into P sentences, and the sentence vector obtained by weighted summing up the word vectors of the p-th sentence according to word frequency is vec(Wp).

[0273] The cosine distance of the sentence vectors of the recognized text and the corresponding sentences of the listening original text is calculated, and the average value of all cosine distances is D(W, T); similarly, the average value of the cosine distance between the recognized text and the reference answer sentence is D(W, R); then feat_8 is the maximum value of D(W, T) and D(W, R).

[0274] 2.3) feat_9, feat_10: n-gram features, n=1, n=2, a total of 2 dimensions:

[0275] This set of features is designed based on the consideration that the greater the n-gram phrase overlap between the recognized text and the listening original text and reference answers, the higher the score is likely to be.

[0276] For a sentence S, define its n-gram combination as the whole string of words of length n in the original sentence, recorded as ngram(S); given two sentences S1 and S2, with lengths L1 and L2 respectively, the number of n-gram combinations contained in the sentences are L1-n+1 and L2-n+1 respectively; let the number of n-gram combinations that appear in ngram(S1) but not in ngram(S2) be N1; let the number of n-gram combinations that appear in ngram(S2) but not in ngram(S1) be N2; let the number of n-gram combinations that appear in both ngram(S1) and ngram(S2) be N;

[0277] The n-gram features of sentences S1 and S2 are:

[0278] ngram_feat(S1, S2, n)=1-(N1+N2+N) / (L1+L2-2n+2).

[0279] Let the student's answer recognition text be W, the listening text be T, and the reference answer be R, then:

[0280] feat_9=max(ngram_feat(W, T, 1), ngram_feat(W, R, 1));

[0281] feat_10=max(ngram_feat(W, T, 2), ngram_feat(W, R, 2)).

[0282] Among them, ngram_feat is a function used to calculate text similarity; feat_9 represents the calculation of the unigram feature similarity between the student answer text W and the listening original text T, and the unigram feature similarity between the student answer text W and the reference answer R. Then, it takes the maximum value of these two similarity values ​​as the value of feat_9; feat_10 represents the similarity between two consecutive words of the student answer text W and the listening original text T, and the similarity between two consecutive words of the student answer text W and the reference answer R. Then, it also takes the maximum value of these two similarity values ​​as the value of feat_10.

[0283] 2.4) feat_11: Difference in sentence length. This feature is calculated based on the closeness between the sentence length of the recognized text and the original listening text or reference answer. Let the number of words in the original listening text be N1, the number of words in the reference answer be N2, and the number of words in the recognized text be N; then feat_11 = 1-min(|N-N1| / max(N, N1), |N-N2| / max(N, N2)).

[0284] 2.5) feat_12: Normalized edit distance. This feature is designed to use the edit distance to measure the degree of match between the recognized text and the listening original text or the reference answer. The smaller the edit distance, the more similar the two sentences are. The number of words in the listening original text is N1, the number of words in the reference answer is N2, and the number of words in the recognized text is N. Using the dynamic programming algorithm, the edit distance between the recognized text and the listening original text is edit1, and the edit distance between the recognized text and the reference answer is edit2; then feat_12 = 1-min(edit1 / max(N, N1), edit2 / max(N, N2)).

[0285] 2.6) feat_13: Common substring length. This feature is designed to use the common substring length to measure the matching degree between the recognition text and the listening original text or reference answer. The larger the common substring length, the more similar the two sentences are.

[0286] The number of words in the listening text is N1, the number of words in the reference answer is N2, and the number of words in the recognition text is N. Using the dynamic programming algorithm, the longest common substring length between the recognition text and the listening text is lcs1, and the longest common substring length between the recognition text and the reference answer is lcs2; then feat_13 = max(lcs1 / max(N, N1), lcs2 / max(N, N2)).

[0287] 2.7) feat_14: common subsequence length. This feature is designed to use the common subsequence length to measure the degree of match between the recognized text and the listening original text or reference answer. The larger the common substring length, the more similar the two sentences are. The difference between a common subsequence and a common substring is that the words in a common substring do not have to be adjacent in a sentence, as long as the order is consistent, while the words in a common subsequence must be adjacent. The number of words in the listening original text is N1, the number of words in the reference answer is N2, and the number of words in the recognized text is N. Using the dynamic programming algorithm, the longest common subsequence length between the recognized text and the listening original text is lcseq1, and the longest common subsequence length between the recognized text and the reference answer is lcseq2; then feat_14 = max(lcseq1 / max(N, N1), lcseq2 / max(N, N2)).

[0288] 2.8) feat_15, feat_16: intersection of two sentence vocabularies: 2 dimensions in total. This feature is designed based on the consideration that the more the vocabulary used in the recognition text intersects with the original listening text, the more likely it is to correctly repeat the content of the original listening text; suppose that after the original listening text is deduplicated, the number of words contained in the vocabulary is V1, the number of words contained in the vocabulary of the recognition text is V2, and the number of words contained in the intersection of the vocabulary of the original listening text and the recognition text is V.

[0289] Then feat_15=V / max(V1, V2), feat_16=V / min(V1, V2).

[0290] 2.9)feat_17: Identify the average length of text words.

[0291] This feature is designed based on the consideration that, on average, the longer the words used in the recognized text, the more complex the vocabulary, and the more likely it is to get a high score; assuming that the string length of word w in the recognized text is len(w), then feat_17 is the average value of len(w) of the recognized text.

[0292] 3) feat_18~feat_30: text vector features: a total of 13 dimensions.

[0293] This set of features is designed to take into account that there are certain semantic features in the recognition text of students' answers, which can distinguish high-scoring answers from low-scoring answers. For example, high-scoring answers use more advanced vocabulary, while low-scoring answers are simple repetitions of some common words. Word vectors can describe this feature from a semantic perspective. When calculating this feature, a pre-trained 50-dimensional word vector table is used. Suppose the word vectors vec(w) corresponding to all words w in the recognition text W are averaged, and the resulting 50-dimensional text vector is Vec(W); take the 4th, 9th, 10th, 12th, 18th, 24th, 27th, 33rd, 34th, 37th, 40th, 46th, and 47th dimensions of the text vector as text vector features, that is, feat_18 corresponds to the 4th dimension of the text vector Vec(W), feat_19 corresponds to the 9th dimension of the text vector Vec(W), ..., feat_30 corresponds to the 47th dimension of the text vector Vec(W).

[0294] 4) feat_31, feat_32: pronunciation features, a total of 2 dimensions.

[0295] 4.1) feat_31: pronunciation score. This feature is designed based on the fact that the better the pronunciation of the student, the more likely they are to get a high score, and the degree of good pronunciation can be characterized by the posterior probability of the phoneme. The acoustic model, pronunciation dictionary and the recognized text of the student's answer are used for forced alignment, and the phoneme string corresponding to the student's answer speech is obtained as P. For the phoneme p in it, the acoustic model is used to calculate the posterior probability of the phoneme p as posterior(p); then feat_31 is the average of the posterior probabilities of all phonemes p in the phoneme string P corresponding to the student's answer audio.

[0296] 4.2) feat_32: Fluency. This feature is designed based on the fact that the more fluent a student’s pronunciation is, the more likely they are to get a high score. Fluency is related to the student’s speaking speed, pause duration, pause frequency, and other features. When calculating fluency, four features are used for regression to get the final score:

[0297] Assume that the number of frames of the student's answering speech is F, the number of speech frames is F_speech, the number of silent frames is F_silence, and the number of recognized text words is N. After the speech and the recognized text are forced to be aligned through the acoustic model, the number of speech phonemes is M_speech, and the number of silent phonemes sil is M_sil. First, calculate the following four fluency features:

[0298] Percentage of effective speech duration: ft1 = F_speech / F;

[0299] The number of valid phonemes per unit time: ft2 = 3M_speech / (100*F);

[0300] How many words does a pause occur on average: ft3 = M_silence / N;

[0301] Total number of words divided by number of pauses: ft4 = N / M_silence;

[0302] Then feat32=4.97-7.85*ft1+26.4*ft2+0.44*ft3+13.47*ft4.

[0303] 6) Scoring engine module: The scoring engine inputs 32-dimensional features, uses the scoring model for regression, and obtains the final score. The scoring model uses the LightGBM (Light Gradient Boost Machine) model, which is an efficient implementation algorithm of the Gradient Boosting Decision Tree (GBDT) model. It has the advantages of fast training speed, high parallelization efficiency, flexible configuration, and the ability to use massive training data.

[0304] According to the principle of the gradient boosting decision tree model, the LightGBM model of the scoring engine is composed of N = 200 regression trees. Similar to the decision tree, each non-leaf node is a judgment condition, and each set of features takes different branches according to the judgment conditions. Each leaf node is a prediction score, in which the latter tree predicts the residual of the previous tree. The predicted value fluctuation of the leaf node is smaller, and the final total prediction score is the sum of the prediction values ​​of all N = 200 trees. In order to make the prediction results meet expectations, it is necessary to output 0 for predicted values ​​less than 0, and force the output of 100 for predicted values ​​greater than 100. In the actual exam, subsequent score adjustments will be made according to demand. The most common ones are shifting the mean and adjusting the variance so that the score distribution meets the school requirements. The scoring engine score prediction process is as follows: Fig.12 shown.

[0305] The training process of the LightGBM scoring model is as follows: extract training audio with manual scoring from the test audio. To ensure the consistency of the review process, each set of training data is delivered to two teachers. If the difference between the two readings is within 20% of the full score, the average score of the two readings is taken as the final score. If the difference between the two readings exceeds 20% of the full score, three readings are conducted and the score of the three readings is taken as the final score. After obtaining the labeled training set, the speech recognition system is used for recognition, and the scoring features are extracted for the recognition results to obtain 32-dimensional features. Together with the manual scoring labels, the LightGBM model is trained. The scoring model training process is as follows: Fig.13 shown.

[0306] From the above, it can be seen that the machine learning feature data processing method for open-ended oral assessment questions provided in the embodiments of the present disclosure, in order to solve the differences between different regions and different sub-question types, collects data from the corresponding regions for retelling questions, paraphrase questions and topic summary questions in different regions, and trains different scoring models to adapt to the scoring standards of different regions and different question types.

[0307] The machine learning feature data processing method for open-ended oral assessment questions provided in the embodiments of the present disclosure can achieve but is not limited to the following technical effects: ① Adding business data to acoustic model training to improve recognition rate and robustness and realize customization of acoustic environment; ② Adding question-related text to language model training with good customization; ③ For open-ended text, calculating pronunciation score and fluency to realize pronunciation evaluation; ④ In terms of feature design and selection, the scoring engine adopts original 32-dimensional features, combined with the LightGBM regression model, to achieve good prediction effect, more reasonable feature design, and better learn the information contained in massive data; ⑤ According to different regions and different question types, corresponding data is collected, and different scoring models are trained separately to adapt to the scoring standards of different regions and different question types. The effect is significantly better than using the parameters of the same question and the same scoring model to adapt to different situations.

[0308] Based on the same inventive concept, the present disclosure also provides a machine learning feature data processing device, as described in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0309] Fig.14 A schematic diagram of a machine learning feature data processing device in an embodiment of the present disclosure is shown. Fig.14 As shown, the device includes: a speech data acquisition module 1401, a speech feature extraction module 1402, a speech feature preliminary screening module 1403, and a speech feature fine screening module 1404.

[0310] Among them, the voice data acquisition module 1401 is used to obtain the original voice data input by multiple users for open-ended oral assessment questions; the voice feature extraction module 1402 is used to extract first voice feature data from the original voice data, wherein the first voice feature data is used to characterize the spectral information of the original voice data, and the first voice feature data includes: voice features of a first quantitative dimension; the voice feature initial screening module 1403 is used to screen out second voice feature data from the first voice feature data according to the correlation between the voice features of each dimension and the oral assessment results, wherein the second voice feature data includes: voice features of a second quantitative dimension, and the second quantitative dimension is smaller than the first quantitative dimension; the voice feature fine screening module 1404 is used to construct multiple voice feature sets based on the second voice feature data, and use the wrapping method to screen out a custom voice feature set for machine learning of the oral assessment model from the multiple voice feature sets, and the custom voice feature set includes: voice features of a third quantitative dimension, and the third quantitative dimension is smaller than the second quantitative dimension.

[0311] As can be seen from the above, the machine learning feature data processing device provided in the embodiment of the present disclosure, after collecting the original voice data input by multiple users for open-ended oral assessment questions through the voice data acquisition module 1401, extracts the voice features of the first quantity dimension from the original voice data through the voice feature extraction module 1402, and then uses the voice feature initial screening module 1403 to screen out the voice features of the second quantity dimension from the voice features of the first quantity dimension according to the correlation between the voice features of each dimension and the oral assessment results, and finally uses the voice feature fine screening module 1404 to construct multiple voice feature sets according to the voice features of the second quantity dimension, and uses the wrapping method to screen out the custom voice feature set used for machine learning of the oral assessment model from the multiple voice feature sets.

[0312] Through the machine learning feature data processing device provided in the embodiments of the present disclosure, speech feature data suitable for training a spoken language assessment model can be screened out, so that the trained spoken language assessment model can be greatly improved in both performance and efficiency.

[0313] In some embodiments, the machine learning feature data processing device provided in the embodiments of the present disclosure may also include: a model training module 1405, which is used to train the lightweight gradient boosting machine LightGBM model according to a custom speech feature set to obtain a spoken language assessment model for evaluating open-ended spoken language assessment questions.

[0314] In some embodiments, the machine learning feature data processing device provided in the embodiments of the present disclosure may also include: an oral evaluation module 1406, which is used to obtain the speech data to be evaluated input by the target user for an open-ended oral evaluation question type; extract corresponding custom speech features from the speech data to be evaluated according to the custom speech feature set; input the extracted custom speech feature set into a pre-trained oral evaluation model, and output the oral evaluation results of the target user for the open-ended oral evaluation question type.

[0315] In some embodiments, the machine learning feature data processing device provided in the embodiments of the present disclosure may also include: a speech recognition module 1407, which is used to perform speech recognition on the speech data to be evaluated based on a predefined speech recognition system to obtain speech recognition results, pronunciation evaluation results, and fluency evaluation results, wherein the speech recognition results are used to extract custom speech features.

[0316] In some embodiments, the speech recognition system may include: an acoustic model module, a pronunciation dictionary module, a topic language model module, and a decoder module.

[0317] In some embodiments, the above-mentioned custom speech feature set includes the following types of speech features: vocabulary features, text similarity features, word vector semantic similarity features and pronunciation features.

[0318] It should be noted that the examples and application scenarios implemented by the modules in the above-mentioned device embodiment are the same as those of the corresponding steps in the method embodiment, but are not limited to the contents disclosed in the above-mentioned method embodiment. It should be noted that the above-mentioned modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0319] Those skilled in the art will appreciate that various aspects of the present disclosure may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation that combines hardware and software aspects, which may be collectively referred to herein as a "circuit," "module," or "system."

[0320] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present disclosure, the electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned machine learning feature data processing methods by executing the executable instructions. Since the principle of solving the problem in the electronic device embodiment is similar to that in the above-mentioned method embodiment, the implementation of the electronic device embodiment can refer to the implementation of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0321] Refer to the following Fig.15 15 to describe the electronic device 1500 according to this embodiment of the present disclosure. Fig.15 The electronic device 1500 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0322] like Fig.15 As shown, the electronic device 1500 is in the form of a general computing device. The components of the electronic device 1500 may include but are not limited to: at least one processing unit 1510, at least one storage unit 1520, and a bus 1530 connecting different system components (including the storage unit 1520 and the processing unit 1510).

[0323] The storage unit stores a program code, and the program code can be executed by the processing unit 1510, so that the processing unit 1510 executes the steps described in the above "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. For example, the processing unit 1510 can execute the following steps of the above method embodiment: obtaining original speech data input by multiple users for open-ended oral evaluation questions; extracting first speech feature data containing speech features of a first quantity dimension from the original speech data; according to the correlation between the speech features of each dimension and the oral evaluation results, filtering out second speech feature data containing speech features of a second quantity dimension from the first speech feature data, wherein the second quantity dimension is smaller than the first quantity dimension; constructing multiple speech feature sets based on the second speech feature data, and using the wrapping method to filter out a speech feature set containing speech features of a third quantity dimension from the multiple speech feature sets as a custom speech feature set, so as to perform machine learning on the oral evaluation model.

[0324] The storage unit 1520 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 15201 and / or a cache storage unit 15202 , and may further include a read-only storage unit (ROM) 15203 .

[0325] The storage unit 1520 may also include a program / utility 15204 having a set (at least one) of program modules 15205, such program modules 15205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0326] Bus 1530 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0327] The electronic device 1500 may also communicate with one or more external devices 1540 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1500, and / or communicate with any device that enables the electronic device 1500 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1550. Furthermore, the electronic device 1500 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 1560. As shown, the network adapter 1560 communicates with other modules of the electronic device 1500 via a bus 1530. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0328] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0329] Based on the same inventive concept, a computer-readable storage medium is also provided in the embodiment of the present disclosure, on which a computer program is stored, and when the computer program is executed by a processor, any of the above-mentioned machine learning feature data processing methods is implemented. Since the principle of solving the problem in the embodiment of the computer-readable storage medium is similar to that in the above-mentioned method embodiment, the implementation of the embodiment of the computer-readable storage medium can refer to the implementation of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0330] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0331] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0332] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0333] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0334] Based on the same inventive concept, a computer program product is also provided in the embodiments of the present disclosure, including a computer program product, including: a computer program or an instruction, which implements the machine learning feature data processing method of any one of the above method embodiments when the computer program or the instruction is executed by the processor. Since the principle of solving the problem in the computer program product embodiment is similar to that in the above method embodiment, the implementation of the computer program product embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0335] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0336] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0337] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0338] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A method for processing feature data of machine learning, characterized in that: include: Obtaining original speech data input by multiple users for open-ended oral assessment questions; Extracting first speech feature data from the original speech data, wherein the first speech feature data is used to characterize the spectrum information of the original speech data, and the first speech feature data includes: speech features of a first quantity dimension; According to the correlation between the speech feature of each dimension and the oral evaluation result, second speech feature data is selected from the first speech feature data, wherein the second speech feature data includes: speech features of a second number dimension, the second number dimension is smaller than the first number dimension; Based on the second speech feature data, multiple speech feature sets are constructed, and a custom speech feature set for machine learning of a spoken language evaluation model is screened out from the multiple speech feature sets using a wrapping method, wherein the custom speech feature set includes: speech features of a third quantitative dimension, wherein the third quantitative dimension is smaller than the second quantitative dimension.

2. The method for processing feature data of machine learning according to claim 1, characterized in that: After constructing a plurality of speech feature sets according to the second speech feature data, and selecting a custom speech feature set for machine learning of a spoken language evaluation model from the plurality of speech feature sets by using a wrapping method, the method further includes: According to the custom speech feature set, a lightweight gradient boosting machine LightGBM model is trained to obtain a spoken language evaluation model for evaluating open-ended spoken language evaluation questions.

3. The method for processing feature data of machine learning according to claim 2, characterized in that: After training the lightweight gradient boosting machine LightGBM model according to the custom speech feature set to obtain a spoken language evaluation model for evaluating open-ended spoken language evaluation questions, the method further includes: Acquire the speech data to be evaluated input by the target user for the open-ended oral assessment question type; According to the custom speech feature set, extracting corresponding custom speech features from the speech data to be evaluated; The extracted custom speech feature set is input into a pre-trained oral evaluation model, and the oral evaluation result of the target user for the open-ended oral evaluation question type is output.

4. The method for processing feature data of machine learning according to claim 3, characterized in that: After obtaining the speech data to be evaluated input by the target user for the open-ended oral evaluation question type, the method further includes: Based on a predefined speech recognition system, speech recognition is performed on the speech data to be evaluated to obtain a speech recognition result, a pronunciation evaluation result, and a fluency evaluation result, wherein the speech recognition result is used to extract the custom speech feature.

5. The method for processing feature data of machine learning according to claim 4, characterized in that: The speech recognition system includes: an acoustic model module, a pronunciation dictionary module, a topic language model module and a decoder module.

6. The method for processing feature data for machine learning according to any one of claims 1 to 5, characterized in that: The custom speech feature set includes the following types of speech features: vocabulary features, text similarity features, word vector semantic similarity features and pronunciation features.

7. A machine learning feature data processing device, characterized in that: include: A voice data acquisition module is used to acquire original voice data input by multiple users for open-ended oral assessment questions; A speech feature extraction module, used to extract first speech feature data from the original speech data, wherein the first speech feature data is used to characterize the spectrum information of the original speech data, and the first speech feature data includes: speech features of a first quantity dimension; A speech feature preliminary screening module, configured to screen out second speech feature data from the first speech feature data according to the correlation between the speech feature of each dimension and the oral evaluation result, wherein the second speech feature data comprises: speech features of a second number dimension, the second number dimension being smaller than the first number dimension; A speech feature fine screening module is used to construct multiple speech feature sets based on the second speech feature data, and use a wrapping method to screen out a custom speech feature set from the multiple speech feature sets for machine learning of the oral evaluation model, wherein the custom speech feature set includes: speech features of a third quantitative dimension, wherein the third quantitative dimension is smaller than the second quantitative dimension.

8. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the machine learning feature data processing method described in any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the machine learning feature data processing method described in any one of claims 1 to 6 is implemented.

10. A computer program product comprising: A computer program or instruction, characterized in that when the computer program or instruction is executed by a processor, it implements the machine learning feature data processing method described in any one of claims 1 to 6.

Images