A method and system for automatically evaluating online tutoring quality based on speech features

Abstract

This invention provides a method and system for automatically evaluating online tutoring quality based on speech features. The method includes: determining a trained tutoring quality evaluation model, comprising a first-layer prediction model and a second-layer prediction model; the tutoring quality evaluation model is used to evaluate the quality of classroom tutoring based on classroom tutoring audio, and is trained based on the audio information of the teacher's and / or student's speech during classroom tutoring; the first-layer prediction model includes a random forest sub-model, a long short-term memory network sub-model, and a feedforward neural network sub-model; the second-layer prediction model fits the outputs of the three sub-models in a segmented manner, learning the high-order features of the classroom tutoring audio, so as to fuse the outputs of the three sub-models to obtain the final classroom tutoring quality evaluation result; and inputting the classroom tutoring audio into the trained tutoring quality evaluation model to evaluate the quality of classroom tutoring online. This invention utilizes machine learning to automatically evaluate classroom tutoring quality, saving time and increasing efficiency.

Description

Technical Field

[0001] This invention belongs to the field of tutoring quality assessment, and more specifically, relates to a method and system for automatically assessing online tutoring quality based on voice features. Background Technology

[0002] Currently, various regions are comprehensively reducing the total amount and duration of homework, improving the level of after-school services, and vigorously enhancing the quality of education and teaching. On the other hand, they are adhering to strict governance, comprehensively regulating off-campus training activities, and greatly reducing the number of subject-based off-campus training institutions.

[0003] Therefore, some cities have launched free one-on-one tutoring, providing online tutoring services to middle school students outside of class time through various means using internet technology and platforms to address their learning difficulties. Teachers actively participate in extracurricular online Q&A and tutoring services, which not only helps to facilitate the online flow of high-quality teacher resources, allowing more students to enjoy these resources, but also enriches the supply of personalized education and promotes educational equity.

[0004] One-on-one tutoring is among the most effective learning methods, but one way to ensure the quality of online tutoring is to employ human evaluators. These experts rate teachers based on pre-agreed criteria and provide assessments. However, this approach presents several challenges. First, the complexity of online tutoring means that defining evaluation criteria is a significant challenge. Second, even after correctly defining these criteria, human evaluators face further challenges in implementing them. The validity and reliability of the criteria may be interpreted differently by each evaluator. Third, even if we can accurately define the success criteria for online tutoring courses and assume that evaluators can objectively and effectively implement them, this evaluation process still relies heavily on human resources, which are expensive, scarce, and difficult to scale. Furthermore, some studies evaluate tutoring through videos or audio, but these often focus on the content of the lessons themselves, neglecting the raw data from the lessons. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the purpose of this invention is to provide a method and system for automatically evaluating the quality of online tutoring based on voice features, which aims to solve the problem that existing classroom tutoring quality evaluation methods rely heavily on manpower and are difficult to scale.

[0006] To achieve the above objectives, in a first aspect, the present invention provides a method for automatically evaluating the quality of online tutoring based on speech features, comprising the following steps:

[0007] A well-trained tutoring quality assessment model is defined, comprising a first-layer prediction model and a second-layer prediction model. This model is used to assess the quality of classroom tutoring based on audio recordings of teacher and / or student speech during tutoring sessions. The first-layer prediction model includes a random forest sub-model, a long short-term memory network sub-model, and a feedforward neural network sub-model. The random forest sub-model is trained based on the temporal features of teacher and student audio, the long short-term memory network sub-model is trained based on features extracted from non-silent teacher audio using OpenSMILE, and the feedforward neural network sub-model is trained based on the spectrograms of teacher and student audio. The second-layer prediction model is trained based on the training outputs of the three sub-models in the first-layer model. The outputs of the three sub-models are fitted using a segmented approach to learn higher-order features of the classroom tutoring audio, allowing the final classroom tutoring quality assessment result to be obtained by fusing the outputs of the three sub-models.

[0008] The audio of classroom tutoring is input into a trained tutoring quality assessment model to evaluate the quality of classroom tutoring online; the audio of classroom tutoring includes audio of teachers and students speaking during classroom tutoring.

[0009] In an optional example, the random forest sub-model is trained based on the temporal features of teacher and student audio, specifically:

[0010] The temporal characteristics of the teacher and student audio include: total audio time, total teacher speaking time, total student speaking time, the proportion of student speaking time to teacher speaking time, number of times the teacher or student is silent, average teacher speaking time, average student speaking time, median of the teacher speaking sequence, median of the student speaking sequence, one-quarter of the teacher speaking sequence, one-quarter of the student speaking sequence, three-quarters of the teacher speaking sequence, three-quarters of the student speaking sequence, variance of the teacher speaking sequence, variance of the student speaking sequence, and frequency of teacher-student interaction.

[0011] The time features and corresponding label values ​​of the pre-acquired classroom tutoring audio are input into the random forest sub-model to train the random forest sub-model.

[0012] In an optional example, the Long Short-Term Memory (LSTM) network sub-model is trained based on feature information extracted from the teacher's non-silent audio using openSMILE, specifically as follows:

[0013] Separate the teacher and student audio from the pre-acquired classroom tutoring audio, define the audio in the teacher's audio that pauses for more than a preset time as silent audio, delete the silent audio in the teacher's audio, and obtain the non-silent teacher audio.

[0014] Based on openSMILE, emotion-related features were extracted from non-silent audio of teachers, and first-order difference was performed on the extracted features to obtain multiple features in multiple dimensions.

[0015] The obtained multi-dimensional features and their corresponding label values ​​are input into the Long Short-Term Memory (LSTM) sub-model to train the LSM sub-model.

[0016] In an optional example, the feedforward neural network sub-model is trained based on the spectrograms of teacher and student audio, specifically:

[0017] The classroom tutoring audio is pre-sampled using a preset window and preset frequency shift, and the tag values ​​of the sampled classroom tutoring audio are obtained;

[0018] Perform a Fast Fourier Transform on each window of the classroom tutoring audio to transform the audio from the time domain to the frequency domain, then convert the frequency domain to a logarithmic scale, and convert the audio amplitude to decibels to obtain the spectrum of the classroom tutoring audio.

[0019] Mapping the frequencies of the spectrum to the Mel scale yields the Mel spectrum.

[0020] The Mel spectrum and its corresponding label values ​​are input into the feedforward neural network sub-model to train the feedforward neural network sub-model.

[0021] In an optional example, the second-layer prediction model is an MLR linear model, and its training process is as follows:

[0022] The three prediction results obtained from the first-layer prediction model are divided into three different regions by a piecewise function. Then, the prediction results of the three regions are weighted and combined by a fitting function, and the classroom tutoring quality assessment result corresponding to the weighted prediction result is determined by an activation function.

[0023] Secondly, this invention provides a system for automatically evaluating the quality of online tutoring based on voice features, comprising:

[0024] A quality assessment model determination unit is used to determine a trained tutoring quality assessment model, which includes a first-layer prediction model and a second-layer prediction model. The tutoring quality assessment model is used to assess the quality of classroom tutoring based on classroom tutoring audio. It is trained based on the audio information of the teacher and / or student speaking during classroom tutoring. The first-layer prediction model includes a random forest sub-model, a long short-term memory network sub-model, and a feedforward neural network sub-model. The random forest sub-model is trained based on the temporal features of the teacher and student audio; the long short-term memory network sub-model is trained based on the feature information extracted from the teacher's non-silent audio using OpenSMILE; and the feedforward neural network sub-model is trained based on the spectrograms of the teacher and student audio. The second-layer prediction model is trained based on the training outputs of the three sub-models in the first-layer prediction model. The outputs of the three sub-models are fitted using a segmented approach to learn the higher-order features of the classroom tutoring audio, so that the outputs of the three sub-models can be fused to obtain the final classroom tutoring quality assessment result.

[0025] The classroom tutoring quality assessment unit is used to input classroom tutoring audio into a trained tutoring quality assessment model to evaluate the quality of classroom tutoring online; the classroom tutoring audio includes audio of teachers and students speaking during classroom tutoring.

[0026] In an optional example, the system also includes: a first training unit;

[0027] The temporal characteristics of the teacher and student audio include: total audio time, total teacher speaking time, total student speaking time, the proportion of student speaking time to teacher speaking time, number of times the teacher or student is silent, average teacher speaking time, average student speaking time, median of the teacher speaking sequence, median of the student speaking sequence, one-quarter of the teacher speaking sequence, one-quarter of the student speaking sequence, three-quarters of the teacher speaking sequence, three-quarters of the student speaking sequence, variance of the teacher speaking sequence, variance of the student speaking sequence, and frequency of teacher-student interaction.

[0028] The first training unit is used to extract the temporal features of the pre-acquired classroom tutoring audio, and input the temporal features and corresponding label values ​​into the random forest sub-model to train the random forest sub-model.

[0029] In an optional example, the system further includes: a second training unit for separating the teacher's and student's audio from pre-acquired classroom tutoring audio, defining audio in the teacher's audio that pauses for more than a preset time as silent audio, deleting silent audio from the teacher's audio to obtain non-silent teacher audio; extracting emotion-related features from the non-silent teacher audio based on openSMILE, and performing first-order differencing on the extracted features to obtain multiple features in multiple dimensions; and inputting the obtained multiple features in multiple dimensions and their corresponding label values ​​into the long short-term memory network sub-model to train the long short-term memory network sub-model.

[0030] In an optional example, the system further includes: a third training unit, configured to pre-sample the classroom tutoring audio with a preset window and a preset frequency shift, and obtain the label values ​​of the sampled classroom tutoring audio; perform a fast Fourier transform on each window of the classroom tutoring audio to convert the classroom tutoring audio from the time domain to the frequency domain, convert the frequency domain to a logarithmic scale, convert the amplitude of the audio to decibels, and obtain a spectrogram of the classroom tutoring audio; map the frequency of the spectrogram to the Mel scale to obtain a Mel spectrogram; and input the Mel spectrogram and the corresponding label values ​​into a feedforward neural network sub-model to train the feedforward neural network sub-model.

[0031] In an optional example, the system further includes: a fourth training unit, which divides the three prediction results obtained from the first-layer prediction model into three different regions through a piecewise function, then weights and combines the prediction results of the three regions through a fitting function, and determines the classroom tutoring quality assessment result corresponding to the weighted prediction result through an activation function.

[0032] Thirdly, this application provides an electronic device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to perform the method described in the first aspect or any possible example of the first aspect.

[0033] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to perform the methods described in the first aspect or any possible example of the first aspect.

[0034] Fifthly, this application provides a computer program product that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible example of the first aspect.

[0035] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0036] This invention provides a method and system for automatically evaluating the quality of online tutoring based on voice features. It changes the traditional method of evaluating online tutoring quality by employing human evaluators who score teachers according to pre-agreed standards. However, this method has various problems. Using machine learning for automatic evaluation is time-saving and efficient. Furthermore, because the method focuses on audio analysis for teaching, it can be applied to different subjects without disciplinary limitations.

[0037] This invention provides a method and system for automatically evaluating the quality of online tutoring based on voice features. By utilizing machine learning of evaluation criteria, the effectiveness and reliability of the evaluation criteria are addressed, and the original subjective factors are eliminated. The evaluation criteria can be implemented objectively and effectively without relying on a large amount of manpower, which is expensive, scarce, and difficult to scale.

[0038] This invention provides a method and system for automatically evaluating the quality of online tutoring based on speech features. It utilizes ensemble learning, combining multiple models to improve classification performance, and improves decision accuracy through group decision-making, resulting in better performance than a single model.

[0039] This invention provides a method and system for automatically evaluating the quality of online tutoring based on speech features. It utilizes multiple methods to extract audio features, overcoming the limitations of single-method approaches. This allows for the extraction of more useful information and considers multiple perspectives. It extracts not only image features but also sound features and considers speech features, establishing 16 representative features. While some scholars have researched methods to monitor the quality of online one-on-one elementary school math tutoring, they only extract seven dimensions of speech features: appropriate pauses after asking questions, context-based questions / answers or examples, prompts, checking and explaining actions / problems, proactive planning of actions, enlightenment-style self-correction, and writing down discussion content on a virtual classroom environment (VCE) interface. This approach does not consider enough dimensions. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the method for automatically evaluating the quality of online tutoring based on voice features provided in an embodiment of the present invention.

[0041] Figure 2 This is a schematic diagram of the entire process of the audio analysis model provided in the embodiments of the present invention.

[0042] Figure 3 This is a schematic diagram of integrated learning provided in an embodiment of the present invention.

[0043] Figure 4 This is a schematic diagram of classification based on audio mel spectrum provided in an embodiment of the present invention.

[0044] Figure 5This is a schematic diagram of the MLR linear model in Stacking provided in this embodiment of the invention.

[0045] Figure 6 This is a system architecture diagram of an embodiment of the present invention for automatically evaluating the quality of online tutoring based on voice features. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0047] In this invention, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. In this invention, the symbol " / " indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0048] Next, the technical solutions provided in the embodiments of this application will be described.

[0049] This invention proposes a novel approach that utilizes the speech during tutoring, converts the speech into a binary sequence, extracts the features of the sequence, and uses ensemble learning to fuse multiple models to evaluate the quality of one-on-one online tutoring.

[0050] Figure 1 This is a schematic diagram of the method for automatically evaluating online tutoring quality based on voice features, provided in an embodiment of the present invention. Figure 1 As shown, it includes the following steps:

[0051] S101, Determine the trained tutoring quality assessment model, which includes a first-layer prediction model and a second-layer prediction model. The tutoring quality assessment model is used to assess the quality of classroom tutoring based on classroom tutoring audio. It is trained based on the audio information of the teacher and / or student speaking during classroom tutoring. The first-layer prediction model includes a random forest sub-model, a long short-term memory network sub-model, and a feedforward neural network sub-model. The random forest sub-model is trained based on the temporal features of the teacher and student audio. The long short-term memory network sub-model is trained based on the feature information of the teacher's non-silent audio extracted by openSMILE. The feedforward neural network sub-model is trained based on the spectrograms of the teacher and student audio. The second-layer prediction model is trained based on the training outputs of the three sub-models in the first-layer prediction model. The outputs of the three sub-models are fitted using a segmented method to learn the high-order features of the classroom tutoring audio, so as to fuse the outputs of the three sub-models to obtain the final classroom tutoring quality assessment result.

[0052] S102, input the classroom tutoring audio into the trained tutoring quality assessment model to evaluate the quality of classroom tutoring online; the classroom tutoring audio includes audio of the teacher and students speaking during the classroom tutoring process.

[0053] Understandably, based on the audio information from the coaching sessions, the coaching process is quantified, and the quality of coaching is automatically monitored and feedback is provided. (See [link to relevant documentation]). Figure 2 As shown, this invention uses ensemble learning to complete learning tasks by constructing and combining different features and models, and improves decision-making accuracy through group decision-making. This technology can automatically evaluate the quality of one-on-one tutoring, increasing the objectivity and accuracy of evaluations, reducing manual costs, and also has positive implications for teacher training.

[0054] This invention is primarily based on the algorithmic framework of Stacking ensemble learning. Using a pre-prepared dataset, in the first-layer prediction model, S... k The base model L is obtained by training using the base learner algorithm m. m Let m = 1, 2, ..., M. For S... k Each sample x in n , using z kn To represent the base learner L m Its prediction. The predicted value y corresponding to the sample. n The new result z obtained from the predictions of each base learner kn Together they form a new sample S ′ S ′ ={(y n ,z 1n ,z 2n ,...,z kn The newly generated S(n) = 1, 2, ..., N. ′ It will be used to train the meta-learner for multiple linear regression (MLR) prediction. See also Figure 3 As shown, the newly generated S is predicted by the second-layer prediction model. ′ L was obtained through training ′ In this invention, three different feature sets are obtained through the extraction of time features, sound features, and MEL (Mean Elastic Array) spectrogram features. These sets are input into the first-layer prediction model, where three classification models—random forest, long short-term memory, and CNN—are trained. The resulting prediction is the output of the first-layer model. This prediction result is then input into the second-layer model, and trained using the meta-learner of this layer, a multiple linear regression (MLR) model, to obtain the final prediction result. The specific steps are as follows:

[0055] Step 1: Feature Extraction and Model Prediction

[0056] Current research shows that most speech models only consider one aspect of audio features, neglecting a comprehensive approach. This invention, however, considers teacher and student speech features, vocal characteristics, and Mel-map features. Based on the characteristics of the data, these features are incorporated into an appropriate model for training, rather than a uniform model. Therefore, ensemble learning yields better training results. The specific training set and model training process are as follows:

[0057] Training set 1: Extraction of temporal features

[0058] Many studies have shown that speech length, teacher and student speech length, teacher-student speaking ratio, average conversation interval, and teacher-student interaction all affect teaching quality. However, no clear research indicates which speech features are most important. Therefore, we summarized 16 representative features from the literature, as shown in Table 1. We then used Python to calculate these features. Based on the characteristics of audio, our calculation method extracts two types of human-engineered audio data per second to protect privacy. The first is silence, indicating teacher or student speaking (0 for silence, 1 for speaking). The second refers to the audio track (since the audio is dual-track), with track 0 representing the teacher and track 1 representing the student. The training set 1 obtained through the above operations will be fed into Model 1's random forest for training.

[0059] Table 1

[0060]

[0061]

[0062] Model 1: Random Forest Algorithm

[0063] In audio classification, we used the Random Forest algorithm. Random Forest has proven to be a robust algorithm, delivering very satisfactory results and overcoming the overfitting problem. The Random Forest classifier uses a set of CARTs for prediction. First, samples are randomly selected to create a decision tree. Then, the Random Forest integrates all the classification votes from the decision trees and finally assigns the class with the most votes as the final output.

[0064] Training set 2: Extraction of vocal features

[0065] Step 1: Audio Preprocessing

[0066] There are many methods for extracting voice features, but for tutoring audio, student and teacher audio are often mixed together. For evaluating tutoring quality, we focus more on the teacher's voice features. Therefore, there is a lack of research into extracting teacher features. This invention addresses this by separating the teacher and student audio. After processing them using the Python toolkit pydub, we obtained the student's and teacher's audio separately. The teacher's audio includes conversation data and silences. Based on literature review and experiments, a teacher pause exceeding 2 seconds is defined as silence. Then, we remove silences from the teacher's audio.

[0067] Step 2: Feature extraction based on openSMILE

[0068] openSMILE (open Speech and Music Interpretation by Large SpaceExtraction) is a feature extractor for signal processing and machine learning, characterized by its high modularity and flexibility. It is widely used in speech recognition feature extraction front-ends, keyword discovery, emotion recognition in affective computing, emotion-sensitive virtual agents, and music information retrieval, chord recognition, beat tracking, and disease detection.

[0069] The IS09_emotion.conf (IS09) feature set is used, which has 384 (16×2×12) dimensional features. The 16 low-level descriptors (LLDs) obtained from the CSV file are named as follows: pcm_RMSenergy represents the root mean square energy of the signal frame, MFCC represents the Mel-frequency cepstral coefficients 1-12, Pcm_zcr represents the zero-crossing rate of the time signal, voiceProb represents the probability of sound generation calculated from the ACF, and F0 represents the fundamental frequency calculated from the cepstral spectrum. Then, first-order differencing is performed on these features.

[0070] Each feature contains 12 dimensions: max (maximum contour value), min (min contour value), range = max - min contour value, absolute position of maxPos (maximum contour value minus minimum contour value), amean (arithmetic mean of contour lines), slope of the linear approximation of linregc1 contour lines, offset of the linear approximation of linregc2 contour lines, linregerrQ quadratic error calculated as the difference between the linear approximation and the actual contour, stddev (standard deviation of contour values), skewness (third moment), and kurtosis (fourth moment). This results in a 384-dimensional feature set. The training set 2 obtained through these operations will be fed into the Long Short-Term Memory (LSTM) model 2 for training.

[0071] Model 2: Long Short-Term Memory Model

[0072] Recurrent neural networks (RNNs) have been used for speech recognition. However, RNNs rely on historical information to maintain their memory, making it difficult for them to discover and exploit long-term dependencies in the data. LSTM addresses this problem effectively.

[0073] The key features of the LSTM network are contained in the hidden layers, which consist of storage units. Furthermore, each storage unit has three gates that maintain and adjust its state: a forget gate, an input gate, and an output gate. It is suitable for modeling speech temporal signals because frame segmentation in speech emotion recognition refers to the segmentation of speech in the time dimension, and there are temporal relationships between each frame. Finally, due to the inconsistent data dimensions, we computed an average value for each audio segment.

[0074] Training set 3: MEL map feature extraction

[0075] In speech classification, the signal is often transformed into an image spectrogram, and then image classification algorithms (such as CNN) are used to classify the speech.

[0076] (1) Collect air pressure samples over time and represent the audio signal digitally. The input is sampled in a window of a certain size, and the next window is sampled with a frame shift of a certain size each time.

[0077] (2) Calculate the FFT (Fast Fourier Transform) for each window to achieve the transformation from the time domain to the frequency domain.

[0078] (3) Convert the frequency to a logarithmic scale and the amplitude to decibels to form a spectrum.

[0079] (4) Map the frequencies to the Mel scale to form the Mel spectrum.

[0080] The MEL graph training set 3 of the audio is fed into the Model 3 CNN network for classification.

[0081] Model 3: CNN Algorithm Classification

[0082] The CNN algorithm is a feedforward neural network that consists of several convolutional layers, max-pooling layers, and fully connected layers. See also Figure 4 As shown, the input layer takes two-dimensional features from the Mel spectrum. The convolutional layer consists of several convolutional kernels to extract different features from the input. A max-pooling layer is used to reduce data dimensionality and maintain important information. These features are then passed to the next layer via an activation function. Finally, a fully connected layer outputs a classification using a softmax function.

[0083] Step 2: MLR linear model prediction

[0084] See Figure 5 As shown, the MLR model fits the data using a piecewise approach, and compared to the LR model, it can learn higher-order feature combinations. The prediction result x = (result 1, result 2, result 3) obtained in step 1 is first processed through... The current feature is divided into m different regions (m is a hyperparameter), and then... The function makes predictions. Then, the predictions for this use case across all m regions are "weighted and combined". Finally, these two parts are merged together using the activation function g().

[0085] The specific expression is as follows:

[0086]

[0087] The parameters of the model are σ(·) is a piecewise function with parameters {u1,…,u m};η(·) fitting function, whose parameters are {w1,…,w m}; u and w are both d-dimensional vectors, where d is the dimension of the features, meaning each feature has two corresponding weights. g can be understood as an activation function, which transforms the model result into the required form, such as probabilities in classification problems.

[0088] In one specific embodiment, this invention uses online tutoring audio data collected from the SCOOT tool, totaling 352 tutoring audio files, most of which are between half an hour and an hour long. After manual encoding, the audio data is divided into high-quality and low-quality categories based on tutoring quality. There are 141 high-quality data files and 211 low-quality data files, as shown in Table 2 below.

[0089] Table 2

[0090]

[0091] The above experiments demonstrate that the method proposed in this invention has certain effectiveness and can, to some extent, replace manual evaluation of teaching quality.

[0092] Figure 6 This is a system architecture diagram for automatically evaluating online tutoring quality based on voice features, provided in an embodiment of the present invention. (For example...) Figure 6 As shown, it includes:

[0093] The quality assessment model determination unit 610 is used to determine the trained tutoring quality assessment model, which includes a first-layer prediction model and a second-layer prediction model. The tutoring quality assessment model is used to assess the quality of classroom tutoring based on classroom tutoring audio. It is trained based on the audio information of the teacher and / or student speaking during classroom tutoring. The first-layer prediction model includes a random forest sub-model, a long short-term memory network sub-model, and a feedforward neural network sub-model. The random forest sub-model is trained based on the temporal features of the teacher and student audio. The long short-term memory network sub-model is trained based on the feature information of the teacher's non-silent audio extracted by openSMILE. The feedforward neural network sub-model is trained based on the spectrograms of the teacher and student audio. The second-layer prediction model is trained based on the training outputs of the three sub-models in the first-layer prediction model. The outputs of the three sub-models are fitted in a segmented manner to learn the high-order features of the classroom tutoring audio, so as to fuse the outputs of the three sub-models to obtain the final classroom tutoring quality assessment result.

[0094] The classroom tutoring quality assessment unit 620 is used to input classroom tutoring audio into a trained tutoring quality assessment model to assess the quality of classroom tutoring online; the classroom tutoring audio includes audio of teachers and students speaking during classroom tutoring.

[0095] The first training unit 630 is used to extract the time features of the pre-acquired classroom tutoring audio, and input the time features and corresponding label values ​​into the random forest sub-model to train the random forest sub-model.

[0096] The second training unit 640 is used to separate the teacher and student audio in the pre-acquired classroom tutoring audio, define the audio in the teacher's audio that pauses for more than a preset time as silent audio, delete the silent audio in the teacher's audio to obtain non-silent teacher audio; extract emotion-related features from the non-silent teacher audio based on openSMILE, and perform first-order difference on the extracted features to obtain multiple features in multiple dimensions; and input the obtained multiple features in multiple dimensions and their corresponding label values ​​into the long short-term memory network sub-model to train the long short-term memory network sub-model.

[0097] The third training unit 650 is used to pre-sample the classroom tutoring audio with a preset window and a preset frequency shift, and obtain the label value of the sampled classroom tutoring audio; perform a fast Fourier transform on each window of the classroom tutoring audio to transform the classroom tutoring audio from the time domain to the frequency domain, and convert the frequency domain to a logarithmic scale, convert the audio amplitude to decibels, and obtain the spectrum of the classroom tutoring audio; map the frequency of the spectrum to the Mel scale to obtain the Mel spectrum; and input the Mel spectrum and the corresponding label value into the feedforward neural network sub-model to train the feedforward neural network sub-model.

[0098] The fourth training unit 660 is used to divide the three prediction results obtained from the first-layer prediction model into three different regions through a piecewise function, then use a fitting function to weight and combine the prediction results of the three regions, and use an activation function to determine the classroom tutoring quality assessment result corresponding to the weighted prediction result.

[0099] It should be understood that the above system is used to execute the methods in the above embodiments. The corresponding program modules in the system are similar in implementation principle and technical effect to those described in the above methods. The working process of the system can be referred to the corresponding process in the above methods, and will not be repeated here.

[0100] Based on the methods described in the above embodiments, this application provides an electronic device. The device may include at least one memory for storing a program and at least one processor for executing the program stored in the memory. When the program stored in the memory is executed, the processor performs the methods described in the above embodiments.

[0101] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0102] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0103] It is understood that the processor in the embodiments of this application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor may be a microprocessor or any conventional processor.

[0104] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0105] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0106] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

[0107] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for automatically evaluating online tutoring quality based on speech features, characterized in that, The method comprises the following steps: determining a trained tutoring quality evaluation model, which comprises a first layer prediction model and a second layer prediction model; the tutoring quality evaluation model is used for evaluating the quality of classroom tutoring based on classroom tutoring audio, which is trained based on audio information of speech of teachers and / or students in the classroom tutoring process; wherein the first layer prediction model comprises a random forest sub-model, a long short-term memory network sub-model and a feedforward neural network sub-model; the random forest sub-model is trained based on time features of the audio of the teachers and the students, the long short-term memory network sub-model is trained based on feature information of non-silence audio of the teachers extracted by openSMILE, and the feedforward neural network sub-model is trained based on a frequency spectrum of the audio of the teachers and the students; the second layer prediction model is trained based on training output results of the three sub-models in the first layer prediction model, the output results of the three sub-models are fitted by segmentation, high-order features of the classroom tutoring audio are learned, and finally the output results of the three sub-models are fused to obtain the final classroom tutoring quality evaluation result; inputting the classroom tutoring audio into the trained tutoring quality evaluation model to evaluate the quality of classroom tutoring online; the classroom tutoring audio comprises audio of speech of the teachers and the students in the classroom tutoring process.

2. The method of claim 1, wherein, The random forest sub-model is trained based on time features of the audio of the teachers and the students, specifically as follows: The time features of the audio of the teachers and the students comprise total audio time, total speaking time of the teachers, total speaking time of the students, proportion of the speaking time of the students relative to the speaking time of the teachers, silence frequency of the teachers or the students, average speaking time of the teachers, average speaking time of the students, median value of the speaking sequence of the teachers, median value of the speaking sequence of the students, quarter value of the speaking sequence of the teachers, quarter value of the speaking sequence of the students, three-quarter value of the speaking sequence of the teachers, three-quarter value of the speaking sequence of the students, variance of the speaking sequence of the teachers, variance of the speaking sequence of the students and teacher-student interaction frequency; The time features of the pre-acquired classroom tutoring audio and corresponding label values are input into the random forest sub-model to train the random forest sub-model.

3. The method of claim 1, wherein, The long short-term memory network sub-model is trained based on feature information of non-silence audio of the teachers extracted by openSMILE, specifically as follows: The audio of the teachers and the students in the pre-acquired classroom tutoring audio is separated, and audio with a pause longer than a preset time in the audio of the teachers is defined as silence audio, the silence audio in the audio of the teachers is deleted, and non-silence audio of the teachers is obtained; features related to emotion in the non-silence audio of the teachers are extracted based on openSMILE, and the extracted features are subjected to first-order difference to obtain multiple features in multiple dimensions; the obtained multiple features in multiple dimensions and corresponding label values are input into the long short-term memory network sub-model to train the long short-term memory network sub-model.

4. The method of claim 1, wherein, The feedforward neural network sub-model is trained based on a frequency spectrum of the audio of the teachers and the students, specifically as follows: the classroom tutoring audio is pre-sampled with a preset window and a preset frequency shift, and label values of the sampled classroom tutoring audio are acquired; performing fast Fourier transform on each window of the tutoring audio, converting the tutoring audio from time domain to frequency domain, and converting the frequency domain to a logarithmic scale, converting the amplitude of the audio to decibels, to obtain a spectrogram of the tutoring audio; mapping the frequencies of the spectrogram to a Mel scale to obtain a Mel spectrogram; inputting the Mel spectrogram and the corresponding label value into the feedforward neural network sub-model, and training the feedforward neural network sub-model.

5. The method of claim 1, wherein, The second layer prediction model is an MLR linear model, and its training process is as follows: The three prediction results obtained by the first layer prediction model are divided into three different regions by a piecewise function, and the prediction results of the three regions are combined by a fitting function, and the prediction result after the combination is determined by an activation function to determine the corresponding classroom tutoring quality evaluation result.

6. A system for automatically evaluating the quality of online tutoring based on speech features, characterized in that, It comprises: The quality evaluation model determination unit is used to determine the trained tutoring quality evaluation model, which comprises a first layer prediction model and a second layer prediction model; the tutoring quality evaluation model is used to evaluate the classroom tutoring quality based on the classroom tutoring audio, which is trained based on the audio information of the teacher and / or student speaking in the classroom tutoring process; wherein the first layer prediction model comprises a random forest sub-model, a long short-term memory network sub-model and a feedforward neural network sub-model, the random forest sub-model is trained based on the time characteristics of the teacher and student audio, the long short-term memory network sub-model is trained based on the feature information of the teacher non-silence audio extracted by openSMILE, and the feedforward neural network sub-model is trained based on the spectrogram of the teacher and student audio; the second layer prediction model is trained based on the training output results of the three sub-models in the first layer prediction model, the output results of the three sub-models are fitted by a piecewise method, the high-order characteristics of the classroom tutoring audio are learned, and the output results of the three sub-models are fused to obtain the final classroom tutoring quality evaluation result; The classroom tutoring quality evaluation unit is used to input the classroom tutoring audio into the trained tutoring quality evaluation model to evaluate the classroom tutoring quality online; the classroom tutoring audio comprises the audio of the teacher and student speaking in the classroom tutoring process.

7. The system of claim 6, wherein, It further comprises: The first training unit; The time characteristics of the teacher and student audio include: total audio time, total teacher speaking time, total student speaking time, student speaking time relative to teacher speaking time, teacher or student silence frequency, average teacher speaking time, average student speaking time, teacher speaking sequence median value, student speaking sequence median value, teacher speaking sequence quarter value, student speaking sequence quarter value, teacher speaking sequence three-quarters value, student speaking sequence three-quarters value, teacher speaking sequence variance, student speaking sequence variance and teacher-student interaction frequency; The first training unit is used to extract the time characteristics of the pre-acquired classroom tutoring audio, and input the time characteristics and the corresponding label value into the random forest sub-model to train the random forest sub-model.

8. The system of claim 6, wherein, It further comprises: The second training unit is configured to separate the audio of the teacher and the student in the pre-acquired tutoring audio, define audio with a pause longer than a preset time in the audio of the teacher as silent audio, delete the silent audio in the audio of the teacher, and obtain non-silent audio of the teacher; extract features related to emotion from the non-silent audio of the teacher based on openSMILE, and perform first-order difference on the extracted features to obtain multiple features in multiple dimensions; and input the obtained multiple features in multiple dimensions and corresponding label values into a long short-term memory network submodel, and train the long short-term memory network submodel.

9. The system of claim 6, wherein, Further comprising: The third training unit is configured to pre-sample the tutoring audio with a preset window and a preset frequency shift, and obtain label values of the sampled tutoring audio; perform fast Fourier transform on each window of the tutoring audio, convert the tutoring audio from time domain to frequency domain, convert the frequency domain to logarithmic scale, convert the amplitude of the audio to decibel, and obtain a spectrogram of the tutoring audio; map the frequency of the spectrogram to Mel scale to obtain a Mel spectrogram; and input the Mel spectrogram and corresponding label values into a feedforward neural network submodel, and train the feedforward neural network submodel.

10. The system of claim 6, wherein, Further comprising: The fourth training unit is configured to divide the three prediction results obtained by the first layer prediction model into three different regions through a segmentation function, combine the prediction results of the three regions through a fitting function, and determine the tutoring quality evaluation result corresponding to the prediction result after the weighted combination through an activation function.

Images