English teaching auxiliary system and method based on human-computer interaction

By using a human-computer interaction-based English teaching system, which employs speech recognition and deep learning technologies to assess students' spoken pronunciation and generate standard pronunciation reference audio, the system addresses the lack of personalized guidance in traditional teaching and enhances students' spoken English and intercultural communication skills.

CN119025067AInactive Publication Date: 2024-11-26XIANGNAN UNIV
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411125179.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-16
Publication Date
2024-11-26
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Traditional English teaching neglects the cultivation of practical language application skills and intercultural communication skills. Limited teacher resources make it difficult to provide personalized guidance, and students have limited self-assessment and pronunciation correction abilities, resulting in uneven teaching outcomes.

Method used

Audio recordings of students' spoken English practice are collected via microphone, and text transcription is generated using speech recognition technology. By combining natural language processing and deep learning technologies, standard English pronunciation reference audio is generated, and acoustic features are extracted and compared to intelligently evaluate students' spoken pronunciation and provide personalized guidance.

Benefits of technology

It enables intelligent assessment and personalized guidance of students' English speaking skills, improving their oral practice ability and intercultural communication skills.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025067A_ABST
    Figure CN119025067A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of English teaching, and particularly discloses an English teaching auxiliary system and method based on man-machine interaction, and the system collects the oral practice audio of a student through a microphone, generates text transcription through a voice recognition technology, carries out the semantic understanding of a text through a natural language processing technology, and improves the teaching efficiency. A standard English pronunciation reference audio is generated based on the text semantic features, and acoustic feature extraction and comparative analysis are performed on the oral practice audio of the student and the generated standard English pronunciation reference audio by using a deep learning technology, so that whether the oral pronunciation of the student is accurate or not is intelligently evaluated; and an audio reference is provided for non-standard pronunciation. Therefore, intelligent evaluation and personalized guidance of the oral English ability of the student can be realized, and the oral practical ability and the cross-cultural communication ability of the student are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of English teaching, and more specifically, to an English teaching auxiliary system and method based on human-computer interaction. Background Art

[0002] With the acceleration of globalization and the increasing frequency of international exchanges, the importance of English as an international common language has become increasingly prominent. However, in the process of English teaching, traditional English teaching methods often focus on grammar explanation and vocabulary memorization, neglecting the cultivation of practical language application ability and cross-cultural communication ability. At the same time, due to limited teacher resources, especially in large-class teaching environments, it is difficult for teachers to pay attention to the learning progress and difficulties of each student, and it is impossible to provide personalized, real-time pronunciation guidance and feedback to each student, resulting in uneven teaching results. In addition, students' ability to self-assess and correct pronunciation is also relatively limited. Due to the lack of effective practice and evaluation tools, the improvement of their English listening and speaking skills is limited.

[0003] Therefore, an English teaching auxiliary system and method based on human-computer interaction is expected. Summary of the invention

[0004] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides an English teaching auxiliary system and method based on human-computer interaction, which collects students' oral practice audio through a microphone, generates text transcription using speech recognition technology, and uses natural language processing technology to understand the text semantically, and generates standard English pronunciation reference audio based on text semantic features, and then uses deep learning technology to extract and compare the acoustic features of the students' oral practice audio and the generated standard English pronunciation reference audio, so as to intelligently evaluate whether the students' oral pronunciation is accurate, and provide audio reference for non-standard pronunciation. In this way, intelligent evaluation and personalized guidance of students' English speaking ability can be achieved, and students' oral practice ability and cross-cultural communication ability can be improved.

[0005] Accordingly, according to one aspect of the present application, an English teaching auxiliary system based on human-computer interaction is provided, which includes:

[0006] The oral practice audio collection module is used to collect students' oral practice audio data through a microphone;

[0007] A standard pronunciation generation module, used to generate standard English pronunciation reference audio based on the spoken language practice audio data;

[0008] An audio spectrum feature extraction module, used to extract the audio spectrum features of the spoken language practice audio data and the standard English pronunciation reference audio data to obtain a spoken language practice audio spectrum feature vector and a reference audio spectrum feature vector;

[0009] A comparative analysis module, used for determining an evaluation result of the student's oral pronunciation based on a feature difference between the oral practice audio spectrum feature vector and the reference audio spectrum feature vector;

[0010] The pronunciation teaching module is used to output the standard English pronunciation reference audio in response to the evaluation result that the student's spoken pronunciation does not meet the predetermined standard.

[0011] In the above-mentioned English teaching auxiliary system based on human-computer interaction, the standard pronunciation generation module includes: a text conversion unit, which is used to perform speech recognition on the oral practice audio data to obtain a student oral practice text; a text semantic encoding unit, which is used to perform semantic encoding on the student oral practice text to obtain a semantic feature vector of the student oral practice text; and a reference audio generation unit, which is used to pass the semantic feature vector of the student oral practice text through a standard English pronunciation generation model based on a generative adversarial network to obtain the standard English pronunciation reference audio.

[0012] In the above-mentioned English teaching auxiliary system based on human-computer interaction, the text semantic encoding unit is used to: pass the student oral practice text through a semantic encoder based on the Bert model to obtain the semantic feature vector of the student oral practice text.

[0013] In the above-mentioned English teaching auxiliary system based on human-computer interaction, the audio spectrum feature extraction module includes: a mel-spectrogram calculation unit, which is used to respectively calculate the mel-spectrograms of the oral practice audio data and the standard English pronunciation reference audio to obtain a oral practice audio mel-spectrogram and a reference audio mel-spectrogram; an acoustic feature extraction unit, which is used to respectively pass the oral practice audio mel-spectrogram and the reference audio mel-spectrogram through an acoustic feature extractor based on a convolutional neural network model to obtain the oral practice audio spectrum feature vector and the reference audio spectrum feature vector.

[0014] In the above-mentioned English teaching auxiliary system based on human-computer interaction, the comparative analysis module includes: a feature difference calculation unit, which is used to calculate the difference feature vector between the oral practice audio spectrum feature vector and the reference audio spectrum feature vector as the oral practice pronunciation standard characterization feature vector; a feature distribution optimization unit, which is used to optimize the feature distribution of the oral practice pronunciation standard characterization feature vector to obtain an optimized oral practice pronunciation standard characterization feature vector; an evaluation result generation unit, which is used to pass the optimized oral practice pronunciation standard characterization feature vector through a pronunciation standard evaluator based on a classifier to obtain the evaluation result, and the evaluation result is used to indicate whether the student's oral pronunciation meets the predetermined standard.

[0015] In the above-mentioned English teaching auxiliary system based on human-computer interaction, the feature distribution optimization unit is used to: perform classification error viewpoint measurement based on backward machine learning on the feature vector representing the pronunciation standard of the oral practice to obtain the optimized feature vector representing the pronunciation standard of the oral practice.

[0016] In the above-mentioned English teaching auxiliary system based on human-computer interaction, the feature distribution optimization unit includes: a probabilistic subunit, which is used to perform probabilistic activation based on the Sigmoid function on the feature vector representing the pronunciation standard of the oral practice to obtain a probabilistic feature vector representing the pronunciation standard of the oral practice; a label probability generation subunit, which is used to input the feature vector representing the pronunciation standard of the oral practice into the classifier to obtain an initial class label probability value; an error measurement subunit, which is used to calculate the absolute value of the difference between the feature value of each position in the probabilistic feature vector representing the pronunciation standard of the oral practice and the initial class label probability value to obtain a backward machine learning fine-grained error measurement vector; an attention optimization subunit, which is used to use the backward machine learning fine-grained error measurement The vector is used as a key vector, the characteristic vector representing the pronunciation standard of the oral practice is used as a query vector and a value vector, and the key vector, the query vector and the value vector are input into a converter structure to obtain a backward learning type attention optimized oral practice pronunciation standard representation characteristic vector; an optimization scaling factor calculation subunit is used to calculate the ratio between the sum of the backward machine learning fine-grained error metric vector and the length of the backward machine learning fine-grained error metric vector, and calculate the natural exponential function value with the ratio as the power as the optimization scaling factor; a scale scaling subunit is used to calculate the product between the optimization scaling factor and the backward learning type attention optimized oral practice pronunciation standard representation characteristic vector to obtain the optimized oral practice pronunciation standard representation characteristic vector.

[0017] According to another aspect of the present application, there is provided an English teaching auxiliary method based on human-computer interaction, which comprises:

[0018] Collect students’ oral practice audio data through microphone;

[0019] Based on the oral practice audio data, generate standard English pronunciation reference audio;

[0020] Extracting audio spectrum features of the spoken language practice audio data and the standard English pronunciation reference audio data respectively to obtain a spoken language practice audio spectrum feature vector and a reference audio spectrum feature vector;

[0021] Determining an evaluation result of the student's spoken pronunciation based on a feature difference between the spoken practice audio spectrum feature vector and the reference audio spectrum feature vector;

[0022] In response to the evaluation result that the student's spoken pronunciation does not meet the predetermined standard, the standard English pronunciation reference audio is output.

[0023] Compared with the prior art, the English teaching auxiliary system and method based on human-computer interaction provided by the present application collects students' oral practice audio through a microphone, generates text transcription using speech recognition technology, and uses natural language processing technology to understand the text semantically, and generates standard English pronunciation reference audio based on the text semantic features, and then uses deep learning technology to extract and compare the acoustic features of the students' oral practice audio and the generated standard English pronunciation reference audio, so as to intelligently evaluate whether the students' oral pronunciation is accurate, and provide audio reference for non-standard pronunciation. In this way, intelligent evaluation and personalized guidance of students' English oral ability can be achieved, and students' oral practice ability and cross-cultural communication ability can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0025] Figure 1 4 is a block diagram of an English teaching auxiliary system based on human-computer interaction according to an embodiment of the present application.

[0026] Figure 2 Schematic diagram of the architecture of an English teaching auxiliary system based on human-computer interaction according to an embodiment of the present application.

[0027] Figure 3 It is a block diagram of a standard pronunciation generation module in an English teaching auxiliary system based on human-computer interaction according to an embodiment of the present application.

[0028] Figure 4 4 is a block diagram of an audio spectrum feature extraction module in an English teaching auxiliary system based on human-computer interaction according to an embodiment of the present application.

[0029] Figure 5 4 is a block diagram of a comparison and analysis module in an English teaching auxiliary system based on human-computer interaction according to an embodiment of the present application.

[0030] Figure 6 Flow chart of an English teaching auxiliary method based on human-computer interaction according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] Below, the embodiments of the present application will be described in more detail in conjunction with the accompanying drawings, and the above and other purposes, features and advantages of the present application will become more apparent. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein.

[0032] Figure 1 4 is a block diagram of an English teaching auxiliary system based on human-computer interaction according to an embodiment of the present application. Figure 2 Schematic diagram of the architecture of the English teaching auxiliary system based on human-computer interaction according to the embodiment of the present application. Figure 1 and Figure 2 As shown, according to the embodiment of the present application, the English teaching auxiliary system 100 based on human-computer interaction includes: a spoken language practice audio acquisition module 110, which is used to collect students' spoken language practice audio data through a microphone; a standard pronunciation generation module 120, which is used to generate standard English pronunciation reference audio based on the spoken language practice audio data; an audio spectrum feature extraction module 130, which is used to extract the audio spectrum features of the spoken language practice audio data and the standard English pronunciation reference audio respectively to obtain a spoken language practice audio spectrum feature vector and a reference audio spectrum feature vector; a comparative analysis module 140, which is used to determine the evaluation result of the student's spoken pronunciation based on the feature difference between the spoken language practice audio spectrum feature vector and the reference audio spectrum feature vector; a pronunciation teaching module 150, which is used to output the standard English pronunciation reference audio in response to the evaluation result that the student's spoken pronunciation does not meet the predetermined standard.

[0033] As mentioned in the above background technology, due to limited teacher resources, especially in large-class teaching, it is difficult to meet the personalized and real-time pronunciation guidance needs of each student, which affects the balance of teaching effect. In addition, students' self-assessment and pronunciation correction abilities are also relatively limited. Due to the lack of effective practice and evaluation tools, their English listening and speaking abilities are limited. In view of the above technical problems, the technical concept of this application is to collect students' oral practice audio through a microphone, generate text transcription using speech recognition technology, and use natural language processing technology to understand the text semantically, and generate standard English pronunciation reference audio based on text semantic features, and then use deep learning technology to extract acoustic features and compare and analyze the students' oral practice audio and the generated standard English pronunciation reference audio, so as to intelligently evaluate whether the students' oral pronunciation is accurate, and provide audio reference for non-standard pronunciation. In this way, intelligent evaluation and personalized guidance of students' English speaking ability can be achieved, and students' oral practice ability and cross-cultural communication ability can be improved.

[0034] In the above-mentioned English teaching auxiliary system 100 based on human-computer interaction, the oral practice audio acquisition module 110 is used to collect students' oral practice audio data through a microphone. It should be understood that the microphone can directly and in real time capture the students' oral pronunciation, reflect the students' real pronunciation, including speech speed, intonation, pronunciation accuracy, etc., and provide raw data for subsequent pronunciation analysis and teaching. By performing pronunciation analysis on the students' oral practice audio data, problems in students' pronunciation can be found in a targeted manner, and immediate feedback guidance can be provided, thereby improving teaching efficiency.

[0035] In the above-mentioned English teaching auxiliary system 100 based on human-computer interaction, the standard pronunciation generation module 120 is used to generate standard English pronunciation reference audio based on the oral practice audio data. It should be understood that by generating a standard English pronunciation reference corresponding to the oral practice audio data, a pronunciation template can be provided for students to help them understand and correct their pronunciation problems.

[0036] Figure 3 FIG. 1 is a block diagram of a standard pronunciation generation module in an English teaching auxiliary system based on human-computer interaction according to an embodiment of the present application. Figure 3 As shown, the standard pronunciation generation module 120 includes: a text conversion unit 121, which is used to perform speech recognition on the oral practice audio data to obtain a student oral practice text; a text semantic encoding unit 122, which is used to perform semantic encoding on the student oral practice text to obtain a semantic feature vector of the student oral practice text; and a reference audio generation unit 123, which is used to generate the semantic feature vector of the student oral practice text through a standard English pronunciation generation model based on a generative adversarial network to obtain the standard English pronunciation reference audio.

[0037] Specifically, the text conversion unit 121 is used for performing speech recognition on the spoken language practice audio data to obtain the student's spoken language practice text. It should be understood that by converting the student's spoken language practice audio data into text form using speech recognition technology, the voice information can be converted into processable data, and the student's pronunciation content can be captured to facilitate the generation of pronunciation reference audio. In a specific embodiment of the present application, a speech recognition model based on deep learning, such as Transformer, etc., can be used to achieve high-precision speech-to-text conversion.

[0038] Specifically, the text semantic encoding unit 122 is used for semantically encoding the student oral practice text to obtain the student oral practice text semantic feature vector. In a specific example of the present application, the processing method for semantically encoding the student oral practice text is to pass the student oral practice text through a semantic encoder based on the Bert model to obtain the student oral practice text semantic feature vector. It should be understood that BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model, and its core advantage is that it can deeply understand the semantic information of the text. In the technical solution of the present application, the student oral practice text is encoded by the BERT model, and the context information of each word in the student oral practice text can be considered at the same time, and the meaning of the word in a specific context can be understood more accurately, thereby providing rich semantic feature guidance for the generation of reference audio.

[0039] Specifically, the reference audio generation unit 123 is used to generate the standard English pronunciation reference audio by the standard English pronunciation generation model based on the adversarial generative network. It should be known to those skilled in the art that the adversarial generative network (AGN) is an advanced generation model that can learn the distribution of input feature vectors and generate new samples with similar features to the input vectors. In the technical solution of the present application, the model structure of the adversarial generative network (GANs) is used to decode and reconstruct the semantic feature vectors of the student oral practice text to generate standard English pronunciation audio that matches the input semantic features. The adversarial generative network model learns the correct pronunciation rules and intonation features of English by learning a large number of standard English pronunciation samples during the training process, so as to generate a reference audio that is consistent with the content of the student oral practice text but with standard pronunciation, as a template for student pronunciation, to help students intuitively compare and correct their pronunciation problems, and improve the accuracy and naturalness of pronunciation.

[0040] In the above-mentioned English teaching auxiliary system 100 based on human-computer interaction, the audio spectrum feature extraction module 130 is used to extract the audio spectrum features of the oral practice audio data and the standard English pronunciation reference audio to obtain the oral practice audio spectrum feature vector and the reference audio spectrum feature vector. Figure 4 FIG. 1 is a block diagram of an audio spectrum feature extraction module in an English teaching auxiliary system based on human-computer interaction according to an embodiment of the present application. Figure 4As shown, the audio spectrum feature extraction module 130 includes: a mel-spectrogram calculation unit 131, which is used to respectively calculate the mel-spectrograms of the spoken language practice audio data and the standard English pronunciation reference audio to obtain a spoken language practice audio mel-spectrogram and a reference audio mel-spectrogram; an acoustic feature extraction unit 132, which is used to respectively pass the spoken language practice audio mel-spectrogram and the reference audio mel-spectrogram through an acoustic feature extractor based on a convolutional neural network model to obtain the spoken language practice audio spectrum feature vector and the reference audio spectrum feature vector.

[0041] Specifically, the mel-spectrogram calculation unit 131 is used to calculate the mel-spectrograms of the spoken language practice audio data and the standard English pronunciation reference audio respectively to obtain the spoken language practice audio mel-spectrogram and the reference audio mel-spectrogram. It should be understood that the mel-spectrogram is a method of converting a sound signal into a frequency domain representation, which is commonly used in speech processing and recognition, and can effectively capture the spectral information and intonation features in the audio. In the technical solution of the present application, by calculating the mel-spectrograms of the spoken language practice audio data and the standard English pronunciation reference audio respectively, a complex audio signal can be converted into a structured spectral feature to facilitate subsequent feature analysis and comparison. In the calculation process of the mel-spectrogram, the audio signal can be transformed by fast Fourier transform (FFT), and filter bank processing can be performed on the mel scale to obtain a mel-spectrogram reflecting the audio spectrum structure.

[0042] Specifically, the acoustic feature extraction unit 132 is used to process the spoken language practice audio mel-spectrogram and the reference audio mel-spectrogram respectively through an acoustic feature extractor based on a convolutional neural network model to obtain the spoken language practice audio spectrum feature vector and the reference audio spectrum feature vector. It should be understood that convolutional neural networks (CNNs) perform well in image processing tasks and can automatically learn and extract high-level features of input data. In the technical solution of the present application, the spoken language practice audio mel-spectrogram and the reference audio mel-spectrogram are processed separately by a CNN model, and the feature extraction capability of the convolutional neural network can be utilized to extract acoustic features related to pronunciation quality from the mel-spectrogram through a sliding convolution operation, such as the start and end of a syllable, pitch changes, intensity patterns, etc., thereby obtaining a more abstract and representative spectrum feature vector, accurately describing the pronunciation characteristics of the audio, and providing an effective feature representation for subsequent pronunciation evaluation and feedback.

[0043] In the above-mentioned English teaching auxiliary system 100 based on human-computer interaction, the comparative analysis module 140 is used to determine the evaluation result of the student's oral pronunciation based on the feature difference between the oral practice audio spectrum feature vector and the reference audio spectrum feature vector. Figure 5FIG. 1 is a block diagram of a comparative analysis module in an English teaching auxiliary system based on human-computer interaction according to an embodiment of the present application. Figure 5 As shown, the comparative analysis module 140 includes: a feature difference calculation unit 141, which is used to calculate the difference feature vector between the oral practice audio spectrum feature vector and the reference audio spectrum feature vector as the oral practice pronunciation standard characterization feature vector; a feature distribution optimization unit 142, which is used to perform feature distribution optimization on the oral practice pronunciation standard characterization feature vector to obtain an optimized oral practice pronunciation standard characterization feature vector; an evaluation result generation unit 143, which is used to pass the optimized oral practice pronunciation standard characterization feature vector through a classifier-based pronunciation standard evaluator to obtain the evaluation result, and the evaluation result is used to indicate whether the student's oral pronunciation meets the predetermined standard.

[0044] Specifically, the feature difference calculation unit 141 is used to calculate the difference feature vector between the oral practice audio spectrum feature vector and the reference audio spectrum feature vector as the feature vector representing the standard of oral practice pronunciation. That is, in order to further analyze the oral pronunciation quality of students and quantify the feature difference between the oral practice audio data and the standard English pronunciation reference audio, in the technical solution of the present application, by calculating the feature difference between the oral practice audio spectrum feature vector and the reference audio spectrum feature vector, the deviation between the student's pronunciation and the standard pronunciation is intuitively reflected, providing an important basis for the evaluation of the student's pronunciation standard.

[0045] Specifically, the feature distribution optimization unit 142 is used to optimize the feature distribution of the feature vector representing the standard of pronunciation of the oral practice to obtain an optimized feature vector representing the standard of pronunciation of the oral practice. In particular, in the technical solution of the present application, it is considered that the speech recognition step may introduce noise interference, resulting in a deviation between the obtained oral practice text and the actual pronunciation of the students, thereby affecting the subsequent semantic encoding and pronunciation evaluation. In addition, the Bert model may not be able to fully capture all the semantic information of the oral practice text when performing semantic encoding, especially when the text contains slang, professional terms or expressions in a specific context. The standard English pronunciation reference audio generated by the adversarial generative network may also deviate from the actual standard pronunciation. The mel spectrogram may not be able to fully express all the acoustic features of the audio, and some subtle features that are important for pronunciation evaluation may not be fully captured. The acoustic feature extractor based on the convolutional neural network model may not be able to extract all acoustic features related to the standard of pronunciation. The above reasons will cause the feature vector representing the standard of pronunciation of the oral practice to have a class harmony offset relative to the predetermined class label of the classifier. Therefore, in order to reduce the category harmony deviation of the feature vector representing the pronunciation standard of the oral practice, in a specific example of the present application, the feature vector representing the pronunciation standard of the oral practice is subjected to classification error viewpoint measurement based on backward machine learning to obtain the optimized feature vector representing the pronunciation standard of the oral practice.

[0046] Specifically, the feature distribution optimization unit 142 includes: a probabilization subunit, which is used to perform probabilistic activation on the feature vector representing the pronunciation standard of the oral practice based on the Sigmoid function to obtain a probabilistic feature vector representing the pronunciation standard of the oral practice; a label probability generation subunit, which is used to input the feature vector representing the pronunciation standard of the oral practice into the classifier to obtain an initial class label probability value; an error measurement subunit, which is used to calculate the absolute value of the difference between the feature value of each position in the probabilistic feature vector representing the pronunciation standard of the oral practice and the initial class label probability value to obtain a backward machine learning fine-grained error measurement vector; an attention optimization subunit, which is used to use the backward machine learning fine-grained error measurement vector as a key The key vector, the query vector and the value vector are input into a converter structure to obtain a backward learning type attention optimized oral practice pronunciation standard representation feature vector; an optimization scaling factor calculation subunit is used to calculate the ratio between the sum of the backward machine learning fine-grained error metric vector and the length of the backward machine learning fine-grained error metric vector, and calculate the natural exponential function value with the ratio as the power as the optimization scaling factor; a scale scaling subunit is used to calculate the product between the optimization scaling factor and the backward learning type attention optimized oral practice pronunciation standard representation feature vector to obtain the optimized oral practice pronunciation standard representation feature vector.

[0047] That is, the classification error viewpoint measurement based on backward machine learning is performed on the feature vector representing the pronunciation standard of the oral practice using the following feature optimization formula to obtain the feature vector representing the pronunciation standard of the optimized oral practice, wherein the feature optimization formula is:

[0048]

[0049] Among them, V c represents the probabilistic oral practice pronunciation standard characterization feature vector obtained after probabilistic activation of the oral practice pronunciation standard characterization feature vector, P represents the initial class label probability value, Indicates subtraction by position, V e represents the fine-grained error metric vector for backward machine learning, v i represents the eigenvalue of the ith position of the backward machine learning fine-grained error metric vector, L represents the length of the backward machine learning fine-grained error metric vector, (·) T represents the transpose of a vector, represents vector multiplication, Softmax represents the normalized exponential function, ⊙ represents point multiplication by position, V c ' represents the characteristic vector representing the standard degree of pronunciation of the optimized oral practice.

[0050] Based on this, in the technical solution of the present application, the feature vector representing the standard degree of pronunciation of oral practice is subjected to classification error viewpoint measurement based on backward machine learning, which first implements the probabilistic activation based on the Sigmoid function on the feature vector representing the standard degree of pronunciation of oral practice to obtain the probabilistic feature vector representing the standard degree of pronunciation of oral practice, and inputs the feature vector representing the standard degree of pronunciation of oral practice into the classifier to obtain the initial class label probability value. Then, the error between the probabilistic feature vector representing the standard degree of pronunciation of oral practice and the initial class label probability value is calculated through backward error analysis to generate a fine-grained error measurement vector, which quantifies the viewpoint measurement contribution of each feature position to the classification error.

[0051] Then, the backward machine learning fine-grained error metric vector is used as a key vector, and the oral practice pronunciation standard characterization feature vector is used as a query vector and a value vector, and the key vector, the query vector and the value vector are input into a transformer structure to perform error metric-based attention-based backward machine learning on the oral practice pronunciation standard characterization feature vector through a Transformer-like architecture. In this way, the oral practice pronunciation standard characterization feature vector is subjected to a backward interpretability analysis (Backward Interpretability Analysis) so that the obtained backward learning class attention optimization oral practice pronunciation standard characterization feature vector has a backward learning sensitive preference relative to the predetermined class label of the classifier. And further, the ratio of the sum of the error metric vector and the vector length is calculated, and the optimization scaling factor obtained by using the ratio as an exponential calculation of the natural exponential function value is used to perform backward scaling optimization (BackwardOptimization) on the backward learning class attention optimization oral practice pronunciation standard characterization feature vector. In this way, the oral practice pronunciation standard characterization feature vector can be effectively improved in terms of the sensitive preference and robustness relative to the predetermined class label of the classifier, thereby improving the accuracy of the classification result.

[0052] Specifically, the evaluation result generating unit 143 is used to pass the feature vector representing the pronunciation standard of the optimized oral practice through a pronunciation standard evaluator based on a classifier to obtain the evaluation result, and the evaluation result is used to indicate whether the student's oral pronunciation meets a predetermined standard. That is, by inputting the feature vector representing the pronunciation standard of the optimized oral practice into a trained classifier, based on the classification decision boundary established by the classifier during the training process, the feature vector representing the pronunciation standard of the optimized oral practice is mapped to a predefined pronunciation standard level, and an evaluation result is generated to indicate whether the student's oral pronunciation meets a predetermined standard, thereby achieving a detailed evaluation and feedback on the student's oral practice.

[0053] In the above-mentioned English teaching auxiliary system 100 based on human-computer interaction, the pronunciation teaching module 150 is used to output the standard English pronunciation reference audio in response to the evaluation result that the student's spoken pronunciation does not meet the predetermined standard. That is, when the evaluation result shows that the student's spoken pronunciation does not meet the predetermined standard, the standard English pronunciation reference audio is automatically called to provide students with an intuitive and accurate pronunciation template. Students can learn standard pronunciation by imitating the reference audio to correct pronunciation errors and improve the accuracy and naturalness of pronunciation.

[0054] In summary, the English teaching auxiliary system based on human-computer interaction according to the embodiment of the present application is explained, which collects students' oral practice audio through a microphone, generates text transcription using speech recognition technology, and uses natural language processing technology to understand the text semantically, and generates standard English pronunciation reference audio based on text semantic features, and then uses deep learning technology to extract and compare the acoustic features of students' oral practice audio and the generated standard English pronunciation reference audio, so as to intelligently evaluate whether the students' oral pronunciation is accurate, and provide audio reference for non-standard pronunciation. In this way, intelligent evaluation and personalized guidance of students' English oral ability can be achieved, and students' oral practice ability and cross-cultural communication ability can be improved.

[0055] Figure 6 Flowchart of the English teaching auxiliary method based on human-computer interaction according to an embodiment of the present application. Figure 6 As shown, according to the embodiment of the present application, the English teaching auxiliary method based on human-computer interaction includes the following steps: S110, collecting students' oral practice audio data through a microphone; S120, generating standard English pronunciation reference audio based on the oral practice audio data; S130, respectively extracting the audio spectrum features of the oral practice audio data and the standard English pronunciation reference audio to obtain an oral practice audio spectrum feature vector and a reference audio spectrum feature vector; S140, determining an evaluation result of the student's oral pronunciation based on the feature difference between the oral practice audio spectrum feature vector and the reference audio spectrum feature vector; S150, in response to the evaluation result that the student's oral pronunciation does not meet the predetermined standard, outputting the standard English pronunciation reference audio.

[0056] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned English teaching auxiliary method based on human-computer interaction have been referred to above. Figures 1 to 5 The description of the English teaching auxiliary system based on human-computer interaction has been introduced in detail, and therefore, its repeated description will be omitted.

[0057] The basic principle of the present invention is described above in conjunction with specific embodiments. However, it should be pointed out that the advantages, strengths, effects, etc. mentioned in the present invention are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present invention. In addition, the specific details of the above embodiments are only for the purpose of illustration and facilitation of understanding, rather than limitation, and the above details do not limit the present invention to being implemented by adopting the above specific details.

[0058] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, please refer to the relevant description of other embodiments. In the several embodiments provided by the present invention, it should be understood that the disclosed system and method can be implemented in other ways. For example, the system embodiment described above is only schematic. For example, the module division is only a logical function division, and there may be other division methods in actual implementation. The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0059] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0060] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0061] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference to a figure in a claim should not be considered as limiting the claim to which it relates.

[0062] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units stated in the system claims can also be implemented by one unit through software or hardware.

[0063] Finally, it should be noted that the above description has been given for the purpose of illustration and description. In addition, the above embodiments are only used to illustrate the technical solution of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. An English teaching auxiliary system based on human-computer interaction, characterized in that: include: The oral practice audio collection module is used to collect students' oral practice audio data through a microphone; A standard pronunciation generation module, used to generate standard English pronunciation reference audio based on the spoken language practice audio data; An audio spectrum feature extraction module, used to extract the audio spectrum features of the spoken language practice audio data and the standard English pronunciation reference audio data to obtain a spoken language practice audio spectrum feature vector and a reference audio spectrum feature vector; A comparative analysis module, used for determining an evaluation result of the student's oral pronunciation based on a feature difference between the oral practice audio spectrum feature vector and the reference audio spectrum feature vector; The pronunciation teaching module is used to output the standard English pronunciation reference audio in response to the evaluation result that the student's spoken pronunciation does not meet the predetermined standard.

2. The English teaching auxiliary system based on human-computer interaction according to claim 1 is characterized in that: The standard pronunciation generation module comprises: A text conversion unit, used for performing speech recognition on the oral practice audio data to obtain a student oral practice text; A text semantic encoding unit, used for semantically encoding the student oral practice text to obtain a semantic feature vector of the student oral practice text; The reference audio generation unit is used to generate the standard English pronunciation reference audio by passing the semantic feature vector of the student oral practice text through a standard English pronunciation generation model based on a generative adversarial network.

3. The English teaching auxiliary system based on human-computer interaction according to claim 2 is characterized in that: The text semantic encoding unit is used to: The student oral practice text is passed through a semantic encoder based on the Bert model to obtain a semantic feature vector of the student oral practice text.

4. The English teaching auxiliary system based on human-computer interaction according to claim 3 is characterized in that: The audio spectrum feature extraction module comprises: A mel-spectrogram calculation unit, used to calculate the mel-spectrograms of the spoken language practice audio data and the standard English pronunciation reference audio respectively to obtain a spoken language practice audio mel-spectrogram and a reference audio mel-spectrogram; The acoustic feature extraction unit is used to pass the spoken language practice audio mel-spectrogram and the reference audio mel-spectrogram through an acoustic feature extractor based on a convolutional neural network model to obtain the spoken language practice audio spectrum feature vector and the reference audio spectrum feature vector.

5. The English teaching auxiliary system based on human-computer interaction according to claim 4 is characterized in that: The comparative analysis module comprises: A feature difference calculation unit, used for calculating a difference feature vector between the spoken language practice audio spectrum feature vector and the reference audio spectrum feature vector as a feature vector representing the standard degree of spoken language practice pronunciation; A feature distribution optimization unit, used for optimizing the feature distribution of the feature vector representing the standard degree of pronunciation of the spoken language practice to obtain an optimized feature vector representing the standard degree of pronunciation of the spoken language practice; The evaluation result generating unit is used for passing the optimized oral practice pronunciation standard characterization feature vector through a pronunciation standard evaluator based on a classifier to obtain the evaluation result, and the evaluation result is used to indicate whether the student's oral pronunciation meets the predetermined standard.

6. The English teaching auxiliary system based on human-computer interaction according to claim 5 is characterized in that: The feature distribution optimization unit is used to: The feature vector representing the standard degree of pronunciation of the oral practice is measured by classification error viewpoint based on backward machine learning to obtain the feature vector representing the optimized standard degree of pronunciation of the oral practice.

7. The English teaching auxiliary system based on human-computer interaction according to claim 6 is characterized in that: The feature distribution optimization unit comprises: A probabilistic subunit, used for performing probabilistic activation based on a Sigmoid function on the feature vector representing the standard degree of pronunciation of the spoken language practice to obtain a probabilistic feature vector representing the standard degree of pronunciation of the spoken language practice; A label probability generating subunit, used for inputting the characteristic vector representing the standard degree of pronunciation of the spoken language practice into the classifier to obtain an initial class label probability value; An error metric subunit, used to calculate the absolute value of the difference between the feature value at each position in the probabilistic oral practice pronunciation standard representation feature vector and the initial class label probability value to obtain a backward machine learning fine-grained error metric vector; An attention optimization subunit is used to use the backward machine learning fine-grained error metric vector as a key vector, the oral practice pronunciation standard characterization feature vector as a query vector and a value vector, and input the key vector, the query vector and the value vector into a converter structure to obtain a backward learning type attention optimization oral practice pronunciation standard characterization feature vector; an optimization scaling factor calculation subunit, configured to calculate a ratio between a sum of the backward machine learning fine-grained error metric vector and a length of the backward machine learning fine-grained error metric vector, and to calculate a natural exponential function value raised to a power of the ratio as an optimization scaling factor; The scale scaling subunit is used to calculate the product between the optimization scaling factor and the backward learning type attention optimized oral practice pronunciation standard representation feature vector to obtain the optimized oral practice pronunciation standard representation feature vector.

8. An English teaching auxiliary method based on human-computer interaction, characterized in that: include: Collect students’ oral practice audio data through microphone; Based on the oral practice audio data, generate standard English pronunciation reference audio; Extracting audio spectrum features of the spoken language practice audio data and the standard English pronunciation reference audio data respectively to obtain a spoken language practice audio spectrum feature vector and a reference audio spectrum feature vector; Determining an evaluation result of the student's spoken pronunciation based on a feature difference between the spoken practice audio spectrum feature vector and the reference audio spectrum feature vector; In response to the evaluation result that the student's spoken pronunciation does not meet the predetermined standard, the standard English pronunciation reference audio is output.

9. The English teaching auxiliary method based on human-computer interaction according to claim 8 is characterized in that: Based on the oral practice audio data, a standard English pronunciation reference audio is generated, including: Performing speech recognition on the oral practice audio data to obtain a student oral practice text; Performing semantic coding on the student oral practice text to obtain a semantic feature vector of the student oral practice text; The semantic feature vector of the student oral practice text is passed through a standard English pronunciation generation model based on a generative adversarial network to obtain the standard English pronunciation reference audio.

10. The English teaching auxiliary method based on human-computer interaction according to claim 9 is characterized in that: The student oral practice text is semantically encoded to obtain a semantic feature vector of the student oral practice text, including: The student oral practice text is passed through a semantic encoder based on the Bert model to obtain a semantic feature vector of the student oral practice text.

Citation Information

Cited By

  • Intelligent control method and system based on visual and auditory fusion and electronic equipment

    CN119036448A