Spoken English man-machine conversation interaction training method and system

Abstract

The invention discloses a spoken English man-machine conversation interaction training method and system, and relates to the technical field of spoken English training, spoken English audio signals of a user are monitored in real time through a microphone, and the frequency spectrum information of the audio signals is refined into each frame of spectrogram and cepstrum coefficient feature vectors by using Fourier transform and feature extraction technologies; therefore, the voice signal characteristics of the user can be completely captured; three core dimensions of semantic similarity, grammatical structure similarity and fluency are incorporated into an evaluation model, a comprehensive oral English training evaluation system is formed, a text sequence is generated by analyzing a time sequence dependency relationship through a convolutional neural network, and the oral English training evaluation model of a deep learning technology is combined, so that the oral English training evaluation accuracy is improved. According to the method, the expressions of the user in the aspects of semantics, grammar and fluency can be accurately measured, the oral training level of the user can be comprehensively judged by the system by comparing the comprehensive evaluation coefficient with the preset qualified threshold value, and a personalized learning guide is provided according to a result.

Description

Technical Field

[0001] The present invention relates to the technical field of spoken language training, and in particular to a method and system for human-computer dialogue interactive training of spoken English. Background Art

[0002] With the development of globalization, the importance of English as an international common language has become increasingly prominent, especially in the fields of education, business and technology, where fluent English speaking skills have become a need for more and more people. However, traditional English speaking training methods mainly rely on classroom teaching or one-on-one tutoring, which often involve high learning costs and time investment. At this time, a human-computer dialogue interaction training method needs to be introduced.

[0003] In the current oral training situation, traditional oral English learning relies on teachers' manual guidance or standardized online practice, which has significant shortcomings. First, teachers' evaluation is highly subjective and feedback is delayed, making it difficult to achieve real-time and objective evaluation; second, online practice lacks flexibility and personalization, making it difficult to provide accurate feedback based on users' specific performance and weaknesses. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a method and system for training human-computer dialogue interaction in spoken English, which solves the problems in the above-mentioned background technology.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: a method for training human-computer dialogue interaction in spoken English, comprising the following steps:

[0006] S1. During the user's English spoken human-computer dialogue interactive training, the user's English spoken audio signal is monitored in real time using a microphone, and the English spoken audio signal is input into the cloud platform;

[0007] S2. Using speech recognition technology, perform Fourier transform on the English spoken audio signal in the cloud platform, and after feature extraction, obtain the spectrum of each frame, construct the corresponding cepstral coefficients Dpxs based on each frame spectrum, and after combination, form the cepstral feature vector D of the corresponding frame spectrum;

[0008] S3, by inputting the cepstrum feature vector D of each frame spectrum obtained in S2 into the convolutional neural network, the time series dependency of the English spoken audio signal is captured to generate a text sequence, and the text sequence after text preprocessing is subjected to semantic and structural relationship analysis to respectively calculate and obtain the semantic similarity Yxd and the grammatical structure similarity Fxd;

[0009] S4. After the human-computer dialogue interaction training is completed, a complete English spoken audio signal is obtained, and based on the complete English spoken audio signal, the fluency of the user's expression during this training process is analyzed to obtain the fluency Lyd, and the semantic similarity Yxd, grammatical structure similarity Fxd and fluency Lyd are input into the oral training evaluation model based on the deep learning technology, and after linear normalization processing, the comprehensive evaluation coefficient Zpxs is fitted and output;

[0010] S5. Preset a qualified threshold W, and compare the comprehensive evaluation coefficient Zpxs with the qualified threshold W to comprehensively judge the user's English speaking training situation, and adopt a personalized learning guide based on the judgment result.

[0011] Preferably, the specific steps of S1 include:

[0012] S11, using a microphone to monitor in real time the English spoken audio signal of the user during the English spoken human-computer dialogue interactive training, and pre-processing the English spoken audio signal, using a frequency filter to remove noise in the English spoken audio signal, and measuring the peak amplitude of the English spoken audio signal, based on the peak amplitude, using automatic gain control technology to automatically adjust the audio gain, and then converting the English spoken audio signal into a digital signal;

[0013] S12 transmits the digital signal in S11 to the cloud platform in real time through the network.

[0014] Preferably, the specific steps of S2 include:

[0015] S21, dividing the digital signal in the cloud platform into a plurality of groups of short-time frames, and applying a Hanning window to each frame to perform windowing processing;

[0016] S22. Based on the content of S21 and combined with Fourier transform technology, the time domain signal of each frame is converted into a frequency domain signal to generate a frequency domain representation for each frame, and the amplitude of each frequency component is obtained to form a spectrum diagram of each frame, and the spectrum diagram of each frame is processed through multiple groups of filters to extract frequency information that matches the auditory characteristics of the human ear, wherein the multiple groups of filters are evenly distributed on the Mel frequency axis, and each filter covers a frequency range, so as to divide the spectrum diagram of each frame into several frequency bands, and obtain the spectrum energy value S(f) in each frequency band.

[0017] Preferably, the specific step S2 also includes:

[0018] S23, taking the logarithm of the spectrum energy value S(f) obtained in S22 to simulate the perception characteristics of the human ear to sound, so as to obtain a logarithmic energy spectrum;

[0019] S24, performing discrete cosine transform on the logarithmic energy spectrum to generate corresponding cepstrum coefficients Dpxs in each frame spectrum diagram, which are obtained specifically in the following manner:

[0020]

[0021] Where Dpxs n is represented by the nth cepstral coefficient, K represents the number of filters, k = 0, 1, 2, ..., K-1, k is the filter index; S(f k ) represents the spectrum energy value obtained after passing through the kth filter, n represents the index of the cepstral coefficient, and π represents a mathematical constant; log(S(f k )) is expressed as the logarithm of the spectrum energy value obtained after passing through the kth filter.

[0022] Preferably, the specific steps of S3 include:

[0023] S31, based on S24, obtain the nth cepstral coefficient Dpxs n In this way, the corresponding cepstral coefficients Dpxs of each frame spectrum are obtained respectively, and after being summarized, the cepstral feature vector D of the corresponding frame spectrum is formed;

[0024] S32. Input the inverse spectrum feature vector D of each frame spectrum graph obtained in S2 into the convolutional neural network. First, in the convolution layer, a convolution operation is performed on the inverse spectrum feature vector D of each frame spectrum graph to extract local features to generate a feature map, and the feature map is input to the pooling layer to downsample the feature map and retain important features. The downsampled feature map is then input to the fully connected layer to perform a fully connected operation and output the probability distribution of each phoneme. Finally, the probability distribution of each phoneme is input to the output layer, and the probability distribution of each phoneme is decoded in the output layer to generate a text sequence.

[0025] Preferably, the specific step S3 also includes:

[0026] S33, using natural language processing technology, preprocessing the text sequence, removing punctuation marks in the text sequence, and then dividing the text into several groups of sentences, and marking the part of speech of each word in the text sequence, and at the same time combining dependency syntax analysis technology to construct a syntax tree of the sentences in the text sequence to identify the syntactic structure, wherein in the syntax tree, the predicate verb in the sentence is used as the root node, and starting from the root node, the dependency relationship with the words other than the predicate verb in the sentence is established in sequence;

[0027] S34, based on the syntax tree obtained in S33, and in combination with the preset reference sentence, analyze the similarity of the structural relationship between the user sentence and the reference sentence to construct the grammatical structure similarity Fxd, which is specifically obtained in the following manner:

[0028]

[0029] Where m represents the number of words in the corresponding sentence, i = 1, 2, 3, ..., m, Yjc i Represents the part of speech of the i-th word in the user sentence, Cjc i denoted as the part of speech of the i-th word in the reference sentence, and δ is denoted as the indicator function;

[0030] S35. Based on the BERT model, the user sentence is divided into a number of word groups and characters to capture the bidirectional context information in the user sentence, and the user sentence is input into the BERT model to obtain the semantic vector Yx of the user sentence. Based on the semantic vector Yx of the user sentence, the semantic similarity between the user sentence and the reference sentence is analyzed to construct the semantic similarity Yxd, which is obtained specifically in the following manner:

[0031]

[0032] Where Cx represents the semantic vector of the reference sentence, ‖Yx‖ represents the modulus of the semantic vector of the user sentence, and ‖Cx‖ represents the modulus of the semantic vector of the reference sentence.

[0033] Preferably, the specific steps of S4 include:

[0034] S41. After the human-computer dialogue interaction training is completed, a complete English spoken audio signal is obtained, and based on the complete English spoken audio signal, the fluency of the user's expression during this training process is analyzed to obtain the fluency Lyd, and the fluency Lyd is obtained by the following formula:

[0035]

[0036] In the formula, Ysz represents the speaking speed, Tdc represents the number of pauses, and a 1 and a 2 are all weight values, P represents the first correction constant, where a 1 and a 2 The specific value is set by the user according to the situation.

[0037] Preferably, the specific step S4 also includes:

[0038] S42, by associating the semantic similarity Yxd, the grammatical structure similarity Fxd and the fluency Lyd, and combining with the trained oral training evaluation model, after linear normalization, the corresponding data values ​​are mapped to the interval [0, 1], so as to fit and output the comprehensive evaluation coefficient Zpxs from the oral training evaluation model, which is specifically obtained in the following manner:

[0039]

[0040] In the formula, α and β are weight values, C is the second correction constant, and the specific values ​​of α and β are set by the user according to the situation.

[0041] Preferably, the specific steps of S5 include:

[0042] S51, comparing and analyzing the qualified threshold W and the comprehensive evaluation coefficient Zpxs to comprehensively judge the English oral training situation of the user this time, and the specific judgment content is as follows:

[0043] If the comprehensive evaluation coefficient Zpxs exceeds the qualified threshold W, it will be comprehensively judged that the user's English oral training is in a qualified state, and positive feedback will be given to the user, pointing out the user's advantages in oral fluency, grammatical structure and semantic expression, and regularly reminding the user to review the qualified English oral training content;

[0044] If the comprehensive evaluation coefficient Zpxs does not exceed the qualified threshold W, it will be comprehensively judged that the user's English oral training is not in a qualified state, and the user will be fed back the performance differences in oral fluency, grammatical structure and semantic expression of this English oral training compared with the historical stage training. According to the performance differences, the user's training project will be arranged, and after the training, a phased test will be conducted again.

[0045] English oral human-computer dialogue interactive training system, including audio acquisition module, signal processing module, training analysis module, training evaluation module and feedback module;

[0046] The audio acquisition module is used to monitor the user's spoken English audio signal in real time using a microphone during the user's English spoken human-computer dialogue interactive training, and input the English spoken audio signal to the cloud platform;

[0047] The signal processing module is used to use speech recognition technology to perform Fourier transform on the English spoken audio signal in the cloud platform, and after feature extraction, obtain the spectrum of each frame, construct the corresponding cepstral coefficients Dpxs based on each frame spectrum, and after combination, form the cepstral feature vector D of the corresponding frame spectrum;

[0048] The training and analysis module is used to capture the time series dependency of the English spoken audio signal by inputting the cepstrum feature vector D of each frame spectrogram into the convolutional neural network to generate a text sequence, and perform semantic and structural relationship analysis on the text sequence after text preprocessing to calculate and obtain the semantic similarity Yxd and grammatical structure similarity Fxd respectively;

[0049] The training evaluation module is used to obtain a complete English spoken audio signal after the human-computer dialogue interaction training is completed, and based on the complete English spoken audio signal, analyze the fluency of the user's expression during this training process to obtain the fluency Lyd, and input the semantic similarity Yxd, grammatical structure similarity Fxd and fluency Lyd into the oral training evaluation model constructed according to the deep learning technology, and after linear normalization processing, fit and output the comprehensive evaluation coefficient Zpxs;

[0050] The feedback module is used to pre-set a qualified threshold W, and compare the comprehensive evaluation coefficient Zpxs with the qualified threshold W to comprehensively judge the user's English speaking training situation this time, and adopt a personalized learning guide based on the judgment result.

[0051] The present invention provides a method and system for training human-computer dialogue interaction in spoken English, which has the following beneficial effects:

[0052] (1) The user's English speaking audio signal is monitored in real time through a microphone, and the frequency spectrum information of the audio signal is refined into a spectrum diagram and cepstral coefficient feature vector of each frame by using Fourier transform and feature extraction technology, so as to fully capture the characteristics of the user's voice signal. This refinement not only retains the key features of the audio, but also effectively captures the subtle changes in the voice signal, allowing the system to more accurately analyze the user's pronunciation characteristics and phoneme composition; the three core dimensions of semantic similarity Yxd, grammatical structure similarity Fxd and fluency Lyd are incorporated into the evaluation model to form a comprehensive English oral training evaluation system. The text sequence is generated by analyzing the time series dependency through a convolutional neural network. The oral training evaluation model combined with deep learning technology can accurately measure the user's performance in semantics, grammar and fluency, making the evaluation results more comprehensive and reliable, and helping users to fully understand their oral level. By comparing the comprehensive evaluation coefficient Zpxs with the preset qualified threshold, the system can comprehensively judge the user's oral training level and provide a personalized learning guide based on the results. For users who meet the qualification standards, the system will generate positive feedback to enhance their self-confidence; for users who do not meet the qualification standards, the system not only points out the deficiencies, but also provides specific guidance on the content of subsequent training through the comprehensive evaluation coefficient Zpxs. Such personalized feedback and guidance can help users improve their oral English skills in a targeted manner, enhance the practicality of the system and user stickiness. Through this comprehensive evaluation and feedback mechanism, this method can further improve the scientificity and practicality of English oral training, provide users with accurate, real-time and personalized learning experience, and help users improve their English oral level efficiently.

[0053] (2) By real-time monitoring and preprocessing the user's English spoken audio signal, the system achieves noise reduction and automatic gain control in the signal acquisition stage to ensure the clarity and stability of the audio signal. The system removes noise through frequency filters, automatically adjusts the audio gain, and normalizes the signal amplitude to ensure that the input signal quality reaches a relatively good state. This process effectively eliminates the interference caused by external environmental noise, making subsequent analysis more accurate. In the cloud platform, the short-time Fourier transform (STFT) technology is used to convert the time domain signal into a frequency domain representation, so that each frame of the audio signal can generate a corresponding spectrogram. The spectrogram represents the energy distribution at each frequency with the amplitude of each frequency component, and applies multiple sets of filters to extract frequency information that matches the auditory characteristics of the human ear. At the same time, the system applies windowing techniques such as Hanning window when dividing the signal into frames to reduce frequency leakage and improve the smoothness between frames. By applying the window function point by point, the system can generate a smoothly transitioned windowed signal to ensure the clarity of each frame signal in the time and frequency domain, thereby obtaining more accurate frequency information during feature extraction. This operation not only ensures the smoothness of the spectrum graph, but also makes the subsequent frequency domain analysis more accurate, providing a solid foundation for the speech recognition and analysis of the overall system. In summary, this method further improves the quality of audio input and the accuracy of analysis through a series of meticulous audio preprocessing and frequency domain feature extraction techniques, allowing the system to more efficiently extract the user's voice features, optimize the subsequent English spoken analysis and evaluation process, and enhance the user's interactive experience and training effect.

[0054] (3) By performing discrete cosine transform (DCT) on the logarithmic energy spectrum, the system can convert the frequency domain information into the cepstral domain, thereby generating the cepstral coefficients Dpxs of each frame. DCT not only effectively compresses the spectral information and retains the main envelope characteristics of the signal, but also provides more concentrated frequency components in different frequency bands, which facilitates subsequent feature extraction and classification. This feature extraction method can further improve the accuracy of the system in phoneme and speech feature recognition.

[0055] (4) By acquiring the cepstral coefficients Dpxs frame by frame and forming cepstral feature vectors, the system can accurately capture the key features of the audio signal and integrate this information into a higher-dimensional feature vector. The cepstral feature vector of each frame effectively condenses the time-frequency information in the speech signal and can fully represent the spectral characteristics of the audio, laying a solid foundation for subsequent speech recognition. The cepstral feature vector is input into the convolutional neural network (CNN). The convolution layer extracts local features from the cepstral feature vector through a sliding convolution kernel, which can capture the local patterns of the audio signal in time and frequency and generate a feature map. This convolution operation enables the system to automatically identify and extract important feature patterns in the audio signal, especially for recognizing pronunciation details, phoneme changes, etc. It has higher sensitivity. Each element in the feature map represents the pattern strength at a specific location, so that features at different locations can be effectively captured. In summary, this method forms a complete speech signal processing and text generation process through cepstral feature extraction, local feature detection in the convolution layer, downsampling in the pooling layer, and classification in the fully connected layer. This method not only further improves the accuracy of feature extraction, but also optimizes the computational efficiency and recognition accuracy, providing users with an efficient and accurate interactive training experience for spoken English.

[0056] (5) Through the construction of the syntax tree, the system can more accurately understand the internal structure of the sentence and the relationship between words, and help identify the advantages and disadvantages of the user in the grammatical structure. This in-depth dependency analysis can effectively feedback the user's grammatical errors or deficiencies, and help the user improve the sentence structure and expression in a targeted manner. By comparing the syntax tree of the user's sentence with the syntax tree of the preset reference sentence, the system can calculate the similarity of the grammatical structure of the two. By comparing the parts of speech one by one, the system can determine the degree of structural similarity between the user's sentence and the reference sentence. In this way, the system can not only point out the user's grammatical errors, but also quantify the gap between the user's grammatical structure and the standard expression, and provide the user with a quantitative grammatical evaluation index. This similarity calculation method improves the system's understanding of the user's grammatical structure and helps improve the correctness of the user's sentence structure and expression. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A schematic diagram of the flow chart of the English spoken human-computer dialogue interactive training method of the present invention;

[0058] Figure 2 This is a block diagram of the English spoken human-computer dialogue interactive training system of the present invention. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] Example 1

[0061] See also Figure 1 The present invention provides an English spoken human-computer dialogue interactive training method, comprising the following steps:

[0062] S1. During the user's English spoken human-computer dialogue interactive training, the user's English spoken audio signal is monitored in real time using a microphone, and the English spoken audio signal is input into the cloud platform;

[0063] S2. Using speech recognition technology, perform Fourier transform on the English spoken audio signal in the cloud platform, and after feature extraction, obtain the spectrum of each frame, construct the corresponding cepstral coefficients Dpxs based on each frame spectrum, and after combination, form the cepstral feature vector D of the corresponding frame spectrum;

[0064] S3, by inputting the cepstrum feature vector D of each frame spectrum obtained in S2 into the convolutional neural network, the time series dependency of the English spoken audio signal is captured to generate a text sequence, and the text sequence after text preprocessing is subjected to semantic and structural relationship analysis to respectively calculate and obtain the semantic similarity Yxd and the grammatical structure similarity Fxd;

[0065] S4. After the human-computer dialogue interaction training is completed, a complete English spoken audio signal is obtained, and based on the complete English spoken audio signal, the fluency of the user's expression during this training process is analyzed to obtain the fluency Lyd, and the semantic similarity Yxd, grammatical structure similarity Fxd and fluency Lyd are input into the oral training evaluation model based on the deep learning technology, and after linear normalization processing, the comprehensive evaluation coefficient Zpxs is fitted and output;

[0066] S5. Preset a qualified threshold W, and compare the comprehensive evaluation coefficient Zpxs with the qualified threshold W to comprehensively judge the user's English speaking training situation, and adopt a personalized learning guide based on the judgment result.

[0067] In this embodiment, real-time monitoring and cloud processing: the user's English spoken audio signal is monitored in real time through a microphone and transmitted to the cloud platform. This process ensures that the system can collect the user's voice in real time and accurately, and with the help of the powerful processing capabilities of cloud computing, it can achieve efficient processing and storage of large-scale data, providing a solid foundation for subsequent voice analysis. Accurate voice feature extraction and analysis: The audio signal is Fourier transformed and feature extracted by using speech recognition technology. By obtaining the spectrum diagram and constructing the cepstrum feature vector, the system can carefully analyze the spectral characteristics of the audio signal, capture the dependencies in the time series, and then generate an accurate text sequence. This process not only improves the accuracy of speech recognition, but also lays the foundation for semantic and structural analysis. Multi-dimensional evaluation model: After the text is generated, the oral training evaluation model constructed by deep learning technology can comprehensively evaluate the user's English speaking from multiple dimensions such as semantic similarity, grammatical structure similarity and fluency. Compared with the traditional single-dimensional scoring method, this multi-dimensional comprehensive evaluation model can measure the user's oral level in a multi-faceted manner, provide more accurate feedback, and help users clearly understand their own strengths and weaknesses. Personalized learning suggestions: By comparing the comprehensive evaluation coefficient Zpxs with the preset qualified threshold, the system can judge whether the user's English oral training meets the standard based on the user's comprehensive evaluation results, and provide personalized learning guides based on the standard. This method can not only provide positive feedback based on the user's performance and encourage users to continue to improve, but also provide targeted learning suggestions when the standard is not met, helping users to improve their weak links and improve learning effects. In summary, this method further realizes the intelligence and personalization of English oral training, which not only improves the efficiency and accuracy of users' oral training, but also enhances the continuity and pertinence of users' learning, and helps users to comprehensively improve their English oral ability.

[0068] Example 2

[0069] Please refer to Figure 1 , specifically: S1 specific steps include:

[0070] S11, using a microphone to monitor the English spoken audio signal of the user in the process of English spoken human-computer dialogue interactive training in real time, and pre-processing the English spoken audio signal, using a frequency filter to remove noise in the English spoken audio signal, and measuring the peak amplitude of the English spoken audio signal, based on the peak amplitude, using automatic gain control technology (AGC) to automatically adjust the audio gain to adjust the amplitude of the English spoken audio signal to an appropriate range, and then converting the English spoken audio signal into a digital signal;

[0071] S12 transmits the digital signal in S11 to the cloud platform in real time through the network.

[0072] The specific steps of S2 include:

[0073] S21, dividing the digital signal in the cloud platform into several groups of short-time frames (for example, one frame every 20 milliseconds) to facilitate subsequent feature extraction such as short-time Fourier transform (STFT); and applying a Hanning window to each frame for windowing processing, thereby reducing frequency leakage and improving smoothness between frames;

[0074] Audio signals are usually non-stationary signals and need to be divided into short stationary frames, each of which is usually 20-30 milliseconds long.

[0075] The specific steps of windowing processing include step 1: selecting a window function (Hamming window or Hanning window) and calculating the window value corresponding to each sampling point; step 2: applying the window function point by point to the signal of each frame; result: generating a smoothly transitioned windowed signal, reducing frequency leakage and improving the clarity of the spectrum.

[0076] S22. Based on the content of S21, combined with Fourier transform (FFT) technology, the time domain signal of each frame is converted into a frequency domain signal to generate a frequency domain representation for each frame, and the amplitude of each frequency component is obtained to form a spectrum diagram of each frame, and the spectrum diagram of each frame is processed through multiple groups of filters (that is, the frequency components of the spectrum diagram are convolved within the coverage range of each filter) to extract frequency information that matches the auditory characteristics of the human ear, wherein the multiple groups of filters are evenly distributed on the Mel frequency axis, and each filter covers a frequency range to divide the spectrum diagram of each frame into several frequency bands, and obtain the spectrum energy value S(f) in each frequency band.

[0077] The spectrum energy value S(f) in each frequency band is obtained by:

[0078] Multiply the spectrum amplitude (power spectrum) in the frequency band covered by the filter by the filter weight point by point, and sum these products to obtain the total energy of the frequency band. The formula is as follows: S(f) = ∑ f |x(f)| 2 ·H k (f); where x(f) is the amplitude at frequency f in the spectrum graph, H k (f) is the weight value of the kth filter at frequency f;

[0079] The product and sum here are actually the weighted average of each frequency band, so that the frequency energy value in each Mel filter frequency band can reflect the auditory characteristics of the human ear.

[0080] In the spectrum diagram, the amplitude (or power) at frequency f can be obtained by calculating the modulus of the complex number;

[0081] Each point in the spectrum graph represents the amplitude of the corresponding frame at a specific frequency.

[0082] Function of multiple groups of filters: The application of multiple groups of filters divides the spectrum into several frequency bands in order to find the energy value in each frequency band.

[0083] The Mel frequency axis is defined based on the human ear's frequency perception characteristics. It has higher resolution in the low frequency band and lower resolution in the high frequency band.

[0084] The filter is based on the Mel frequency scale distribution, and its purpose is to enable the system to respond differently to sounds of different frequencies, just like the human ear. The Mel frequency scale is a nonlinear frequency scale defined based on the human ear's perception of frequency. The human ear is more sensitive to low frequencies and less sensitive to high frequencies. Therefore, the Mel frequency axis has a higher resolution in the low frequency band and a lower resolution in the high frequency band. This nonlinear distribution makes the filter interval in the low frequency band smaller and the filter interval in the high frequency band larger.

[0085] In the preprocessing process, the English spoken audio signal is first converted into a digital signal through analog-to-digital conversion (ADC). This digital signal is still a time domain signal, which represents the change of the audio signal over time. After that, the digital time domain signal is converted into a frequency domain signal using short-time Fourier transform (STFT), so that the spectrum information (i.e. frequency components and energy distribution) at each moment can be obtained, which is more convenient for subsequent feature extraction and analysis.

[0086] In this embodiment, real-time and clear audio signal acquisition and enhancement: the user's spoken English audio signal is monitored in real time through a microphone, and noise is removed through a frequency filter in the preprocessing stage. At the same time, the automatic gain control (AGC) technology is used to automatically adjust the gain according to the peak amplitude of the audio signal to keep the amplitude of the audio signal in an appropriate range. This design can significantly improve the clarity of the audio without losing the sound quality, ensuring that the signal quality is high enough in the subsequent speech recognition process, and further avoiding the recognition effect affected by audio signal distortion or noise interference. High-quality feature extraction of the spectrum graph: After dividing the digital signal into short-time frames, the audio signal is converted into a frequency domain signal using short-time Fourier transform (STFT), and Hanning window processing is applied to each frame signal to reduce frequency leakage. Through this process, a smoother and clearer spectrum graph can be obtained, which retains the subtle features of the audio signal and reduces the frequency interference between frames. This processing method can more accurately present the frequency and energy distribution of each frequency band, making subsequent feature extraction more reliable. Frequency feature extraction simulating human ear perception: Based on the spectrum graph, the audio frequency band is segmented through multiple groups of filters evenly distributed on the Mel frequency axis to extract frequency information that matches the human ear's auditory characteristics. These filters are designed based on the frequency perception characteristics of the human ear to ensure high resolution in the low frequency band and low resolution in the high frequency band, which further conforms to the nonlinear perception characteristics of the human ear to frequency. The frequency extraction process designed in this way can not only better fit the human ear's auditory experience, but also effectively reduce the amount of calculation for processing high-frequency band data, providing a double guarantee for the system's computational efficiency and accuracy. In summary, this method not only improves the recognition accuracy and robustness of the system through real-time preprocessing of audio signals, high-precision generation of spectrum graphs, and feature extraction that conforms to human ear perception, but also provides users with a more natural and accurate voice training experience, enabling the human-computer interaction training system to adapt to different environments and stably output high-quality training results.

[0087] Example 3

[0088] Please refer to Figure 1 Specifically: S2 includes the following specific steps:

[0089] S23. Taking the logarithm of the spectrum energy value S(f) obtained in S22 to simulate the human ear's perception characteristics of sound (i.e., the logarithmic relationship of sound intensity) to obtain a logarithmic energy spectrum. The purpose of taking the logarithm is to convert the linear value of energy into a logarithmic scale to be closer to the human ear's perception of sound intensity.

[0090] S24, performing discrete cosine transform (DCT) on the logarithmic energy spectrum to generate corresponding cepstral coefficients Dpxs in each frame spectrum diagram, which are specifically obtained in the following manner:

[0091]

[0092] Where Dpxs n is represented by the nth cepstral coefficient, K represents the number of filters, k = 0, 1, 2, ..., K-1, k is the filter index, which is used to represent the frequency band of the kth filter; S(f k ) represents the spectrum energy value obtained after passing through the kth filter, n represents the index of the cepstral coefficient, and π represents a mathematical constant, which is approximately equal to 3.14159; log(S(f k )) is expressed as the logarithm of the spectrum energy value obtained after passing through the kth filter, that is, the logarithmic energy spectrum; It is a proportional term of discrete cosine transform, which is used to determine the "position" of frequency components; k+0.5 represents the offset of discrete points. In discrete cosine transform (DCT), this offset makes the discrete points more concentrated, thereby more effectively compressing the spectrum information and retaining the main low-frequency components. Adding 0.5 is a standard step of DCT to ensure smooth transition of the spectrum and improve the accuracy of the transformation result; cos is the cosine function; The frequency component representing the nth cepstral coefficient is weighted on the spectral energy value obtained after passing through the kth filter. Through cosine transform, we can convert the logarithmic energy spectrum from the frequency domain to the cepstral domain in order to generate cepstral coefficients (i.e. MFCC features) and retain the envelope characteristics of the signal.

[0093] In this embodiment, the feature extraction conforms to the human auditory perception: by taking the logarithm of the spectrum energy value, the linear value of energy is converted to a logarithmic scale, so as to be closer to the human ear's perception characteristics of sound intensity. The human ear's perception of sound intensity is nonlinear. The use of the logarithmic energy spectrum is not only more in line with the natural way of human ear perception of sound, but also can effectively reduce the influence of high energy components and enhance the subtle features of low energy components, so that the system can more accurately capture important voice information. This step significantly improves the effectiveness of voice signal features and provides a more accurate feature basis for subsequent voice recognition and analysis. Generate cepstrum features through discrete cosine transform (DCT) to retain the main information of the voice signal: DCT processing is performed on the basis of the logarithmic energy spectrum, which can convert the frequency domain information to the cepstrum domain and generate cepstrum coefficients Dpxs. The standard steps of DCT include adding 0.5 as an offset to make the discrete points more concentrated, thereby effectively compressing the spectrum information and retaining the main low-frequency features. The cepstral coefficient is a compact representation of the speech signal envelope, which can effectively reflect the structural characteristics of speech and reduce feature redundancy, making the generated feature vector more concise and highly recognizable, providing high-quality input for subsequent processing of convolutional neural networks. Accurate frequency position control enhances the accuracy of feature extraction: In DCT processing, the proportional term is used to determine the specific position of the frequency component, and the weight distribution is achieved through the cosine function, so that each cepstral coefficient reflects the characteristic intensity of the corresponding frequency component in different filter bands. This frequency position control can construct a feature expression in the cepstral domain that is consistent with the original frequency domain frequency position, ensuring that the feature extraction process is both effective and does not lose the integrity of information, and retains the key features of the speech signal as much as possible. In summary, this method achieves an efficient feature extraction method that matches the human ear perception characteristics and accurately controls the frequency position by taking the logarithm of the spectral energy value and applying DCT to generate cepstral coefficients. This not only improves the recognizability of the feature and the information compression rate, but also improves the sensitivity and accuracy of the system when processing speech signals, providing a solid data foundation for speech recognition in human-computer dialogue interaction systems.

[0094] Example 4

[0095] Please refer to Figure 1 , specifically: S3 specific steps include:

[0096] S31, based on S24, obtain the nth cepstral coefficient Dpxs n In this way, the corresponding cepstral coefficients Dpxs of each frame spectrum are obtained respectively, and after being summarized, the cepstral feature vector D of the corresponding frame spectrum is formed;

[0097] S32. Input the inverse spectrum feature vector D of each frame spectrum graph obtained in S2 into the convolutional neural network. First, in the convolution layer, a convolution operation is performed on the inverse spectrum feature vector D of each frame spectrum graph to extract local features to generate a feature map, wherein the convolution kernel detects features at different positions in the audio signal, and the feature map is input into the pooling layer to downsample the feature map, thereby reducing the size of the feature map, reducing the amount of calculation, and retaining important features. The downsampled feature map is then input into the fully connected layer for a fully connected operation, and the probability distribution of each phoneme is output. Finally, the probability distribution of each phoneme is input into the output layer, and the probability distribution of each phoneme is decoded in the output layer to generate a text sequence.

[0098] Convolution operation: The convolution kernel slides on the input feature matrix, and each time it slides, it multiplies the elements of the feature matrix (the same size as the convolution kernel) and sums them up to get a value. Generate feature map: Combine the convolution results of all positions to form a new matrix, which is the feature map. Each element in the feature map represents the pattern strength or response of the input feature at a certain position. Multiple feature maps can be generated by multiple convolution kernels, each of which captures different feature patterns.

[0099] In this embodiment, accurate feature extraction and characterization: by inputting the cepstral feature vector of each frame of the spectrum graph into the convolutional neural network, the system can gradually extract the local features of the audio signal using the convolution kernel in the convolution layer. The convolution operation can effectively detect the characteristic patterns at different positions in the audio signal, thereby generating a feature map reflecting the audio feature pattern. Each element in the feature map represents the pattern strength or response of the input feature at a certain position. Through this structured feature extraction method, the system can more finely identify the spectral features in the speech signal and achieve a deep characterization of the audio features. Efficient downsampling and computing resource optimization: In the pooling layer, the system downsamples the feature map, which not only reduces the size of the feature map and reduces the amount of data, but also retains important feature patterns. This downsampling strategy significantly reduces the computing resource usage while ensuring the accurate capture of key features, thereby improving the operating efficiency of the model and the utilization of computing resources, saving time and storage space for subsequent training and reasoning. Accurate phoneme probability distribution output and text generation: The system performs a fully connected operation on the downsampled features in the fully connected layer, and finally generates the probability distribution of each phoneme, and decodes these probability distributions in the output layer to generate a text sequence. This processing method ensures that the system can accurately convert the user's audio signal into text, thereby achieving a high accuracy in the mapping of speech to text. Through the layer-by-layer feature extraction and decoding mechanism of the convolutional neural network, the system can provide reliable text output in speech recognition, providing intuitive and accurate feedback for the user's English speaking training. In summary, this method can achieve efficient and accurate phoneme recognition and text generation through multi-layer feature extraction, downsampling and decoding of audio signals by convolutional neural networks. This process not only ensures the operating efficiency of the system, but also improves the accuracy of speech recognition and text conversion, providing stronger real-time feedback and data support for users' English speaking training.

[0100] Example 5

[0101] Please refer to Figure 1 , specifically: S3 specific steps also include:

[0102] S33, using natural language processing technology (NLP), preprocessing the text sequence, removing punctuation marks in the text sequence, and then dividing the text into several groups of sentences, and marking the part of speech (such as verb, noun, adjective) of each word in the text sequence, and at the same time combining dependency syntax analysis technology to construct a grammatical tree of the sentences in the text sequence to identify the syntactic structure, wherein in the grammatical tree, the predicate verb in the sentence is used as the root node, and starting from the root node, the dependency relationship with the words other than the predicate verb in the sentence is established in sequence, wherein the dependency relationship includes but is not limited to the subject-predicate relationship, the verb-object relationship, the modification relationship, the attributive relationship and the preposition relationship;

[0103] Subject-predicate relationship: indicates the relationship between the subject and the predicate. Verb-object relationship: indicates the relationship between the verb and the object. Modification relationship: indicates the relationship between the modifier (such as adjectives, prepositional phrases) and the modified word. Attribution-predicate relationship: indicates the relationship between the noun and the word that modifies the noun. Preposition relationship: indicates the modification relationship of prepositional phrases;

[0104] S34, based on the syntax tree obtained in S33, and in combination with the preset reference sentence, analyze the similarity of the structural relationship between the user sentence and the reference sentence to construct the grammatical structure similarity Fxd, which is specifically obtained in the following manner:

[0105]

[0106] Where m represents the number of words in the corresponding sentence, i = 1, 2, 3, ..., m, Yjc i Represents the part of speech of the i-th word in the user sentence, Cjc i It represents the part of speech of the i-th word in the reference sentence, and δ represents the indicator function, which is 1 when the part of speech is consistent, otherwise it is 0;

[0107] S35. Based on the BERT model, the user sentence is divided into a number of word groups and characters to capture the bidirectional context information in the user sentence, and the user sentence is input into the BERT model to obtain the semantic vector Yx of the user sentence. Based on the semantic vector Yx of the user sentence, the semantic similarity between the user sentence and the reference sentence is analyzed to construct the semantic similarity Yxd, which is obtained specifically in the following manner:

[0108]

[0109] Where Cx represents the semantic vector of the reference sentence, ‖Yx‖ represents the modulus of the semantic vector of the user sentence, and ‖Cx‖ represents the modulus of the semantic vector of the reference sentence.

[0110] In this embodiment, accurate syntactic structure analysis: by preprocessing the text sequence through natural language processing (NLP) technology, the system can efficiently remove punctuation marks and accurately identify sentence boundaries and part-of-speech tags in the text. Combined with dependency syntactic analysis technology, the system can automatically construct a grammatical tree of the sentence, with the predicate verb as the root node, and establish dependencies with other words step by step. The construction of the grammatical tree can help the system accurately identify syntactic relationships such as subject-predicate, verb-object, and modification, and clearly present the sentence structure. This deep syntactic analysis can provide users with detailed grammatical feedback, which helps them master the correct structure of sentence composition and improve the accuracy of expression. Quantitative grammatical structure similarity evaluation: by comparing with preset reference sentences, the system can analyze the grammatical structure similarity between the user sentence and the reference sentence based on the dependency tree, and construct a quantitative grammatical structure similarity Fxd. This similarity is based on part-of-speech and dependency comparison, which can accurately reflect the structural proximity of the user sentence to the reference sentence. This step not only provides an objective structural evaluation standard, but also allows users to clearly understand their own grammatical defects, which is convenient for them to make targeted improvements. Semantic understanding and context analysis capabilities: By using the BERT model to capture the bidirectional context information of user sentences, the system can obtain the deep semantic representation of user sentences (semantic vector Yx) and compare them with reference sentences to construct a quantitative semantic similarity Yxd. The BERT model can capture the contextual relationship between words in a sentence, thereby accurately parsing the overall meaning of the user's sentence. This semantic similarity not only helps the system identify subtle differences in semantics, but also effectively evaluates the accuracy of the user's expression, allowing users to more clearly understand the gap in their expressions at the semantic level. In summary, this method further improves the system's ability in syntactic understanding and semantic analysis through technologies such as syntax tree construction, grammatical structure similarity evaluation, and semantic vector comparison. The system can not only deeply understand the user's syntactic and semantic structure, but also quantitatively compare reference sentences, provide users with detailed and objective feedback, and help them continuously optimize their grammatical and semantic expressions, and improve the accuracy and expression ability of spoken English.

[0111] Example 6

[0112] Please refer to Figure 1 , specifically: S4 specific steps include:

[0113] S41. After the human-computer dialogue interaction training is completed, a complete English spoken audio signal is obtained, and based on the complete English spoken audio signal, the fluency of the user's expression during this training process is analyzed to obtain the fluency Lyd, and the fluency Lyd is obtained by the following formula:

[0114]

[0115] In the formula, Ysz represents the speaking speed, Tdc represents the number of pauses, and a 1 and a 2 are all weight values, P represents the first correction constant, where a 1 and a 2 The specific value is set by the user according to the situation.

[0116] The above-mentioned number of pauses Tdc is identified by analyzing the silent segment or low-energy segment of the audio signal. The data source of the speech rate Ysz and the number of pauses Tdc is the user's voice signal, which is mainly collected through a microphone.

[0117] In this embodiment, accurate fluency measurement: by obtaining a complete English spoken audio signal and analyzing the fluency of the user's expression during the training process, the system can comprehensively evaluate the user's fluency from two core dimensions: speech rate and number of pauses. This evaluation method combines key parameters such as speech rate and pauses, and uses linear weighted calculation to make the fluency evaluation more objective and comprehensive. Personalized adjustment and accurate feedback: The weight values ​​and correction constants in the formula can be personalized according to the specific situation of the user, making the fluency calculation more flexible. For example, for users of different levels, the system can adjust the weights to adapt to their fluency evaluation criteria, thereby providing more personalized feedback. Through this accurate fluency feedback, users can understand their specific performance in speech rate and pauses, helping them to further improve their expression fluency. Fluency evaluation model based on actual performance: by establishing a fluency calculation model, the system can accurately measure the user's actual oral performance, especially the subtle fluency differences in oral expression. Compared with the feedback method based on subjective judgment alone, the model provides a quantitative fluency index, making the evaluation results more scientific and reliable. Users can conduct targeted practice and improvement based on the system's fluency scores and feedback, thereby effectively improving the overall effect of oral training. In summary, this method provides users with a scientific and intuitive fluency evaluation through precise speech rate and pause analysis, personalized weight setting, and a quantitative fluency model, helping users to more clearly identify and improve their own expression fluency, thereby optimizing the oral training experience.

[0118] Example 7

[0119] Please refer to Figure 1 , specifically:

[0120] The specific steps of S4 also include:

[0121] S42, by associating the semantic similarity Yxd, the grammatical structure similarity Fxd and the fluency Lyd, and combining with the trained oral training evaluation model, after linear normalization, the corresponding data values ​​are mapped to the interval [0, 1], so as to fit and output the comprehensive evaluation coefficient Zpxs from the oral training evaluation model, which is specifically obtained in the following manner:

[0122]

[0123] In the formula, α and β are weight values, C is the second correction constant, and the specific values ​​of α and β are set by the user according to the situation.

[0124] The specific steps of S5 include:

[0125] S51, comparing and analyzing the qualified threshold W and the comprehensive evaluation coefficient Zpxs to comprehensively judge the English oral training situation of the user this time, and the specific judgment content is as follows:

[0126] If the comprehensive evaluation coefficient Zpxs exceeds the qualified threshold W, it will be comprehensively judged that the user's oral English training is in a qualified state, and positive feedback will be given to the user, pointing out the user's advantages in oral fluency, grammatical structure and semantic expression to enhance the user's self-confidence, and at the same time regularly remind the user to review the qualified oral English training content to prevent forgetting;

[0127] If the comprehensive evaluation coefficient Zpxs does not exceed the qualified threshold W, it will be comprehensively judged that the user's English oral training is not in a qualified state, and the user will be fed back the performance differences in oral fluency, grammatical structure and semantic expression of this English oral training compared with the historical stage training. According to the performance differences, the user's training project will be arranged, and after the training, a phased test will be conducted again.

[0128] In this embodiment, a multi-dimensional comprehensive evaluation system: by combining multiple indicators such as semantic similarity, grammatical structure similarity and fluency, and associating these indicators with the trained oral evaluation model, the system can provide comprehensive evaluation results. Through linear normalization, the data values ​​of different dimensions are mapped to the same interval, ensuring the consistency of the comparison of different evaluation dimensions and the reliability of the results. The system provides users with accurate feedback based on the comprehensive evaluation coefficient adjusted by different weights, so that the evaluation results can truly reflect the user's overall oral level. Flexible personalized standard setting and feedback: adjustable weight values ​​and second correction constants are introduced in the evaluation model. Users can adjust these parameters according to their own needs, so that the system's evaluation is more in line with personalized learning goals and training levels. For qualified users, the system automatically generates positive feedback, points out the advantages in fluency, grammar and semantic expression, and regularly reminds users to review to help them consolidate the content they have mastered, thereby further enhancing their self-confidence. Precise differential feedback and intensive training: For users who do not meet the qualification standards, the system compares the current training results with the training results of the historical stage, and points out the specific differences in the user's fluency, grammatical structure and semantic expression. This feedback not only helps users to clarify their deficiencies in different dimensions, but also automatically generates personalized follow-up training plans. The system will arrange targeted training projects based on the performance differences of users to help users improve their weak links in a targeted manner, and conduct periodic tests after the training to ensure the stability and durability of the improvement effect. In summary, this method helps users fully understand and improve their English speaking level by constructing a multi-dimensional comprehensive evaluation system, a flexible personalized feedback mechanism and a precise differential feedback method. The system not only provides accurate and scientific training evaluation, but also achieves significant improvements in personalized guidance and learning optimization, thereby further enhancing the user's learning experience and training effect.

[0129] Example 8

[0130] Please refer to Figure 2 ,Specifically: English spoken human-computer dialogue interactive training system, including audio acquisition module, signal processing module, training analysis module, training evaluation module and feedback module;

[0131] The audio acquisition module is used to monitor the user's spoken English audio signal in real time using a microphone during the user's English spoken human-computer dialogue interactive training, and input the English spoken audio signal to the cloud platform;

[0132] The signal processing module is used to use speech recognition technology to perform Fourier transform on the English spoken audio signal in the cloud platform, and after feature extraction, obtain the spectrum of each frame, construct the corresponding cepstral coefficients Dpxs based on each frame spectrum, and after combination, form the cepstral feature vector D of the corresponding frame spectrum;

[0133] The training and analysis module is used to capture the time series dependency of the English spoken audio signal by inputting the cepstrum feature vector D of each frame spectrogram into the convolutional neural network to generate a text sequence, and perform semantic and structural relationship analysis on the text sequence after text preprocessing to calculate and obtain the semantic similarity Yxd and grammatical structure similarity Fxd respectively;

[0134] The training evaluation module is used to obtain a complete English spoken audio signal after the human-computer dialogue interaction training is completed, and based on the complete English spoken audio signal, analyze the fluency of the user's expression during this training process to obtain the fluency Lyd, and input the semantic similarity Yxd, grammatical structure similarity Fxd and fluency Lyd into the oral training evaluation model constructed according to the deep learning technology, and after linear normalization processing, fit and output the comprehensive evaluation coefficient Zpxs;

[0135] The feedback module is used to pre-set a qualified threshold W, and compare the comprehensive evaluation coefficient Zpxs with the qualified threshold W to comprehensively judge the user's English speaking training situation this time, and adopt a personalized learning guide based on the judgment result.

[0136] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for interactive training of spoken English by human and computer, characterized by: The following steps are included: S1. During the user's English spoken human-computer dialogue interactive training, the user's English spoken audio signal is monitored in real time using a microphone, and the English spoken audio signal is input into the cloud platform; S2. Using speech recognition technology, perform Fourier transform on the English spoken audio signal in the cloud platform, and after feature extraction, obtain the spectrum of each frame, construct the corresponding cepstral coefficients Dpxs based on each frame spectrum, and after combination, form the cepstral feature vector D of the corresponding frame spectrum; S3, by inputting the cepstrum feature vector D of each frame spectrum obtained in S2 into the convolutional neural network, the time series dependency of the English spoken audio signal is captured to generate a text sequence, and the text sequence after text preprocessing is subjected to semantic and structural relationship analysis to respectively calculate and obtain the semantic similarity Yxd and the grammatical structure similarity Fxd; S4. After the human-computer dialogue interaction training is completed, a complete English spoken audio signal is obtained, and based on the complete English spoken audio signal, the fluency of the user's expression during this training process is analyzed to obtain the fluency Lyd, and the semantic similarity Yxd, grammatical structure similarity Fxd and fluency Lyd are input into the oral training evaluation model based on the deep learning technology, and after linear normalization processing, the comprehensive evaluation coefficient Zpxs is fitted and output; S5. Preset a qualified threshold W, and compare the comprehensive evaluation coefficient Zpxs with the qualified threshold W to comprehensively judge the user's English speaking training situation, and adopt a personalized learning guide based on the judgment result.

2. The English spoken human-computer dialogue interactive training method according to claim 1 is characterized by: The specific steps of S1 include: S11, using a microphone to monitor in real time the English spoken audio signal of the user during the English spoken human-computer dialogue interactive training, and pre-processing the English spoken audio signal, using a frequency filter to remove noise in the English spoken audio signal, and measuring the peak amplitude of the English spoken audio signal, based on the peak amplitude, using automatic gain control technology to automatically adjust the audio gain, and then converting the English spoken audio signal into a digital signal; S12 transmits the digital signal in S11 to the cloud platform in real time through the network.

3. The English spoken human-computer dialogue interactive training method according to claim 2 is characterized by: The specific steps of S2 include: S21, dividing the digital signal in the cloud platform into a plurality of groups of short-time frames, and applying a Hanning window to each frame to perform windowing processing; S22. Based on the content of S21 and combined with Fourier transform technology, the time domain signal of each frame is converted into a frequency domain signal to generate a frequency domain representation for each frame, and the amplitude of each frequency component is obtained to form a spectrum diagram of each frame, and the spectrum diagram of each frame is processed through multiple groups of filters to extract frequency information that matches the auditory characteristics of the human ear, wherein the multiple groups of filters are evenly distributed on the Mel frequency axis, and each filter covers a frequency range, so as to divide the spectrum diagram of each frame into several frequency bands, and obtain the spectrum energy value S(f) in each frequency band.

4. The English spoken human-computer dialogue interactive training method according to claim 3 is characterized by: The specific steps of S2 also include: S23, taking the logarithm of the spectrum energy value S(f) obtained in S22 to simulate the perception characteristics of the human ear to sound, so as to obtain a logarithmic energy spectrum; S24, performing discrete cosine transform on the logarithmic energy spectrum to generate corresponding cepstrum coefficients Dpxs in each frame spectrum diagram, which are obtained specifically in the following manner: Where Dpxs n is represented by the nth cepstral coefficient, K represents the number of filters, k = 0, 1, 2, ..., K-1, k is the filter index; S(f k ) represents the spectrum energy value obtained after passing through the kth filter, n represents the index of the cepstral coefficient, and π represents a mathematical constant; log(S(f k )) is expressed as the logarithm of the spectrum energy value obtained after passing through the kth filter.

5. The English spoken human-computer dialogue interactive training method according to claim 4 is characterized by: The specific steps of S3 include: S31, based on S24, obtain the nth cepstral coefficient Dpxs n In this way, the corresponding cepstral coefficients Dpxs of each frame spectrum are obtained respectively, and after being summarized, the cepstral feature vector D of the corresponding frame spectrum is formed; S32. Input the inverse spectrum feature vector D of each frame spectrum graph obtained in S2 into the convolutional neural network. First, in the convolution layer, a convolution operation is performed on the inverse spectrum feature vector D of each frame spectrum graph to extract local features to generate a feature map, and the feature map is input to the pooling layer to downsample the feature map and retain important features. The downsampled feature map is then input to the fully connected layer to perform a fully connected operation and output the probability distribution of each phoneme. Finally, the probability distribution of each phoneme is input to the output layer, and the probability distribution of each phoneme is decoded in the output layer to generate a text sequence.

6. The English spoken human-computer dialogue interactive training method according to claim 5 is characterized by: The specific steps of S3 also include: S33, using natural language processing technology, preprocessing the text sequence, removing punctuation marks in the text sequence, and then dividing the text into several groups of sentences, and marking the part of speech of each word in the text sequence, and at the same time combining dependency syntax analysis technology to construct a syntax tree of the sentences in the text sequence to identify the syntactic structure, wherein in the syntax tree, the predicate verb in the sentence is used as the root node, and starting from the root node, the dependency relationship with the words other than the predicate verb in the sentence is established in sequence; S34, based on the syntax tree obtained in S33, and in combination with the preset reference sentence, analyze the similarity of the structural relationship between the user sentence and the reference sentence to construct the grammatical structure similarity Fxd, which is specifically obtained in the following manner: Where m represents the number of words in the corresponding sentence, i = 1, 2, 3, ..., m, Yjc i Represents the part of speech of the i-th word in the user sentence, Cjc i denoted as the part of speech of the i-th word in the reference sentence, and δ is denoted as the indicator function; S35. Based on the BERT model, the user sentence is divided into a number of word groups and characters to capture the bidirectional context information in the user sentence, and the user sentence is input into the BERT model to obtain the semantic vector Yx of the user sentence. Based on the semantic vector Yx of the user sentence, the semantic similarity between the user sentence and the reference sentence is analyzed to construct the semantic similarity Yxd, which is obtained specifically in the following manner: Where Cx represents the semantic vector of the reference sentence, ‖Yx‖ represents the modulus of the semantic vector of the user sentence, and ‖Cx‖ represents the modulus of the semantic vector of the reference sentence.

7. The English spoken human-computer dialogue interactive training method according to claim 1 is characterized by: The specific steps of S4 include: S41. After the human-computer dialogue interaction training is completed, a complete English spoken audio signal is obtained, and based on the complete English spoken audio signal, the fluency of the user's expression during this training process is analyzed to obtain the fluency Lyd, and the fluency Lyd is obtained by the following formula: Wherein, Ysz represents the speaking speed, Tdc represents the number of pauses, a1 and a2 are weight values, and P represents the first correction constant. The specific values ​​of a1 and a2 are set by the user according to the situation.

8. The English spoken human-computer dialogue interactive training method according to claim 7 is characterized by: The specific steps of S4 also include: S42, by associating the semantic similarity Yxd, the grammatical structure similarity Fxd and the fluency Lyd, and combining with the trained oral training evaluation model, after linear normalization, the corresponding data values ​​are mapped to the interval [0, 1], so as to fit and output the comprehensive evaluation coefficient Zpxs from the oral training evaluation model, which is specifically obtained in the following manner: In the formula, α and β are weight values, C is the second correction constant, and the specific values ​​of α and β are set by the user according to the situation.

9. The English spoken human-computer dialogue interactive training method according to claim 1, characterized in that: The specific steps of S5 include: S51, comparing and analyzing the qualified threshold W and the comprehensive evaluation coefficient Zpxs to comprehensively judge the English oral training situation of the user this time, and the specific judgment content is as follows: If the comprehensive evaluation coefficient Zpxs exceeds the qualified threshold W, it will be comprehensively judged that the user's English oral training is in a qualified state, and positive feedback will be given to the user, pointing out the user's advantages in oral fluency, grammatical structure and semantic expression, and regularly reminding the user to review the qualified English oral training content; If the comprehensive evaluation coefficient Zpxs does not exceed the qualified threshold W, it will be comprehensively judged that the user's English oral training is not in a qualified state, and the user will be fed back the performance differences in oral fluency, grammatical structure and semantic expression of this English oral training compared with the historical stage training. According to the performance differences, the user's training project will be arranged, and after the training, a phased test will be conducted again.

10. A system for training spoken English human-computer dialogue interaction, used to implement the method for training spoken English human-computer dialogue interaction according to any one of claims 1 to 9, characterized in that: It includes audio acquisition module, signal processing module, training analysis module, training evaluation module and feedback module; The audio acquisition module is used to monitor the user's spoken English audio signal in real time using a microphone during the user's English spoken human-computer dialogue interactive training, and input the English spoken audio signal to the cloud platform; The signal processing module is used to use speech recognition technology to perform Fourier transform on the English spoken audio signal in the cloud platform, and after feature extraction, obtain the spectrum of each frame, construct the corresponding cepstral coefficients Dpxs based on each frame spectrum, and after combination, form the cepstral feature vector D of the corresponding frame spectrum; The training and analysis module is used to capture the time series dependency of the English spoken audio signal by inputting the cepstrum feature vector D of each frame spectrogram into the convolutional neural network to generate a text sequence, and perform semantic and structural relationship analysis on the text sequence after text preprocessing to calculate and obtain the semantic similarity Yxd and grammatical structure similarity Fxd respectively; The training evaluation module is used to obtain a complete English spoken audio signal after the human-computer dialogue interaction training is completed, and based on the complete English spoken audio signal, analyze the fluency of the user's expression during this training process to obtain the fluency Lyd, and input the semantic similarity Yxd, grammatical structure similarity Fxd and fluency Lyd into the oral training evaluation model constructed according to the deep learning technology, and after linear normalization processing, fit and output the comprehensive evaluation coefficient Zpxs; The feedback module is used to pre-set a qualified threshold W, and compare the comprehensive evaluation coefficient Zpxs with the qualified threshold W to comprehensively judge the user's English speaking training situation this time, and adopt a personalized learning guide based on the judgment result.

Images