Teaching aids and methods for pronunciation training

Abstract

This invention discloses a teaching aid system and method for pronunciation training, relating to the field of intelligent assistance. It employs artificial intelligence technology based on deep neural network models to acquire pronunciation speech signals and their waveforms. Features are extracted using convolutional neural network models, and then spatial correlation is performed to obtain classification results indicating pronunciation accuracy. This provides objective feedback, helping students understand their pronunciation problems and enabling targeted training and improvement.

Description

Technical Field

[0001] This application relates to the field of intelligent assistance, and more specifically, to a teaching assistance system and method for pronunciation training. Background Technology

[0002] Accurate and fluent pronunciation is key to effective language communication. Students often make pronunciation errors and inaccuracies when practicing pronunciation and language expression, which affects their spoken expression and communication with others. However, current technology makes it difficult to accurately judge whether a student's pronunciation is correct and to provide proper correction so that they can express themselves clearly and reduce misunderstandings during communication.

[0003] Therefore, we look forward to an optimized teaching aid for pronunciation training. Summary of the Invention

[0004] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide a teaching aid system and method for pronunciation training. It employs artificial intelligence technology based on a deep neural network model to acquire the pronunciation speech signal and its waveform. Features are extracted using a convolutional neural network model, and then spatial correlation is performed to obtain a classification result representing the accuracy of pronunciation. This provides objective feedback, helping students understand their pronunciation problems and enabling targeted training and improvement.

[0005] According to one aspect of this application, a teaching aid system for pronunciation training is provided, comprising:

[0006] The speech signal acquisition module is used to acquire the speech signal and its waveform.

[0007] The frequency domain transformation module is used to perform Fourier transform on the spoken speech signal to obtain multiple frequency domain feature values;

[0008] The arrangement module is used to arrange the plurality of frequency domain feature values ​​into a frequency domain feature vector;

[0009] The speech domain feature extraction module is used to pass the frequency domain feature vector through a convolutional neural network model as a feature extractor to obtain the speech domain feature vector;

[0010] The pronunciation speech feature extraction module is used to pass the waveform of the pronunciation speech signal through a convolutional neural network model as a filter to obtain the pronunciation speech feature vector;

[0011] The spatial association module is used to perform feature space association between the audio domain feature vector and the pronunciation speech feature vector to obtain the classification feature vector;

[0012] The classification result module is used to pass the classification feature vector through a classifier to obtain a classification result, which is used to indicate whether the pronunciation is accurate.

[0013] In the aforementioned teaching aid system for pronunciation training, the speech domain feature extraction module is used to: perform the following operations on the input data during the forward propagation of the convolutional neural network model, which serves as the feature extractor: perform convolution processing on the input data based on a one-dimensional convolution kernel to obtain a convolutional feature map; perform pooling processing on the convolutional feature map based on the feature matrix to obtain a pooled feature map; and perform nonlinear activation on the pooled feature map to obtain an activation feature map; wherein, the input of the first layer of the convolutional neural network model serving as the feature extractor is the frequency domain feature vector, and the output of the last layer of the convolutional neural network model serving as the feature extractor is the speech domain feature vector.

[0014] In the aforementioned teaching aid system for pronunciation training, the pronunciation speech feature extraction module is configured to: use each layer of the convolutional neural network model acting as a filter to perform the following operations on the input data during the forward propagation of the layer: perform convolution processing on the input data based on the convolution kernel to obtain a convolutional feature map; perform global mean pooling on each feature matrix of the channel dimension of the convolutional feature map to obtain a pooled feature map; and perform nonlinear activation on the feature values ​​at each position in the pooled feature map to generate an activation feature map; wherein, the output of the last layer of the convolutional neural network acting as a filter is the pronunciation speech feature vector, the input of the second to last layers of the convolutional neural network acting as a filter is the output of the previous layer, and the input of the first layer of the convolutional neural network acting as a filter is the pronunciation speech signal waveform.

[0015] In the aforementioned teaching aid system for pronunciation training, the spatial association module includes: an inner product calculation unit for calculating the inner product between the speech domain feature vector and the pronunciation speech feature vector; a basis construction unit for using the speech domain feature vector and the pronunciation speech feature vector as the basis of the classification feature vector in response to the inner product between the speech domain feature vector and the pronunciation speech feature vector being zero; a Schmitt orthogonalization unit for performing Schmitt orthogonalization on the speech domain feature vector and the pronunciation speech feature vector to obtain orthogonal first unit feature vector and second unit feature vector in response to the inner product between the speech domain feature vector and the pronunciation speech feature vector being non-zero; and a fusion unit for fusing the first unit feature vector and the second unit feature vector with error minimization to obtain the classification feature vector.

[0016] In the aforementioned teaching aid system for pronunciation training, the classification result module includes: a fully connected encoding unit, used to perform fully connected encoding on the classification feature vector using the fully connected layer of the classifier to obtain an encoded classification feature vector; and a classification result generation unit, used to input the encoded classification feature vector into the Softmax classification function of the classifier to obtain the classification result.

[0017] According to another aspect of this application, a teaching aid method for pronunciation training is provided, comprising:

[0018] Obtain the pronunciation speech signal and its waveform;

[0019] Perform a Fourier transform on the spoken speech signal to obtain multiple frequency domain feature values;

[0020] Arrange the plurality of frequency domain feature values ​​into a frequency domain feature vector;

[0021] The frequency domain feature vector is passed through a convolutional neural network model as a feature extractor to obtain the speech domain feature vector;

[0022] The waveform of the spoken speech signal is passed through a convolutional neural network model as a filter to obtain the spoken speech feature vector;

[0023] To obtain classification feature vectors, feature space correlation is performed between the audio domain feature vectors and the pronunciation speech feature vectors.

[0024] The classification feature vector is passed through a classifier to obtain a classification result, which is used to indicate whether the pronunciation is accurate.

[0025] Compared with existing technologies, this application provides a teaching aid system and method for pronunciation training. It employs artificial intelligence technology based on deep neural network models to acquire pronunciation speech signals and their waveforms. Features are extracted using convolutional neural network models, and then spatial correlation is performed to obtain classification results indicating whether pronunciation is accurate. This provides objective feedback, helping students understand their pronunciation problems and enabling targeted training and improvement. Attached Figure Description

[0026] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0027] Figure 1 This is a block diagram of a teaching aid system for pronunciation training according to an embodiment of this application.

[0028] Figure 2 This is a schematic diagram of the architecture of a teaching aid system for pronunciation training according to an embodiment of this application.

[0029] Figure 3 This is a block diagram of a spatial association module in a teaching aid system for pronunciation training according to an embodiment of this application.

[0030] Figure 4 This is a block diagram of the classification result module in a teaching aid system for pronunciation training according to an embodiment of this application.

[0031] Figure 5 This is a flowchart of a teaching aid method for pronunciation training according to an embodiment of this application.

[0032] Figure 6 This is a block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0033] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0034] Application Overview

[0035] As mentioned above, accurate pronunciation is a crucial factor in language communication and expression. When students practice pronunciation, inaccurate pronunciation and varying speaking speeds can lead to difficulties in expressing themselves clearly and cause misunderstandings. However, current technology cannot accurately provide feedback on students' pronunciation training to determine accuracy and thus facilitate correction and targeted training. Therefore, there is a need for an optimized teaching aid solution for pronunciation training.

[0036] To address the aforementioned technical issues, the applicant of this application obtains the pronunciation speech signal and the waveform of the pronunciation speech signal, extracts features through a convolutional neural network model, and then performs spatial correlation to obtain a classification result that indicates whether the pronunciation is accurate.

[0037] Accordingly, in the technical solution of this application, considering the pronunciation speech signal, the sound characteristics of the pronunciation, such as pitch, volume, and tone, can be analyzed to assess the accuracy of pronunciation. Simultaneously, the waveform of the pronunciation speech signal allows observation of the temporal characteristics of the sound, such as its fluctuations and duration, thus providing a more comprehensive understanding of the pronunciation performance. Therefore, by acquiring the speech signal and waveform, accurate pronunciation feedback and personalized pronunciation training can be provided to students.

[0038] In recent years, deep learning and neural networks have been widely applied in fields such as computer vision, natural language processing, and text signal processing. Furthermore, deep learning and neural networks have demonstrated near-human or even superior performance in areas such as image classification, object detection, semantic segmentation, and text translation.

[0039] Specifically, in the technical solution of this application, firstly, the spoken speech signal and its waveform are acquired. Considering that Fourier transform can decompose a signal into a superposition of sine and cosine functions of different frequencies, these sine and cosine functions are called frequency domain components or the spectrum. Specifically, through Fourier transform, the energy distribution of the spoken speech signal at different frequencies can be obtained. These frequency domain features can provide more detailed information about the sound characteristics of the pronunciation, such as pitch, formants, harmonics, etc. By analyzing these features, the accuracy of pronunciation can be more accurately assessed, and targeted pronunciation training suggestions can be provided. Furthermore, Fourier transform can also help remove noise or interference from the spoken speech signal, improving the accuracy of pronunciation assessment and training. By removing irrelevant frequency domain components, the focus can be better on features related to pronunciation. Therefore, Fourier transform is performed on the spoken speech signal to obtain multiple frequency domain feature values ​​to extract the frequency domain features of the spoken speech signal, providing students with more comprehensive and accurate pronunciation training and feedback.

[0040] Next, by arranging frequency domain feature values ​​into feature vectors, we can organize different frequency domain features together to form a more dimensional data structure. Such a data structure is better understood and processed by machine learning algorithms. Feature vectors can be used as input to models such as convolutional neural networks for feature encoding and classification. The arrangement of feature vectors also helps preserve the relationships and contextual information between features. In this technical solution, in pronunciation-assisted teaching, there may be certain correlations between different frequency domain features; the arrangement of feature vectors can better capture these correlations and provide richer information. Furthermore, arranging frequency domain feature values ​​into feature vectors facilitates feature standardization and normalization, ensuring scale consistency between different features and preventing certain features from having an excessive impact on model training. Therefore, arranging multiple frequency domain feature values ​​into frequency domain feature vectors helps provide a more structured, consistent, and information-rich data representation to support subsequent feature encoding and classification tasks.

[0041] Then, considering the frequency domain features of speech signals, convolutional neural networks (CNNs) can effectively capture local and global frequency domain feature patterns and extract useful information. Specifically, through multiple layers of convolution and pooling operations, CNNs can gradually extract abstract features at different levels, from low-level local features to high-level global features. These features can be used to distinguish different speech pronunciation features, such as pitch, timbre, and resonance. By training a CNN, it can automatically learn the optimal feature representation, thereby improving the accuracy and robustness of pronunciation accuracy assessment. Furthermore, CNNs have parallel computing capabilities, enabling them to efficiently process large-scale speech data. This is crucial for pronunciation-assisted teaching and training with high real-time requirements. By using frequency domain feature vectors as input, CNNs can quickly evaluate and provide feedback on pronunciation, offering immediate pronunciation training results.

[0042] Therefore, by using the frequency domain feature vectors through a convolutional neural network model as a feature extractor to obtain speech domain feature vectors, the accuracy, real-time performance, and adaptability of pronunciation accuracy assessment can be effectively improved, providing students with accurate pronunciation feedback and personalized pronunciation training, thereby helping to improve their language communication skills.

[0043] Next, considering that the waveform is a time-domain representation of the speech signal, it contains information about the amplitude and temporal variations of the speech signal. However, directly using the waveform as input for pronunciation accuracy assessment presents several challenges. First, the waveform has a high dimensionality, requiring significant computational resources and storage space. Second, noise and redundant information in the waveform may interfere with the extraction of pronunciation features and accuracy assessment. In this technical solution, the above problems can be solved by using a convolutional neural network (CNN) to extract features from the waveform. The CNN can automatically learn abstract features from the waveform, such as speech frequency, intonation, and phonemes. Through multiple layers of convolution and pooling operations, the network can gradually extract higher-level features, thereby reducing the dimensionality of the features and removing redundant information. These feature vectors can better represent the key features of pronunciation, such as the articulation position, duration, and intensity of phonemes.

[0044] Furthermore, convolutional neural networks can be trained to adapt to different pronunciation samples and individual differences, improving the adaptability of pronunciation accuracy assessment. By training on a large number of pronunciation samples, the network can learn general rules of pronunciation and adjust according to individual differences, thereby more accurately assessing the correctness of pronunciation. Therefore, by passing the waveform of the pronunciation speech signal through a convolutional neural network model as a filter to obtain the pronunciation speech feature vector, useful speech features can be extracted, redundant information can be reduced, and the accuracy and adaptability of pronunciation accuracy assessment can be improved.

[0045] Then, the speech domain feature vector and the pronunciation speech feature vector are correlated to obtain a classification feature vector. Considering that in pronunciation-assisted teaching and training, the speech domain feature vector and the pronunciation speech feature vector are two different feature representation methods, the speech domain feature vector can capture the spectral information of the speech signal, while the pronunciation speech feature vector can represent specific characteristic patterns of pronunciation. By correlating these two different feature representation methods, more comprehensive and richer feature information can be obtained. The purpose of correlation is to fuse relevant information from the two feature representation methods to better describe the characteristics and patterns of pronunciation. Specifically, through correlation, the speech domain feature vector and the pronunciation speech feature vector can be combined to form a classification feature vector. This classification feature vector can contain more feature information and can more accurately describe the characteristics and patterns of pronunciation. Based on this classification feature vector, pronunciation accuracy assessment and pronunciation training can be performed, providing accurate pronunciation feedback and personalized pronunciation training. Therefore, correlating the speech domain feature vector and the pronunciation speech feature vector to obtain a classification feature vector can improve the accuracy of pronunciation accuracy assessment. Further, the classification feature vector is passed through a classifier to obtain a classification result, which is used to indicate whether the pronunciation is accurate.

[0046] Specifically, the audio domain feature vector and the pronunciation speech feature vector are extracted from game features at different scales or perspectives. By associating and fusing them, multi-scale feature representations can be obtained, effectively capturing the changes and correlations of game features at different scales. This helps improve the robustness and generalization ability of the features, thereby enhancing classification performance. Furthermore, considering that the audio domain feature vector and the pronunciation speech feature vector represent different semantic information, by associating these two feature vectors in the feature space, the different information they contain can be comprehensively utilized, resulting in richer and more comprehensive classification feature vectors. Such comprehensive features can better reflect the potential characteristics of the game and the correlation of features.

[0047] To obtain a classification feature vector, the method involves associating the feature space of the speech domain feature vector and the pronunciation speech feature vector, including: calculating the inner product between the speech domain feature vector and the pronunciation speech feature vector; using the speech domain feature vector and the pronunciation speech feature vector as the basis for the classification feature vector when the inner product is zero; performing Schmitt orthogonalization on the speech domain feature vector and the pronunciation speech feature vector when the inner product is not zero to obtain an orthogonal first unit feature vector and a second unit feature vector; and fusing the first unit feature vector and the second unit feature vector by minimizing the error to obtain the classification feature vector.

[0048] The feature space correlation between the audio domain feature vector and the pronunciation speech feature vector is performed to analyze and characterize the spatial correlation structure and characteristics between the feature vectors using the inner product between the feature vectors. When there is no normalized structural correlation between the feature vectors, orthogonal unit feature vectors are constructed by Schmitt orthogonalization. The first unit feature vector and the second unit feature vector are fused based on error minimization to obtain the classification feature vector. In this way, the main features and structure of the source domain feature vectors are preserved and the feature correlation pattern features between the feature vectors are used to enhance the fusion to capture the main and secondary change trends of the data, thereby reducing information loss.

[0049] Based on this, this application provides a teaching aid system for pronunciation training, comprising: a pronunciation speech signal acquisition module for acquiring a pronunciation speech signal and a waveform of the pronunciation speech signal; a domain transformation module for performing a Fourier transform on the pronunciation speech signal to obtain multiple frequency domain feature values; an arrangement module for arranging the multiple frequency domain feature values ​​into a frequency domain feature vector; a speech domain feature extraction module for passing the frequency domain feature vector through a convolutional neural network model as a feature extractor to obtain a speech domain feature vector; a pronunciation speech feature extraction module for passing the waveform of the pronunciation speech signal through a convolutional neural network model as a filter to obtain a pronunciation speech feature vector; a spatial association module for performing feature space association between the speech domain feature vector and the pronunciation speech feature vector to obtain a classification feature vector; and a classification result module for passing the classification feature vector through a classifier to obtain a classification result, wherein the classification result is used to indicate whether the pronunciation is accurate.

[0050] After introducing the basic principles of this application, various non-limiting embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0051] Exemplary System

[0052] Figure 1 This is a block diagram of a teaching aid system for pronunciation training according to an embodiment of this application. Figure 1 As shown, the teaching aid system 100 for pronunciation training according to an embodiment of this application includes: a pronunciation speech signal acquisition module 110, used to acquire a pronunciation speech signal and a waveform of the pronunciation speech signal; a domain transformation module 120, used to perform a Fourier transform on the pronunciation speech signal to obtain multiple frequency domain feature values; an arrangement module 130, used to arrange the multiple frequency domain feature values ​​into a frequency domain feature vector; a speech domain feature extraction module 140, used to pass the frequency domain feature vector through a convolutional neural network model as a feature extractor to obtain a speech domain feature vector; a pronunciation speech feature extraction module 150, used to pass the waveform of the pronunciation speech signal through a convolutional neural network model as a filter to obtain a pronunciation speech feature vector; a spatial association module 160, used to perform feature space association between the speech domain feature vector and the pronunciation speech feature vector to obtain a classification feature vector; and a classification result module 170, used to pass the classification feature vector through a classifier to obtain a classification result, the classification result being used to indicate whether the pronunciation is accurate.

[0053] Figure 2 This is a schematic diagram of the architecture of a teaching aid system for pronunciation training according to an embodiment of this application. Figure 2As shown, firstly, the spoken speech signal and its waveform are acquired. Next, a Fourier transform is performed on the spoken speech signal to obtain multiple frequency domain feature values. Then, these frequency domain feature values ​​are arranged into a frequency domain feature vector. Next, the frequency domain feature vector is passed through a convolutional neural network model acting as a feature extractor to obtain a speech domain feature vector. Then, the waveform of the spoken speech signal is passed through a convolutional neural network model acting as a filter to obtain a spoken speech feature vector. Next, feature space correlation is performed between the speech domain feature vector and the spoken speech feature vector to obtain a classification feature vector. Finally, the classification feature vector is passed through a classifier to obtain a classification result, which indicates whether the pronunciation is accurate.

[0054] In this embodiment, the pronunciation speech signal acquisition module 110 is used to acquire the pronunciation speech signal and its waveform. Considering the pronunciation speech signal, the sound characteristics of the pronunciation, such as pitch, volume, and tone, can be analyzed to assess the accuracy of the pronunciation. Simultaneously, the waveform of the pronunciation speech signal allows observation of the temporal characteristics of the sound, such as its fluctuations and duration, thus providing a more comprehensive understanding of the pronunciation performance. Therefore, by acquiring the speech signal and waveform, accurate pronunciation feedback and personalized pronunciation training can be provided to students.

[0055] In this embodiment, the domain transformation module 120 is used to perform a Fourier transform on the spoken speech signal to obtain multiple frequency domain feature values. Considering that the Fourier transform can decompose a signal into a superposition of sine and cosine functions of different frequencies, these sine and cosine functions are called frequency domain components or the spectrum. Specifically, through the Fourier transform, the energy distribution of the spoken speech signal at different frequencies can be obtained. These frequency domain features can provide more detailed information about the sound characteristics of the pronunciation, such as pitch, formants, harmonics, etc. By analyzing these features, the accuracy of pronunciation can be more accurately assessed, and targeted pronunciation training suggestions can be provided. Furthermore, the Fourier transform can also help remove noise or interference from the spoken speech signal, improving the accuracy of pronunciation assessment and training. By removing irrelevant frequency domain components, the focus can be better on features related to pronunciation. Therefore, performing a Fourier transform on the spoken speech signal to obtain multiple frequency domain feature values ​​extracts the frequency domain features of the spoken speech signal and provides students with more comprehensive and accurate pronunciation training and feedback.

[0056] In this embodiment, the arrangement module 130 is used to arrange the multiple frequency domain feature values ​​into a frequency domain feature vector. By arranging the frequency domain feature values ​​into feature vectors, we can organize different frequency domain features together to form a more dimensional data structure. Such a data structure can be better understood and processed by machine learning algorithms. The feature vectors can be used as input to models such as convolutional neural networks for feature encoding and classification. The arrangement of feature vectors also helps preserve the relationships and contextual information between features. In this technical solution, in pronunciation-assisted teaching, there may be certain correlations between different frequency domain features. The arrangement of feature vectors can better capture these correlations and provide richer information. Furthermore, arranging the frequency domain feature values ​​into feature vectors facilitates feature standardization and normalization to ensure scale consistency between different features and avoid excessive influence of certain features on model training. Therefore, arranging multiple frequency domain feature values ​​into frequency domain feature vectors helps provide a more structured, consistent, and information-rich data representation to support subsequent feature encoding and classification tasks.

[0057] In this embodiment, the speech domain feature extraction module 140 is used to obtain a speech domain feature vector by passing the frequency domain feature vector through a convolutional neural network model, which acts as a feature extractor. Then, considering the frequency domain features of the speech signal, the convolutional neural network can effectively capture local and global frequency domain feature patterns and extract useful information. Specifically, through multi-layer convolution and pooling operations, the convolutional neural network can gradually extract abstract features at different levels, from low-level local features to high-level global features. These features can be used to distinguish different speech pronunciation features, such as pitch, timbre, and resonance. By training the convolutional neural network, it can automatically learn the optimal feature representation, thereby improving the accuracy and robustness of pronunciation accuracy assessment. Furthermore, the convolutional neural network also has parallel computing capabilities, enabling efficient processing of large-scale speech data. This is crucial for pronunciation-assisted teaching and training with high real-time requirements. By using the frequency domain feature vector as input, the convolutional neural network can quickly evaluate and provide feedback on pronunciation, offering immediate pronunciation training results. Therefore, by using the frequency domain feature vectors through a convolutional neural network model as a feature extractor to obtain speech domain feature vectors, the accuracy, real-time performance, and adaptability of pronunciation accuracy assessment can be effectively improved, providing students with accurate pronunciation feedback and personalized pronunciation training, thereby helping to improve their language communication skills.

[0058] Specifically, in this embodiment, the speech domain feature extraction module is used to: use each layer of the convolutional neural network model, which serves as the feature extractor, to perform the following operations on the input data during the forward propagation of the layers: perform convolution processing on the input data based on a one-dimensional convolution kernel to obtain a convolutional feature map; perform pooling processing on the convolutional feature map based on the feature matrix to obtain a pooled feature map; and perform nonlinear activation on the pooled feature map to obtain an activation feature map; wherein, the input of the first layer of the convolutional neural network model, which serves as the feature extractor, is the frequency domain feature vector, and the output of the last layer of the convolutional neural network model, which serves as the feature extractor, is the speech domain feature vector.

[0059] In this embodiment, the pronunciation speech feature extraction module 150 is used to pass the waveform of the pronunciation speech signal through a convolutional neural network model as a filter to obtain pronunciation speech feature vectors. Considering that the waveform is a time-domain representation of the speech signal, it contains information about the amplitude and temporal variations of the speech signal. However, directly using the waveform as input for pronunciation accuracy evaluation presents several challenges. First, the waveform has a high dimensionality, requiring significant computational resources and storage space. Second, noise and redundant information in the waveform may interfere with the extraction of pronunciation features and accuracy evaluation. In this technical solution, the above problems can be solved by extracting features from the waveform through a convolutional neural network. The convolutional neural network can automatically learn abstract features from the waveform, such as speech frequency, intonation, and phonemes. Through multi-layer convolution and pooling operations, the network can gradually extract higher-level features, thereby reducing the dimensionality of the features and removing redundant information. These feature vectors can better represent key pronunciation features, such as the pronunciation position, duration, and intensity of phonemes. Furthermore, the convolutional neural network can be trained to adapt to different pronunciation samples and individual differences, improving the adaptability of pronunciation accuracy evaluation. By training on a large number of pronunciation samples, the network can learn general rules of pronunciation and adjust according to individual differences, thereby more accurately assessing the correctness of pronunciation. Therefore, by passing the waveform of the pronunciation speech signal through a convolutional neural network model as a filter to obtain the pronunciation speech feature vector, useful speech features can be extracted, redundant information can be reduced, and the accuracy and adaptability of pronunciation accuracy assessment can be improved.

[0060] Specifically, in this embodiment, the speech feature extraction module is configured to: use each layer of the convolutional neural network model serving as a filter to perform the following operations on the input data during the forward propagation of the layer: perform convolution processing on the input data based on the convolution kernel to obtain a convolutional feature map; perform global mean pooling on each feature matrix of the channel dimension of the convolutional feature map to obtain a pooled feature map; and perform nonlinear activation on the feature values ​​at each position in the pooled feature map to generate an activation feature map; wherein, the output of the last layer of the convolutional neural network serving as a filter is the speech feature vector, the input of the second to last layers of the convolutional neural network serving as a filter is the output of the previous layer, and the input of the first layer of the convolutional neural network serving as a filter is the speech signal waveform.

[0061] In this embodiment of the application, the spatial association module 160 is used to perform feature space association between the audio domain feature vector and the pronunciation speech feature vector to obtain the classification feature vector.

[0062] Specifically, the audio domain feature vector and the pronunciation speech feature vector are extracted from game features at different scales or perspectives. By associating and fusing them, multi-scale feature representations can be obtained, effectively capturing the changes and correlations of game features at different scales. This helps improve the robustness and generalization ability of the features, thereby enhancing classification performance. Furthermore, considering that the audio domain feature vector and the pronunciation speech feature vector represent different semantic information, by associating these two feature vectors in the feature space, the different information they contain can be comprehensively utilized, resulting in richer and more comprehensive classification feature vectors. Such comprehensive features can better reflect the potential characteristics of the game and the correlation of features.

[0063] Figure 3 This is a block diagram of a spatial association module in a teaching aid system for pronunciation training according to an embodiment of this application. Specifically, in the embodiments of this application, as... Figure 3As shown, the spatial association module 160 includes: an inner product calculation unit 161, used to calculate the inner product between the audio domain feature vector and the pronunciation speech feature vector; a basis construction unit 162, used to use the audio domain feature vector and the pronunciation speech feature vector as the basis of the classification feature vector in response to the inner product between the audio domain feature vector and the pronunciation speech feature vector being zero; a Schmitt orthogonalization unit 163, used to perform Schmitt orthogonalization on the audio domain feature vector and the pronunciation speech feature vector in response to the inner product between the audio domain feature vector and the pronunciation speech feature vector being non-zero to obtain an orthogonal first unit feature vector and a second unit feature vector; and a fusion unit 164, used to fuse the first unit feature vector and the second unit feature vector with error minimization to obtain the classification feature vector.

[0064] More specifically, in the embodiments of this application, the error minimization includes: in some cases, minimizing the error between the classification feature vector and the original feature vector using the classification feature vector. The weights can be determined using least squares or other optimization methods to minimize the error between the classification feature vector and the original feature vector.

[0065] The feature space correlation between the audio domain feature vector and the pronunciation speech feature vector is performed to analyze and characterize the spatial correlation structure and characteristics between the feature vectors using the inner product between the feature vectors. When there is no normalized structural correlation between the feature vectors, orthogonal unit feature vectors are constructed by Schmitt orthogonalization. The first unit feature vector and the second unit feature vector are fused based on error minimization to obtain the classification feature vector. In this way, the main features and structure of the source domain feature vectors are preserved and the feature correlation pattern features between the feature vectors are used to enhance the fusion to capture the main and secondary change trends of the data, thereby reducing information loss.

[0066] In this embodiment of the application, the classification result module 170 is used to pass the classification feature vector through a classifier to obtain a classification result, which is used to indicate whether the pronunciation is accurate.

[0067] Figure 4 This is a block diagram of a classification result module in a teaching aid system for pronunciation training according to an embodiment of this application. Specifically, in an embodiment of this application, as... Figure 4 As shown, the classification result module 170 includes: a fully connected encoding unit 171, used to perform fully connected encoding on the classification feature vector using the fully connected layer of the classifier to obtain an encoded classification feature vector; and a classification result generation unit 172, used to input the encoded classification feature vector into the Softmax classification function of the classifier to obtain the classification result.

[0068] In summary, the teaching aid system 100 for pronunciation training based on the embodiments of this application is explained. It employs artificial intelligence technology based on a deep neural network model to acquire the pronunciation speech signal and its waveform, extracts features using a convolutional neural network model, and then performs spatial correlation to obtain a classification result indicating whether the pronunciation is accurate. This provides objective feedback, helping students understand their pronunciation problems and enabling targeted training and improvement.

[0069] Exemplary methods

[0070] Figure 5 This is a flowchart of a teaching aid method for pronunciation training according to an embodiment of this application. Figure 5 As shown, the teaching aid method for pronunciation training according to an embodiment of this application includes: S110, acquiring a pronunciation speech signal and a waveform of the pronunciation speech signal; S120, performing a Fourier transform on the pronunciation speech signal to obtain multiple frequency domain feature values; S130, arranging the multiple frequency domain feature values ​​into a frequency domain feature vector; S140, passing the frequency domain feature vector through a convolutional neural network model as a feature extractor to obtain a speech domain feature vector; S150, passing the waveform of the pronunciation speech signal through a convolutional neural network model as a filter to obtain a pronunciation speech feature vector; S160, performing feature space correlation between the speech domain feature vector and the pronunciation speech feature vector to obtain a classification feature vector; and S170, passing the classification feature vector through a classifier to obtain a classification result, the classification result being used to indicate whether the pronunciation is accurate.

[0071] In one example, in the above-described teaching aid method for pronunciation training, the frequency domain feature vector is processed by a convolutional neural network model acting as a feature extractor to obtain a speech domain feature vector. This includes: using each layer of the convolutional neural network model acting as a feature extractor to process the input data in the forward pass of the layer, performing the following operations on the input data: performing convolution processing based on a one-dimensional convolutional kernel on the input data to obtain a convolutional feature map; performing pooling processing based on a feature matrix on the convolutional feature map to obtain a pooled feature map; and performing nonlinear activation on the pooled feature map to obtain an activation feature map. The input of the first layer of the convolutional neural network model acting as a feature extractor is the frequency domain feature vector, and the output of the last layer of the convolutional neural network model acting as a feature extractor is the speech domain feature vector.

[0072] In one example, in the above-described teaching aid method for pronunciation training, the waveform of the pronunciation speech signal is passed through a convolutional neural network model acting as a filter to obtain a pronunciation speech feature vector. This includes: using each layer of the convolutional neural network model acting as a filter to perform the following on the input data during the forward propagation of the layer: performing convolution processing on the input data based on the convolution kernel to obtain a convolutional feature map; performing global mean pooling on each feature matrix of the channel dimension of the convolutional feature map to obtain a pooled feature map; and performing nonlinear activation on the feature values ​​at each position in the pooled feature map to generate an activation feature map. The output of the last layer of the convolutional neural network acting as a filter is the pronunciation speech feature vector, the input of the second to last layers of the convolutional neural network acting as a filter is the output of the previous layer, and the input of the first layer of the convolutional neural network acting as a filter is the waveform of the pronunciation speech signal.

[0073] Here, those skilled in the art will understand that the specific operations of each step in the above-described teaching aids for pronunciation training have been referenced above. Figures 1 to 4 The description of the teaching aid system for pronunciation training is detailed therein, and therefore, its repeated description will be omitted.

[0074] Exemplary electronic devices

[0075] Below, for reference Figure 6 This describes an electronic device according to embodiments of the present application. Figure 6 This is a block diagram of an electronic device according to an embodiment of this application. Figure 6 As shown, the electronic device 10 includes one or more processors 11 and memory 12.

[0076] The processor 11 may be a central processing module (CPU) or other form of processing module with data processing and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0077] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the functions and / or other desired functions in the teaching aid system and method for pronunciation training described in the various embodiments of this application above. Various contents such as pronunciation speech signals and waveforms of pronunciation speech signals may also be stored in the computer-readable storage medium.

[0078] In one example, the electronic device 10 may also include an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0079] The input device 13 may include, for example, a keyboard, a mouse, etc.

[0080] The output device 14 can output various information to the outside, including classification results. The output device 14 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0081] Of course, for the sake of simplicity, Figure 6 Only some of the components of the electronic device 10 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 10 may include any other suitable components depending on the specific application.

[0082] Exemplary computer program products and computer-readable storage media

[0083] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps of the functions in the teaching aid methods for pronunciation training according to various embodiments of this application described in the "Exemplary Methods" section above.

[0084] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0085] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform steps of the functions in the teaching aid methods for pronunciation training according to various embodiments of this application described in the "Exemplary Methods" section above.

[0086] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0087] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0088] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0089] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0090] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0091] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

[0092] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0093] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0094] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0095] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0096] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A teaching aid system for pronunciation training, characterized in that, include: The speech signal acquisition module is used to acquire the speech signal and its waveform. The frequency domain transformation module is used to perform Fourier transform on the spoken speech signal to obtain multiple frequency domain feature values; The arrangement module is used to arrange the plurality of frequency domain feature values ​​into a frequency domain feature vector; The speech domain feature extraction module is used to pass the frequency domain feature vector through a convolutional neural network model as a feature extractor to obtain the speech domain feature vector; The pronunciation speech feature extraction module is used to pass the waveform of the pronunciation speech signal through a convolutional neural network model as a filter to obtain the pronunciation speech feature vector; The spatial association module is used to perform feature space association between the audio domain feature vector and the pronunciation speech feature vector to obtain the classification feature vector; The classification result module is used to pass the classification feature vector through a classifier to obtain a classification result, which is used to indicate whether the pronunciation is accurate. The spatial association module includes: The inner product calculation unit is used to calculate the inner product between the audio domain feature vector and the pronunciation speech feature vector; A basis construction unit is used to use the audio domain feature vector and the pronunciation speech feature vector as the basis of the classification feature vector in response to the inner product between the audio domain feature vector and the pronunciation speech feature vector being zero. The Schmitt orthogonalization unit is used to perform Schmitt orthogonalization on the speech domain feature vector and the speech feature vector in response to the inner product between the speech domain feature vector and the speech feature vector being non-zero, so as to obtain an orthogonal first unit feature vector and a second unit feature vector. A fusion unit is used to fuse the first unit feature vector and the second unit feature vector to minimize the error and obtain the classification feature vector.

2. The teaching aid system for pronunciation training according to claim 1, characterized in that, The speech domain feature extraction module is used for: Each layer of the convolutional neural network model, which uses the aforementioned feature extractor, processes the input data during the forward propagation of the layer as follows: The input data is subjected to convolution processing based on a one-dimensional convolution kernel to obtain a convolutional feature map; The convolutional feature map is subjected to pooling based on the feature matrix to obtain a pooled feature map; The pooled feature map is nonlinearly activated to obtain an activated feature map; The input to the first layer of the convolutional neural network model, which serves as the feature extractor, is the frequency domain feature vector, and the output of the last layer of the convolutional neural network model, which serves as the feature extractor, is the speech domain feature vector.

3. The teaching aid system for pronunciation training according to claim 2, characterized in that, The pronunciation speech feature extraction module is used for: Each layer of the convolutional neural network model, which uses the aforementioned filter, processes the input data during the forward propagation of the layer as follows: The input data is convolved using convolution kernels to obtain a convolutional feature map. Global mean pooling is performed on each feature matrix of the channel dimension of the convolutional feature map to obtain a pooled feature map; Nonlinear activation is applied to the feature values ​​at each position in the pooled feature map to generate an activated feature map; Wherein, the output of the last layer of the convolutional neural network that serves as a filter is the speech feature vector, the input of the second to last layers of the convolutional neural network that serves as a filter is the output of the previous layer, and the input of the first layer of the convolutional neural network that serves as a filter is the waveform of the speech signal.

4. The teaching aid system for pronunciation training according to claim 3, characterized in that, The classification result module includes: A fully connected encoding unit is used to perform fully connected encoding on the classification feature vector using the fully connected layer of the classifier to obtain an encoded classification feature vector; The classification result generation unit is used to input the encoded classification feature vector into the Softmax classification function of the classifier to obtain the classification result.

5. A teaching aid method for pronunciation training, characterized in that, include: Obtain the pronunciation speech signal and its waveform; Perform a Fourier transform on the spoken speech signal to obtain multiple frequency domain feature values; Arrange the plurality of frequency domain feature values ​​into a frequency domain feature vector; The frequency domain feature vector is passed through a convolutional neural network model as a feature extractor to obtain the speech domain feature vector; The waveform of the spoken speech signal is passed through a convolutional neural network model as a filter to obtain the spoken speech feature vector; To obtain classification feature vectors, feature space correlation is performed between the audio domain feature vectors and the pronunciation speech feature vectors. The classification feature vector is passed through a classifier to obtain a classification result, which is used to indicate whether the pronunciation is accurate. Specifically, the feature space correlation between the speech audio domain feature vector and the pronunciation speech feature vector is performed to obtain the classification feature vector, including: Calculate the inner product between the audio domain feature vector and the pronunciation speech feature vector; In response to the inner product between the audio domain feature vector and the pronunciation speech feature vector being zero, the audio domain feature vector and the pronunciation speech feature vector are used as the basis of the classification feature vector; In response to the fact that the inner product between the speech domain feature vector and the pronunciation speech feature vector is not zero, the speech domain feature vector and the pronunciation speech feature vector are subjected to Schmitt orthogonalization to obtain an orthogonal first unit feature vector and second unit feature vector; The first unit feature vector and the second unit feature vector are fused to minimize the error in order to obtain the classification feature vector.

6. The teaching aid method for pronunciation training according to claim 5, characterized in that, The frequency domain feature vector is processed through a convolutional neural network model as a feature extractor to obtain the speech domain feature vector, including: Each layer of the convolutional neural network model, which uses the aforementioned feature extractor, processes the input data during the forward propagation of the layer as follows: The input data is subjected to convolution processing based on a one-dimensional convolution kernel to obtain a convolutional feature map; The convolutional feature map is subjected to pooling based on the feature matrix to obtain a pooled feature map; The pooled feature map is nonlinearly activated to obtain an activated feature map; The input to the first layer of the convolutional neural network model, which serves as the feature extractor, is the frequency domain feature vector, and the output of the last layer of the convolutional neural network model, which serves as the feature extractor, is the speech domain feature vector.

7. The teaching aid method for pronunciation training according to claim 6, characterized in that, The waveform of the spoken speech signal is passed through a convolutional neural network model as a filter to obtain the spoken speech feature vector, including: Each layer of the convolutional neural network model, which uses the aforementioned filter, processes the input data during the forward propagation of the layer as follows: The input data is convolved using convolution kernels to obtain a convolutional feature map. Global mean pooling is performed on each feature matrix of the channel dimension of the convolutional feature map to obtain a pooled feature map; Nonlinear activation is applied to the feature values ​​at each position in the pooled feature map to generate an activated feature map; Wherein, the output of the last layer of the convolutional neural network that serves as a filter is the speech feature vector, the input of the second to last layers of the convolutional neural network that serves as a filter is the output of the previous layer, and the input of the first layer of the convolutional neural network that serves as a filter is the waveform of the speech signal.

8. The teaching aid method for pronunciation training according to claim 7, characterized in that, The classification feature vector is passed through a classifier to obtain a classification result, which is used to indicate whether the pronunciation is accurate, including: The classifier's fully connected layer is used to encode the classification feature vector to obtain the encoded classification feature vector. The encoded classification feature vector is input into the Softmax classification function of the classifier to obtain the classification result.

Images