Spoken English pronunciation correction auxiliary system based on speech recognition

Abstract

The invention relates to the field of intelligent speech recognition, in particular to a spoken English pronunciation correction auxiliary system based on speech recognition, which comprises a data acquisition module, a speech recognition module, a pronunciation analysis module and a pronunciation correction module, according to the visual motion feature extraction method based on dimension fusion and feature simplification, dynamic motions and subtle changes of lips are captured, text semantics corresponding to detail features of lip motions of standard pronunciation are accurately recognized, the number of channels is increased on the basis of dimension reduction, deeper and finer lip visual information is captured, and the lip motion feature extraction efficiency is improved. More accurate lip action guidance is provided for pronunciation correction; according to the method, understanding and recognition of lip actions are guided through semantic features, a multi-head self-attention mechanism is introduced to optimize the semantic features, the incidence relation between the lip actions and voice texts is captured, and corresponding position codes are added, so that the model understands the pronunciation sequence of the oral text in the lip actions, and the lip actions can be recognized. And more accurate lip action guidance is realized.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent speech recognition, and specifically is an English spoken pronunciation correction assistance system based on speech recognition. Background Art

[0002] The English spoken pronunciation correction assistance system is a tool based on computer technology and phonetic principles, aiming to combine speech recognition technology, machine learning algorithms and computer vision technology to help learners improve their English spoken pronunciation and provide personalized pronunciation guidance and feedback. However, when the existing spoken pronunciation correction assistance system conducts correction guidance, the guidance suggestions are relatively vague, and it is difficult to accurately identify the pronunciation accuracy and lip movement guidance according to the oral text. Summary of the Invention

[0003] In view of the above situation, to overcome the defects of the prior art, the present invention provides an English spoken pronunciation correction assistance system based on speech recognition. Aiming at the problem that the existing technology has relatively vague guidance suggestions and it is difficult to accurately identify the pronunciation accuracy and lip movement guidance according to the oral text, the present invention adopts a visual action feature extraction method based on dimension fusion and feature simplification to extract the lip movement of the standard pronunciation corresponding to the oral text, extracts the convolutional features of the pronunciation video through a visual feature extraction method based on dimension transposition and reshaping, converts the three-dimensional video data into two-dimensional image data through a dimension transposition operation, retains the key information in the video while generating the two-dimensional image, and uses 3D convolution to capture the dynamic actions and subtle changes of the lips, accurately identifying the detailed features of the lip movement of the standard pronunciation. On this basis, a joint optimization process based on a multi-level dense layer and a Swin Transformer module is carried out to further extract image features, increase the number of channels on the basis of dimensionality reduction, capture deeper and more detailed lip visual information, and provide more accurate lip movement guidance for pronunciation correction; The present invention creatively, on the basis of a visual action feature extraction method based on dimension fusion and feature simplification, through semantic feature guidance, has a more detailed understanding and recognition of lip movements, can accurately select the corresponding lip movement label description according to the recognition classification result of the lip movement image, introduces a multi-head self-attention mechanism to optimize the semantic features, better captures the correlation between lip movements and speech texts, and adds corresponding position encodings to the words in the speech text, enabling the model to understand the order of the pronunciation actions of the oral text in the lip movement label description, and realizing more accurate lip movement guidance.

[0004] The English spoken pronunciation correction assistance system provided by the present invention includes a data acquisition module, a speech recognition module, a pronunciation analysis module and a pronunciation correction module;

[0005] The data acquisition module collects the audio of the standard pronunciation of spoken English and the lip movement video of the standard pronunciation of spoken English.

[0006] The speech recognition module uses a speech recognition model to convert the audio of the standard pronunciation into a speech text.

[0007] The pronunciation analysis module extracts the lip movement features of the pronunciation video through a text-guided cross-modal feature fusion extraction method, activates and classifies the action features and adds lip movement description labels, and extracts the audio features of the corresponding standard pronunciation audio to obtain the joint analysis result of the standard pronunciation of spoken English - speech text - lip movement description.

[0008] The pronunciation correction module collects the audio and the corresponding speech text that need to be corrected for pronunciation and sets a correction threshold, extracts the audio features of the audio that need to be corrected for pronunciation, and compares the similarity with the audio features of the standard pronunciation. When the similarity is higher than the correction threshold, no correction assistance is provided. If the similarity is lower than the correction threshold, the corresponding lip movement description is provided to guide the correction.

[0009] Further, in the pronunciation analysis module, the lip movement features of the pronunciation video are extracted through a text-guided cross-modal feature fusion extraction method, which specifically includes the following steps:

[0010] Step S1: Semantic feature extraction optimized based on the multi-head self-attention mechanism. Semantic features are extracted from the speech text through the multi-head self-attention mechanism optimization, which specifically includes the following steps:

[0011] Step S11: Contextual text feature extraction based on the self-attention mechanism. Contextual text features of the speech text are extracted based on the self-attention mechanism to obtain text features: ;

[0012] where represents the query vector, key vector, and value vector obtained by encoding the speech text through the self-attention mechanism, represents the preset key dimension, represents the relative position deviation parameter, which represents the text features;

[0013] Step S12: Multi-head self-attention mechanism optimization. The text features are optimized through the multi-head self-attention mechanism to obtain optimized text features: ; ;

[0014] where represents the head of the multi-head self-attention mechanism, , , and respectively represent the preset projection parameter matrices, represents the concatenation operation, which represents the optimized text features;

[0015] Step S13: Position encoding embedding. Perform position encoding embedding on the optimized text features corresponding to all words in the speech text to obtain semantic features. The position encoding calculation method is as follows: ;

[0016] In the formula, represents the position of the word in the speech text, represents the dimension of the optimized text feature corresponding to the word, represents the preset embedding dimension, represents the speech text the position encoding corresponding to the word at;

[0017] Step S2: Visual action feature extraction based on dimension fusion and feature simplification. Use a visual action feature extraction method based on dimension fusion and feature simplification to extract the optimized visual features of the pronunciation video, which specifically includes the following steps:

[0018] Step S21: Convolutional feature extraction. Use a visual feature extraction method based on dimension transposition and reshaping to extract the convolutional features of the pronunciation video, which specifically includes the following steps:

[0019] Step S211: Video input. Input the pronunciation video and obtain its dimensions (BZ, C, L, H, W), which respectively represent the batch size, number of channels, video length, video height, and video width of the pronunciation video;

[0020] Step S212: Dimension transposition. Swap the first and second dimensions of the pronunciation video to obtain the transposed video;

[0021] Step S213: Reshaping. Reshape the dimensions of the transposed video to obtain the reshaped video, whose dimensions are (BZ×L, C, H, W);

[0022] Step S214: 3D convolution. Perform convolutional feature extraction based on 3D convolution on the reshaped video to obtain convolutional features;

[0023] Step S22: Feature dimensionality reduction. Use the Linear Embedding layer to reduce the dimensionality of the convolutional features to obtain the dimensionality-reduced features;

[0024] Step S23: Feature Optimization. The dimensionality-reduced features are subjected to joint optimization processing based on a multi-level dense layer and a Swin Transformer module to obtain optimized visual features, which specifically include the following steps:

[0025] Step S231: Dense layer processing based on multi-layer 2D convolution. The dimensionality-reduced features are subjected to three-layer BN-ReLU-2DConv processing to obtain densely optimized convolutional features, where the convolutional kernel sizes of the 2DConv layers in each layer are 1×1, 3×3, and 1×1 respectively;

[0026] Step S232: Swin Transformer module processing. The dimensionality-reduced features are processed using the Swin Transformer module to obtain highly efficient optimized convolutional features;

[0027] Step S233: Feature fusion. The densely optimized convolutional features and the highly efficient optimized convolutional features are subjected to patch merging processing to obtain primary optimized convolutional features;

[0028] Step S234: Multi-layer optimization. Steps S32 - S34 are repeated n times to obtain optimized visual features;

[0029] Step S3: Cross-modal feature fusion. The semantic features are scaled to align with the optimized visual features, and then feature fusion is performed to obtain lip movement features.

[0030] Furthermore, in the pronunciation analysis module, the extraction of audio features specifically includes the following steps:

[0031] Step Q1: Audio preprocessing. The audio signal is preprocessed by denoising, equalization, and downsampling;

[0032] Step Q2: Audio framing. The audio signal is divided into short time periods, i.e., audio frames, using a Hamming Window;

[0033] Step Q3: Fourier transform. The Fourier transform is performed on each audio frame to obtain a spectrogram;

[0034] Step Q4: Feature extraction. Feature extraction based on spectral information is performed on the spectrogram to obtain spectral features;

[0035] Step Q5: Feature dimensionality reduction and normalization. The extracted spectral features are subjected to dimensionality reduction processing and normalization to obtain audio features.

[0036] The beneficial effects achieved by the present invention using the above solution are as follows:

[0037] (1) The present invention creatively adopts a visual action feature extraction method based on dimension fusion and feature simplification to extract the lip movements corresponding to the standard pronunciation of oral text, extracts the convolutional features of the pronunciation video through a visual feature extraction method based on dimension transposition and reshaping, converts the three-dimensional video data into two-dimensional image data through a dimension transposition operation, retains the key information in the video while generating the two-dimensional image, and uses 3D convolution to capture the dynamic actions and subtle changes of the lips, accurately identifying the detailed features of the lip movements of the standard pronunciation. On this basis, a joint optimization process based on a multi-level dense layer and a Swin Transformer module is carried out to further extract image features, increase the number of channels on the basis of dimensionality reduction, capture more profound and detailed lip visual information, and provide more accurate lip movement guidance for pronunciation correction;

[0038] (2) Based on a visual action feature extraction method based on dimension fusion and feature simplification, the present invention can more precisely understand and identify lip movements through semantic feature guidance, can accurately select the corresponding lip movement label description according to the recognition and classification results of lip movement images, introduces a multi-head self-attention mechanism to optimize semantic features, better captures the correlation between lip movements and speech texts, and adds corresponding position encoding to the words in the speech text, enabling the model to understand the order of the pronunciation actions of the oral text in the lip movement label description, and realizing more accurate lip movement guidance. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a module diagram of the English oral pronunciation correction assistance system based on speech recognition provided by the present invention;

[0040] Figure 2 is a flow schematic diagram of a text-guided cross-modal feature fusion extraction method.

[0041] The drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0042] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments; based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present invention.

[0043] Example 1, refer to Figure 1, An English spoken pronunciation correction assistance system based on speech recognition, comprising a data acquisition module, a speech recognition module, a pronunciation analysis module, and a pronunciation correction module;

[0044] The data acquisition module acquires the audio of the standard pronunciation of English spoken pronunciation and the lip pronunciation video of the standard pronunciation of English spoken pronunciation;

[0045] The speech recognition module uses a speech recognition model to convert the audio of the standard pronunciation into a speech text;

[0046] The pronunciation analysis module extracts the lip movement features of the pronunciation video through a text-guided cross-modal feature fusion extraction method, activates and classifies the movement features and adds lip movement description labels, and extracts the audio features of the corresponding standard pronunciation audio to obtain a joint analysis result of the standard pronunciation of spoken English - speech text - lip movement description;

[0047] The pronunciation correction module acquires the audio to be corrected and the corresponding speech text and sets a correction threshold, extracts the audio features of the audio to be corrected, and compares the similarity with the audio features of the standard pronunciation. When the similarity is higher than the correction threshold, no correction assistance is provided. If the similarity is lower than the correction threshold, the corresponding lip movement description is provided to provide correction guidance.

[0048] Example 2, refer to Figure 2 , Based on the above example, in the pronunciation analysis module, the lip movement features of the pronunciation video are extracted through a text-guided cross-modal feature fusion extraction method, which specifically includes the following steps:

[0049] Step S1: Semantic feature extraction optimized based on the multi-head self-attention mechanism, perform semantic feature extraction optimized based on the multi-head self-attention mechanism on the speech text to obtain semantic features;

[0050] Step S2: Visual action feature extraction based on dimension fusion and feature simplification, use a visual action feature extraction method based on dimension fusion and feature simplification to extract the optimized visual features of the pronunciation video;

[0051] Step S3: Cross-modal feature fusion, scale the semantic features to align them with the optimized visual features, and then perform feature fusion to obtain the lip movement features.

[0052] By performing the above operations, the convolutional features of the pronunciation video are extracted through a visual feature extraction method based on dimension transposition and reshaping. The three-dimensional video data is converted into two-dimensional image data through the dimension transposition operation, while the key information in the video is retained when generating the two-dimensional image. 3D convolution is used to capture the dynamic actions and subtle changes of the lips, accurately identify the detailed features of the lip movements of standard pronunciation. On this basis, a joint optimization process based on a multi-level dense layer and a Swin Transformer module is carried out to further extract image features, increase the number of channels on the basis of dimensionality reduction, capture deeper and more detailed lip visual information, and provide more accurate lip movement guidance for pronunciation correction.

[0053] Example 3. This example is based on the above example. Step S1 specifically includes the following steps:

[0054] Step S11: Extract context text features based on the self-attention mechanism. Extract context text features of the speech text based on the self-attention mechanism to obtain text features: ;

[0055] In the formula, represents the query vector, key vector, and value vector obtained by encoding the speech text through the self-attention mechanism, represents the preset key dimension, represents the relative position deviation parameter, which represents the text features;

[0056] Step S12: Optimize the multi-head self-attention mechanism. Optimize the text features through the multi-head self-attention mechanism to obtain optimized text features: ; ;

[0057] In the formula, represents the head of the multi-head self-attention mechanism, , , and respectively represent the preset projection parameter matrices, represents the concatenation operation, which represents the optimized text features;

[0058] Step S13: Embed position encoding. Embed position encoding for the optimized text features corresponding to all words in the speech text to obtain semantic features. The position encoding calculation method is as follows: ;

[0059] In the formula, Represents the position of the word in the speech text, Represents the dimension of the optimized text feature corresponding to the word, Represents the preset embedding dimension, Represents in the speech text The position encoding corresponding to the word at the position.

[0060] Example 4. This example is based on the above example. Step S2 specifically includes the following steps:

[0061] Step S21: Convolutional feature extraction. A visual feature extraction method based on dimension transposition and reshaping is used to extract the convolutional features of the pronunciation video;

[0062] Step S22: Feature dimensionality reduction. The Linear Embedding layer is used to reduce the dimensionality of the convolutional features to obtain the dimensionality-reduced features;

[0063] Step S23: Feature optimization. The dimensionality-reduced features are jointly optimized based on a multi-level dense layer and the Swin Transformer module to obtain the optimized visual features.

[0064] Example 5. This example is based on the above example. Step S21 specifically includes the following steps:

[0065] Step S211: Video input. The pronunciation video is input, and its dimensions (BZ, C, L, H, W) are obtained, representing the batch size, number of channels, video length, video height, and video width of the pronunciation video respectively;

[0066] Step S212: Dimension transposition. The first and second dimensions in the pronunciation video are exchanged to obtain the transposed video;

[0067] Step S213: Reshaping. The dimensions of the transposed video are reshaped to obtain the reshaped video, whose dimensions are (BZ×L, C, H, W);

[0068] Step S214: 3D convolution. 3D convolution-based convolutional feature extraction is performed on the reshaped video to obtain the convolutional features.

[0069] Example 6. This example is based on the above example. Step S23 specifically includes the following steps:

[0070] Step S231: Dense layer processing based on multi-layer 2D convolution. The dimensionality-reduced features are processed with three layers of BN-ReLU-2DConv to obtain the densely optimized convolutional features, where the kernel sizes of the 2DConv layers in each layer are 1×1, 3×3, and 1×1 respectively;

[0071] Step S232: Process by the Swin Transformer module. Use the Swin Transformer module to process the dimensionality-reduced features to obtain highly optimized convolutional features;

[0072] Step S233: Feature fusion. Perform patch merging on the dense optimized convolutional features and the highly optimized convolutional features to obtain primary optimized convolutional features;

[0073] Step S234: Multilayer optimization. Repeat steps S32 - S34 three times to obtain optimized visual features.

[0074] By performing the above operations, based on a visual action feature extraction method based on dimensionality fusion and feature simplification, through semantic feature guidance, a more detailed understanding and recognition of lip movements can be achieved. The corresponding lip movement label description can be accurately selected according to the recognition classification result of the lip movement image. The multi-head self-attention mechanism is introduced to optimize the semantic features, better capturing the correlation between lip movements and speech texts, and adding corresponding position encodings to the words in the speech text, enabling the model to understand the order of the pronunciation actions of the oral text in the lip movement label description, and realizing more accurate lip movement guidance.

[0075] Embodiment Seven. Based on the above embodiment, in the pronunciation analysis module, the extraction of audio features specifically includes the following steps:

[0076] Step Q1: Audio preprocessing. Perform denoising, equalization, and downsampling preprocessing on the audio signal;

[0077] Step Q2: Audio framing. Use the Hamming Window to divide the audio signal into short time periods, namely audio frames;

[0078] Step Q3: Fourier transform. Perform Fourier transform on each audio frame to obtain a spectrogram;

[0079] Step Q4: Feature extraction. Perform feature extraction based on spectral information on the spectrogram to obtain spectral features;

[0080] Step Q5: Feature dimensionality reduction and normalization. Perform dimensionality reduction processing and normalization on the extracted spectral features to obtain audio features.

[0081] Embodiment Eight. Based on the above embodiment, a simple example of the optimized implementation of the present invention's technology is as follows:

[0082] (1) Speech recognition and feature extraction

[0083] Speech recognition technology:

[0084] Use pre-trained automatic speech recognition (ASR) models (such as Google Speech-to-Text, Wav2Vec2.0);

[0085] Audio feature extraction:

[0086] Extract the acoustic features of speech (such as Mel spectrogram, MFCC);

[0087] Analyze the duration, frequency, and energy parameters of phonemes, that is, spectrogram features;

[0088] (2) Standard pronunciation comparison and error detection (i.e., similarity comparison of the pronunciation correction module)

[0089] Dynamic time warping (DTW):

[0090] Compare the time series differences between the user's pronunciation and the standard pronunciation to detect phoneme deviations;

[0091] Phoneme-level scoring:

[0092] Use an acoustic model (such as HMM or DNN-HMM) to calculate the accuracy rate of each phoneme;

[0093] (3) Correction and feedback generation

[0094] Error localization:

[0095] Locate the mispronounced phonemes according to the acoustic features of phonemes (such as frequency deviation, abnormal duration);

[0096] Feedback generation:

[0097] Generate correction suggestions in combination with the error type;

[0098] For example:

[0099] If the mispronounced phoneme is / i / , the system prompts: "Raise the tongue position and lengthen the pronunciation time;".

[0100] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent in such process, method, article or device.

[0101] Although embodiments of the present invention have been shown and described, those of ordinary skill in the art will understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

[0102] The above description of the present invention and its embodiments is not restrictive. What is shown in the drawings is only one of the embodiments of the present invention, and the actual structure is not limited thereto. In general, if those of ordinary skill in the art are inspired by it and, without departing from the purpose of the present invention, design similar structural modes and embodiments to this technical solution without creative efforts, they should all fall within the protection scope of the present invention.

Claims

1. An English spoken language pronunciation correction assistance system based on speech recognition, characterized in that: It includes a data acquisition module, a speech recognition module, a pronunciation analysis module, and a pronunciation correction module; The data acquisition module acquires the audio of the standard pronunciation of spoken English and the lip movement video of the standard pronunciation of spoken English; The speech recognition module uses a speech recognition model to convert the audio of the standard pronunciation into a speech text; The pronunciation analysis module extracts the lip movement features of the pronunciation video through a text-guided cross-modal feature fusion extraction method, activates and classifies the action features and adds lip movement description labels, and extracts the audio features of the corresponding standard pronunciation audio to obtain a joint analysis result of the standard pronunciation of spoken English - speech text - lip movement description; The pronunciation correction module acquires the audio and the corresponding speech text that need to be corrected in pronunciation and sets a correction threshold, extracts the audio features of the audio that need to be corrected in pronunciation, and compares the similarity with the audio features of the standard pronunciation. When the similarity is higher than the correction threshold, no correction assistance is provided. If the similarity is lower than the correction threshold, it provides the corresponding lip movement description for correction guidance.

2. The speech recognition-based English spoken pronunciation correction assistance system according to claim 1, characterized in that: In the pronunciation analysis module, a text-guided cross-modal feature fusion extraction method specifically includes the following steps: Step S1: Extract semantic features by optimizing the semantic features of the speech text based on the multi-head self-attention mechanism to obtain semantic features; Step S2: Use a visual action feature extraction method based on dimension fusion and feature simplification to extract the optimized visual features of the pronunciation video; Step S3: Scale the semantic features to align them with the optimized visual features, and then perform feature fusion to obtain lip movement features.

3. The speech recognition-based English spoken pronunciation correction assistance system according to claim 2, characterized in that: Step S1 specifically includes the following steps: Step S11: Extract the context text features of the speech text based on the self-attention mechanism to obtain text features; Step S12: Optimize the text features through the multi-head self-attention mechanism to obtain optimized text features; Step S13: Perform position encoding embedding on the optimized text features corresponding to all words in the speech text to obtain semantic features, where the position encoding calculation method is as follows: ； In the formula, represents the position of the word in the speech text, represents the dimension of the optimized text feature corresponding to the word, represents the preset embedding dimension, represents in the speech text the position encoding corresponding to the word at the 4. The speech recognition-based English spoken pronunciation correction assistance system according to claim 3, characterized in that: Step S2 specifically includes the following steps: Step S21: Use a visual feature extraction method based on dimension transposition and reshaping to extract the convolutional features of the pronunciation video; Step S22: Use the Linear Embedding layer to reduce the dimension of the convolutional features to obtain reduced-dimensional features; Step S23: Perform joint optimization processing on the reduced-dimensional features based on the multi-level dense layer and the Swin Transformer module to obtain optimized visual features.

5. The voice recognition-based English spoken pronunciation correction assistance system according to claim 4, characterized in that: Step S21 specifically includes the following steps: Step S211: Input the pronunciation video and obtain its dimensions (BZ, C, L, H, W), which represent the batch size, number of channels, video length, video height, and video width of the pronunciation video respectively; Step S212: Swap the first and second dimensions of the pronunciation video to obtain a transposed video; Step S213: Reshape the dimensions of the transposed video to obtain a reshaped video; Step S214: Perform convolutional feature extraction based on 3D convolution on the reshaped video to obtain convolutional features.

6. The voice recognition-based English spoken pronunciation correction assistance system according to claim 5, characterized in that: Step S23 specifically includes the following steps: Step S231: Perform three-layer BN-ReLU-2DConv processing on the dimensionality-reduced features to obtain dense optimized convolutional features, where the convolutional kernel sizes of the 2DConv layers in each layer are 1×1, 3×3, and 1×1 respectively; Step S232: Use the Swin Transformer module to process the dimensionality-reduced features to obtain efficient optimized convolutional features; Step S233: Perform patch merging on the dense optimized convolutional features and the efficient optimized convolutional features to obtain primary optimized convolutional features; Step S234: Repeat steps S231 - S233 three times to obtain optimized visual features.

Images