Multi-country English pronunciation data expansion and recognition compensation model construction method

Abstract

The invention relates to a multi-country English pronunciation data expansion and recognition compensation model construction method. The method comprises the following steps: firstly, collecting multi-country English pronunciation samples, and carrying out preprocessing operations such as denoising, segmentation and format conversion; mel frequency cepstrum coefficient (MFCC) acoustic features are extracted, and timing features are captured using RNN and LSTM networks. And generating voice data of different countries through the adversarial network, and combining the voice data with real data to construct a multi-country voice database. The scheme further comprises construction of a nationality recognition module and a voice recognition model, the model is trained through an LSTM or Transform architecture, and a nationality recognition result is used as input to improve the recognition effect. According to the invention, multi-country English pronunciation data can be effectively expanded, the data collection cost is reduced, and the robustness and accuracy of a speech recognition system in diversified accent environments are improved. Through the technical means, the scheme provides effective data support and model optimization strategies for speech recognition application under a globalized background.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition, and in particular to a method for expanding English pronunciation data of multiple countries and constructing a recognition compensation model. Background Art

[0002] English pronunciation is influenced by the cultures of different countries and regions, and presents diverse accent characteristics. This diversity poses a challenge to speech recognition technology, because traditional speech recognition systems are usually trained based on standard or single-accent English, and have low recognition accuracy for non-standard accents. In order to improve the speech recognition system's ability to recognize English pronunciation from multiple countries, it is necessary to build a speech recognition model that can understand and adapt to different accents.

[0003] The collection and annotation of English pronunciation samples from different countries is labor-intensive and costly, which limits model training and performance improvement. Therefore, studying an effective method for expanding English pronunciation data from different countries is of great significance for improving the robustness and accuracy of speech recognition systems under diverse accents. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a method for expanding English pronunciation data of multiple countries, comprising the following steps:

[0005] Collect English pronunciation samples from different countries and regions and provide corresponding text labels;

[0006] Perform denoising, segmentation, and format conversion preprocessing operations on the collected voice data;

[0007] Extract Mel-frequency cepstral coefficient (MFCC) acoustic features from the preprocessed speech signal; use recurrent neural network (RNN) and long short-term memory network (LSTM) to capture the temporal characteristics of speech;

[0008] Generate voice data from different countries through adversarial networks;

[0009] The generated voice data is merged with the real voice data to build a multi-national voice database.

[0010] In the collection step, samples are obtained from natural English conversation datasets with accents from more than 20 countries.

[0011] The denoising process uses spectral subtraction to estimate the power spectrum of the speech signal and the power spectrum of the noise, and then subtract the power spectrum of the noise from the speech signal to achieve noise reduction. Wiener filtering is further used to reduce noise by minimizing the mean square error between the original signal and the estimated signal.

[0012] Using voice activity detection (VAD) technology, we can automatically identify speech segments and non-speech segments in speech signals and segment long recordings into separate sentences or phrases. We can also add text labels to the segmented speech data through automatic transcription to facilitate subsequent feature extraction and model training.

[0013] Use the FFmpeg tool to convert the collected voice data into a WAV or FLAC format suitable for processing for subsequent speech recognition model training. During the conversion, ensure that the converted audio files have a uniform sampling rate and bit depth.

[0014] Extract Mel-frequency cepstral coefficient (MFCC) acoustic features from the preprocessed speech signal, including:

[0015] Pre-emphasis: enhance the high frequency part to simulate the human ear's sensitivity to high frequencies;

[0016] Framing: Speech signals are divided into short frames with a frame length of 20-40ms and a frame shift of 10-20ms.

[0017] Fourier transform: Perform fast Fourier transform on each frame to obtain the spectrum;

[0018] Mel filter bank: The spectrum is passed through the Mel filter bank to obtain the Mel spectrum;

[0019] Take the logarithm: Take the natural logarithm of each filter output of the Mel spectrum;

[0020] Discrete cosine transform: Perform discrete cosine transform on the logarithmic Mel spectrum, keeping only the first 12-20 coefficients;

[0021] The use of recurrent neural networks (RNN) and long short-term memory networks (LSTM) to capture the temporal features of speech specifically includes:

[0022] Build an RNN or LSTM model to process the extracted acoustic features. The model input is a 39-dimensional MFCC feature sequence, and the model output is an English pronunciation sample including a country text label.

[0023] Design a single-layer RNN network to capture short-term dependencies;

[0024] Design a two-layer LSTM network to capture long-term dependencies, with 128 units in each layer, and the output layer is mapped to a 39-dimensional feature space using a fully connected layer;

[0025] The data is divided into training set, validation set and test set. The mean square error is used as the loss function. The goal is to minimize the difference between the generated features and the target features. The Adam optimizer is used, and the learning rate is set to 0.001. Each batch contains 20 sequences, and each sequence is 100 time steps long. L2 regularization is applied to prevent overfitting.

[0026] Forward propagation: The RNN layer first processes the input MFCC feature sequence and calculates the hidden state. The LSTM layer then processes the output of the RNN layer to further capture the temporal features. The fully connected layer maps the output of the LSTM layer to a 39-dimensional feature space.

[0027] Backward propagation: Calculate the gradient of the loss function with respect to the model parameters and update the parameters using the Adam optimizer;

[0028] The hidden state of the last time step of the LSTM layer is used as the temporal feature representation of the entire sequence as the extracted temporal feature.

[0029] The specific examples of generating speech data from different countries through adversarial networks are as follows:

[0030] Generator design:

[0031] Design a generator network that receives random noise, time series features extracted by RNN and LSTM as input, and standard English speech data as input; this is achieved by combining the feature vectors of random noise and standard English speech data with the input time series feature vector as the input of the generator; the generator outputs Mel spectrum features;

[0032] Vocoder: Use Griffin-Lim, WaveNet or WaveGlow to receive the Mel spectrum features output by the generator and output the corresponding audio waveform;

[0033] Conduct adversarial training, where the generator tries to generate more and more realistic speech data, and the discriminator tries to more accurately identify the generated data and real data. Use Wasserstein GAN technology to stabilize the training process.

[0034] Each country’s English speech data is used to set up its own adversarial network to generate speech data with country-specific characteristics.

[0035] The generated voice data is combined with the real voice data to build a multi-national voice database. Specifically:

[0036] Annotate speech data, including transcripts, language identifiers, and speaker information;

[0037] Language identifiers include country, region, and proficiency level, and speaker information includes gender, age, and education level.

[0038] A method for constructing a multi-national speech recognition compensation model based on the multi-national English pronunciation data expansion method includes:

[0039] Data review: Before using the database, review the data to ensure the quality of recordings and the accuracy of annotations;

[0040] Denoising and segmentation: Denoise the speech data in the database to reduce the interference of background noise and segment it by sentences or phrases;

[0041] Feature extraction: Extract key features from speech signals, including Mel-frequency cepstral coefficients, pitch, duration, and volume;

[0042] Country identification module: This module can identify the country of origin of a speech sample, and is trained by using the country-annotated speech data in the database obtained by the multi-country English pronunciation data expansion method described in any one of claims 1 to 6;

[0043] Select the LSTM or Transformer deep learning model architecture to build a speech recognition model, use multi-country speech data to train the speech recognition model, and use the country recognition results as the input of the speech recognition model to adjust the parameters of the speech recognition model and improve the recognition effect of specific country accents.

[0044] The country identification module includes a country identification model. The country identification model adopts the LSTM architecture and includes:

[0045] Input layer: The input layer of the model receives the normalized MFCC feature sequence;

[0046] Hidden layer: Design one or more LSTM layers to capture the temporal information of speech features;

[0047] Output layer: The design of the output layer depends on the number of country categories, and the softmax activation function is used to obtain the probability distribution of each category.

[0048] The beneficial effects of the present invention are:

[0049] This scheme proposes a method for expanding English pronunciation data for different countries. It generates speech data of different countries through adversarial networks and merges them with real speech data to build a multi-national speech database. This method uses recurrent neural networks (RNN) and long short-term memory networks (LSTM) to capture the temporal characteristics of speech. Combined with adversarial network generators and vocoders, it can efficiently expand English pronunciation data for different countries and significantly reduce the cost of data collection and annotation.

[0050] By building a country recognition module and a speech recognition model, this solution can improve the recognition effect of the speech recognition system on the accent of a specific country. Technically, this solution achieves accurate recognition of English pronunciations from multiple countries through the country recognition model with LSTM architecture and the speech recognition model with deep learning model architecture, improving the robustness and accuracy of the speech recognition system in a diverse accent environment. The training process of the adversarial network is stabilized by the Wasserstein GAN technology, ensuring the quality and diversity of the generated speech data. Overall, this solution improves the speech recognition system's ability to process English pronunciations from multiple countries on a technical level, providing an effective solution for speech recognition applications in a global context. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0052] Attached Figure 1 A flow chart for constructing a database for the present invention;

[0053] Attached Figure 2 The present invention is supplemented with a flow chart of the model building method. DETAILED DESCRIPTION

[0054] Embodiment 1:

[0055] See also Figure 1 The present invention provides a method for expanding English pronunciation data of multiple countries, comprising the following steps:

[0056] Collect English pronunciation samples from different countries and regions and provide corresponding text labels;

[0057] Perform denoising, segmentation, and format conversion preprocessing operations on the collected voice data;

[0058] Extract Mel-frequency cepstral coefficient (MFCC) acoustic features from the preprocessed speech signal; use recurrent neural network (RNN) and long short-term memory network (LSTM) to capture the temporal characteristics of speech;

[0059] Generate voice data from different countries through adversarial networks;

[0060] The generated voice data is merged with the real voice data to build a multi-national voice database.

[0061] In the collection step, samples are obtained from natural English conversation datasets with accents from more than 20 countries.

[0062] The denoising process uses spectral subtraction to estimate the power spectrum of the speech signal and the power spectrum of the noise, and then subtract the power spectrum of the noise from the speech signal to achieve noise reduction. Wiener filtering is further used to reduce noise by minimizing the mean square error between the original signal and the estimated signal.

[0063] Using voice activity detection (VAD) technology, we can automatically identify speech segments and non-speech segments in speech signals and segment long recordings into separate sentences or phrases. We can also add text labels to the segmented speech data through automatic transcription to facilitate subsequent feature extraction and model training.

[0064] Use the FFmpeg tool to convert the collected voice data into a WAV or FLAC format suitable for processing for subsequent speech recognition model training. During the conversion, ensure that the converted audio files have a uniform sampling rate and bit depth.

[0065] Extract Mel-frequency cepstral coefficient (MFCC) acoustic features from the preprocessed speech signal, including:

[0066] Pre-emphasis: enhance the high frequency part to simulate the human ear's sensitivity to high frequencies;

[0067] Framing: Speech signals are divided into short frames with a frame length of 20-40ms and a frame shift of 10-20ms.

[0068] Fourier transform: Perform fast Fourier transform on each frame to obtain the spectrum;

[0069] Mel filter bank: The spectrum is passed through the Mel filter bank to obtain the Mel spectrum;

[0070] Take the logarithm: Take the natural logarithm of each filter output of the Mel spectrum;

[0071] Discrete cosine transform: Perform discrete cosine transform on the logarithmic Mel spectrum, keeping only the first 12-20 coefficients;

[0072] The use of recurrent neural networks (RNN) and long short-term memory networks (LSTM) to capture the temporal features of speech specifically includes:

[0073] Build an RNN or LSTM model to process the extracted acoustic features. The model input is a 39-dimensional MFCC feature sequence, and the model output is an English pronunciation sample including a country text label.

[0074] Design a single-layer RNN network to capture short-term dependencies;

[0075] Design a two-layer LSTM network to capture long-term dependencies, with 128 units in each layer, and the output layer is mapped to a 39-dimensional feature space using a fully connected layer;

[0076] The data is divided into training set, validation set and test set. The mean square error is used as the loss function. The goal is to minimize the difference between the generated features and the target features. The Adam optimizer is used, and the learning rate is set to 0.001. Each batch contains 20 sequences, and each sequence is 100 time steps long. L2 regularization is applied to prevent overfitting.

[0077] Forward propagation: The RNN layer first processes the input MFCC feature sequence and calculates the hidden state. The LSTM layer then processes the output of the RNN layer to further capture the temporal features. The fully connected layer maps the output of the LSTM layer to a 39-dimensional feature space.

[0078] Backward propagation: Calculate the gradient of the loss function with respect to the model parameters and update the parameters using the Adam optimizer;

[0079] The hidden state of the last time step of the LSTM layer is used as the temporal feature representation of the entire sequence as the extracted temporal feature.

[0080] The specific examples of generating speech data from different countries through adversarial networks are as follows:

[0081] Generator design:

[0082] Design a generator network that receives random noise, time series features extracted by RNN and LSTM as input, and standard English speech data as input; this is achieved by combining the feature vectors of random noise and standard English speech data with the input time series feature vector as the input of the generator; the generator outputs Mel spectrum features;

[0083] Vocoder: Use Griffin-Lim, WaveNet or WaveGlow to receive the Mel spectrum features output by the generator and output the corresponding audio waveform;

[0084] Conduct adversarial training, where the generator tries to generate more and more realistic speech data, and the discriminator tries to more accurately identify the generated data and real data. Use Wasserstein GAN technology to stabilize the training process.

[0085] Each country’s English speech data is used to set up its own adversarial network to generate speech data with country-specific characteristics.

[0086] The generated voice data is combined with the real voice data to build a multi-national voice database. Specifically:

[0087] Annotate speech data, including transcripts, language identifiers, and speaker information;

[0088] Language identifiers include country, region, and proficiency level, and speaker information includes gender, age, and education level.

[0089] Embodiment 2:

[0090] See also Figure 2 A method for constructing a multi-national speech recognition compensation model based on the multi-national English pronunciation data expansion method, comprising:

[0091] Data review: Before using the database, review the data to ensure the quality of recordings and the accuracy of annotations;

[0092] Denoising and segmentation: Denoise the speech data in the database to reduce the interference of background noise and segment it by sentences or phrases;

[0093] Feature extraction: Extract key features from speech signals, including Mel-frequency cepstral coefficients, pitch, duration, and volume;

[0094] Country identification module: This module can identify the country of origin of a speech sample, and is trained by using the country-annotated speech data in the database obtained by the multi-country English pronunciation data expansion method described in any one of claims 1 to 6;

[0095] Select the LSTM or Transformer deep learning model architecture to build a speech recognition model, use multi-country speech data to train the speech recognition model, and use the country recognition results as the input of the speech recognition model to adjust the parameters of the speech recognition model and improve the recognition effect of specific country accents.

[0096] The country identification module includes a country identification model. The country identification model adopts the LSTM architecture and includes:

[0097] Input layer: The input layer of the model receives the normalized MFCC feature sequence;

[0098] Hidden layer: Design one or more LSTM layers to capture the temporal information of speech features;

[0099] Output layer: The design of the output layer depends on the number of country categories, and the softmax activation function is used to obtain the probability distribution of each category.

[0100] Thus far, the description of the above-described embodiments is provided for the purpose of illustration and description. It is not intended to be exhaustive or to limit the present disclosure. The individual elements or features of a specific embodiment are generally not subject to the limitations of the specific embodiment, but when applicable, they can be interchanged and used for selected embodiments even if not specifically shown or described. In many respects, the same element or feature also can be changed. This variation is not considered to be a departure from the present disclosure, and all such modifications are intended to be included in the scope of the present disclosure.

[0101] Example embodiments are provided so that the present disclosure will be thorough and the scope will be fully conveyed to those skilled in the art. In order to thoroughly understand the embodiments of the present disclosure, numerous details are set forth, such as examples of specific parts, devices, and methods. Obviously, for those skilled in the art, it is not necessary to use specific details, the example embodiments can be implemented in many different forms, and neither should be construed as limiting the scope of the present disclosure. In certain example embodiments, well-known processes, well-known device structures, and well-known technologies are not described in detail.

[0102] Here, specialized vocabulary is used only for the purpose of describing specific example embodiments, and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "a" and "the" used herein may be intended to include plural forms as well. The terms "including" and "having" are inclusive, and therefore specify the presence of the claimed features, wholes, steps, operations, elements and / or components, but do not exclude the presence or additional presence of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof. Unless the order of execution is clearly indicated, the method steps, processing and operations described herein are not interpreted as necessarily needing to be performed in the specific order discussed and shown. It should also be understood that additional or optional steps may be adopted.

Claims

1. A method for expanding English pronunciation data for multiple countries, characterized in that: The steps include: Collect English pronunciation samples from different countries and regions and provide corresponding text labels; Perform denoising, segmentation, and format conversion preprocessing operations on the collected voice data; Extracting Mel-frequency cepstral coefficient (MFCC) acoustic features from the preprocessed speech signal; Use recurrent neural networks (RNN) and long short-term memory networks (LSTM) to capture the temporal characteristics of speech; generate speech data from different countries through adversarial networks; The generated voice data is merged with the real voice data to build a multi-national voice database.

2. The method for expanding English pronunciation data for multiple countries according to claim 1, characterized in that: In the collection step, samples are obtained from natural English conversation datasets with accents from more than 20 countries.

3. The method for expanding English pronunciation data for multiple countries according to claim 1, characterized in that: The denoising process uses spectral subtraction to estimate the power spectrum of the speech signal and the power spectrum of the noise, and then subtract the power spectrum of the noise from the speech signal to achieve noise reduction. Wiener filtering is further used to reduce noise by minimizing the mean square error between the original signal and the estimated signal. Using voice activity detection (VAD) technology, we can automatically identify speech segments and non-speech segments in speech signals and segment long recordings into separate sentences or phrases. We can also add text labels to the segmented speech data through automatic transcription to facilitate subsequent feature extraction and model training. Use the FFmpeg tool to convert the collected voice data into a WAV or FLAC format suitable for processing for subsequent speech recognition model training. During the conversion, ensure that the converted audio files have a uniform sampling rate and bit depth.

4. The method for expanding English pronunciation data for multiple countries according to claim 1, characterized in that: Extract Mel-frequency cepstral coefficient (MFCC) acoustic features from the preprocessed speech signal, including: Pre-emphasis: enhance the high frequency part to simulate the human ear's sensitivity to high frequencies; Framing: Speech signals are divided into short frames with a frame length of 20-40ms and a frame shift of 10-20ms. Fourier transform: Perform fast Fourier transform on each frame to obtain the spectrum; Mel filter bank: The spectrum is passed through the Mel filter bank to obtain the Mel spectrum; Take the logarithm: Take the natural logarithm of each filter output of the Mel spectrum; Discrete cosine transform: Perform discrete cosine transform on the logarithmic Mel spectrum, keeping only the first 12-20 coefficients; The use of recurrent neural networks (RNN) and long short-term memory networks (LSTM) to capture the temporal features of speech specifically includes: Build an RNN or LSTM model to process the extracted acoustic features. The model input is a 39-dimensional MFCC feature sequence, and the model output is an English pronunciation sample including a country text label. Design a single-layer RNN network to capture short-term dependencies; Design a two-layer LSTM network to capture long-term dependencies, with 128 units in each layer, and the output layer is mapped to a 39-dimensional feature space using a fully connected layer; The data is divided into training set, validation set and test set. The mean square error is used as the loss function. The goal is to minimize the difference between the generated features and the target features. The Adam optimizer is used, and the learning rate is set to 0.

001. Each batch contains 20 sequences, and each sequence is 100 time steps long. L2 regularization is applied to prevent overfitting. Forward propagation: The RNN layer first processes the input MFCC feature sequence and calculates the hidden state. The LSTM layer then processes the output of the RNN layer to further capture the temporal features. The fully connected layer maps the output of the LSTM layer to a 39-dimensional feature space. Backward propagation: Calculate the gradient of the loss function with respect to the model parameters and update the parameters using the Adam optimizer; The hidden state of the last time step of the LSTM layer is used as the temporal feature representation of the entire sequence as the extracted temporal feature.

5. The method for expanding English pronunciation data for multiple countries according to claim 4, characterized in that: The specific examples of generating speech data from different countries through adversarial networks are as follows: Generator design: Design a generator network that receives random noise, time series features extracted by RNN and LSTM as input, and standard English speech data as input; this is achieved by combining the feature vectors of random noise and standard English speech data with the input time series feature vector as the input of the generator; the generator outputs Mel spectrum features; Vocoder: Use Griffin-Lim, WaveNet or WaveGlow to receive the Mel spectrum features output by the generator and output the corresponding audio waveform; Conduct adversarial training, where the generator tries to generate more and more realistic speech data, and the discriminator tries to more accurately identify the generated data and real data. Use Wasserstein GAN technology to stabilize the training process. Each country’s English speech data is used to set up its own adversarial network to generate speech data with country-specific characteristics.

6. The method for expanding English pronunciation data for multiple countries according to claim 5, characterized in that: The generated voice data is combined with the real voice data to build a multi-national voice database. Specifically: Annotate speech data, including transcripts, language identifiers, and speaker information; Language identifiers include country, region, and proficiency level, and speaker information includes gender, age, and education level.

7. A method for constructing a multi-national speech recognition compensation model based on the multi-national English pronunciation data expansion method according to any one of claims 1 to 6, characterized in that: Data review: Before using the database, review the data to ensure the quality of recordings and the accuracy of annotations; Denoising and segmentation: Denoise the speech data in the database to reduce the interference of background noise and segment it by sentences or phrases; Feature extraction: Extract key features from speech signals, including Mel-frequency cepstral coefficients, pitch, duration, and volume; Country identification module: This module can identify the country of origin of a speech sample, and is trained by using the country-annotated speech data in the database obtained by the multi-country English pronunciation data expansion method described in any one of claims 1 to 6; Select the LSTM or Transformer deep learning model architecture to build a speech recognition model, use multi-country speech data to train the speech recognition model, and use the country recognition results as the input of the speech recognition model to adjust the parameters of the speech recognition model and improve the recognition effect of specific country accents.

8. The method for constructing a multi-national speech recognition compensation model according to claim 7, characterized in that: The country identification module includes a country identification model. The country identification model adopts the LSTM architecture and includes: Input layer: The input layer of the model receives the normalized MFCC feature sequence; Hidden layer: Design one or more LSTM layers to capture the temporal information of speech features; Output layer: The design of the output layer depends on the number of country categories, and the softmax activation function is used to obtain the probability distribution of each category.

Images