A method and system for correcting mandarin dialect accents using acoustic units

Abstract

The application discloses a Chinese dialect accent correction method and system using acoustic units. The main steps include: first, generating the acoustic unit sequence corresponding to the mandarin audio and the accent audio respectively through the acoustic unit construction module; then, transcribing the mandarin audio and the accent audio into texts respectively through the speech-to-text module to obtain the corresponding Chinese character sequences; then, aligning the Chinese character sequences and the acoustic unit sequences of the mandarin and the accent through the time alignment module to obtain the corresponding Chinese character sequences and acoustic unit sequences and the timestamp information; then, aligning the above sequences and comparing them through the text alignment module to determine the specific position in the audio that needs to be modified; finally, converting the accent audio into the accent-to-mandarin audio through the speech editing and correction module. The application can more accurately correct the accent in the case that the speaker makes an incorrect pronunciation that does not belong to the Chinese phoneme.

Description

Technical Field

[0001] This invention belongs to the field of speech processing, and specifically relates to a method for detecting and correcting Chinese dialect accents. It involves a method and system for correcting Chinese dialect accents using acoustic units. Background Technology

[0002] Chinese dialect accent correction is an important technique that uses accent correction models to detect and correct accented parts of speech, converting dialect audio into standard Mandarin audio. Currently, most existing accent correction methods follow a speech conversion approach, aiming to transform second language (L2) speech features into features closer to the first language (L1). Traditionally, many studies have relied on phoneme-level error matching to identify and correct accent differences. However, using phoneme-level accent correction often fails to adequately consider the profound influence of language habits on speaker pronunciation, resulting in actual pronunciation deviations that do not perfectly match the types of phonemes contained in the target language.

[0003] During communication, speakers may exhibit regional accents for various reasons. Accent correction technology can convert accented audio into accent-free audio, facilitating daily communication. However, to date, most existing accent correction methods primarily detect and correct accents through phoneme-level comparisons, failing to address errors in pronunciation of phonemes from non-target languages. Summary of the Invention

[0004] To address the problems existing in the background technology, this invention proposes a novel method for correcting Chinese dialect accents, entitled "A Method and System for Correcting Chinese Dialect Accents Using Acoustic Units," denoted as the AU-ACM method. The AU-ACM method uses finer-grained acoustic units instead of phonemes to detect and correct dialect accent components in audio. This allows for more accurate accent correction even when the speaker makes incorrect pronunciations that do not belong to Chinese phonemes.

[0005] To solve the above problems, the technical solution of the present invention is as follows:

[0006] I. A method for correcting Chinese dialect accents using acoustic units

[0007] S1: Generate acoustic unit sequences corresponding to Mandarin audio and accent audio respectively through the acoustic unit construction module;

[0008] S2: Based on the Mandarin acoustic unit sequence and the accent acoustic unit sequence, the Mandarin audio and the accent audio are transcribed into text by the speech-to-text module to obtain the corresponding Chinese character sequence;

[0009] S3: Align the Chinese character sequences of Mandarin and accents with the acoustic unit sequences through the time alignment module to obtain the corresponding Chinese character sequences, acoustic unit sequences and timestamp information;

[0010] S4: Based on the Chinese character sequence and acoustic unit sequence of Mandarin and accent, as well as the obtained timestamp information, the above sequences are aligned by the text alignment module and then compared to determine the specific locations in the accent audio that need to be modified.

[0011] S5: Based on the specific locations in the accented audio that need modification, the voice editing and correction module is used to edit the accented audio and convert it into accent-to-Mandarin audio.

[0012] Specifically, S1 is:

[0013] The acoustic unit construction module includes a semantic vector extraction module and a clustering model. The semantic vector extraction module extracts the corresponding semantic vectors from Mandarin audio and accent audio respectively. The clustering model clusters the semantic vectors of Mandarin and the semantic vectors of accent, so that the cluster centers obtained by clustering are the acoustic units corresponding to Mandarin / accent, thereby obtaining the acoustic unit sequence corresponding to Mandarin / accent.

[0014] The semantic vector extraction module includes a self-supervised acoustic model based on contrastive learning.

[0015] Specifically, S4 is:

[0016] Based on the obtained timestamp information, the acoustic unit sequence of Mandarin and the acoustic unit sequence of accent are accurately matched. Then, an editing operation is assigned to each Chinese character. After comparing the Chinese character sequence of Mandarin with the Chinese character sequence of accent, the specific position in the accent audio that needs to be modified is obtained.

[0017] Specifically, S5 is:

[0018] A Mandarin phoneme sequence is generated based on the sequence of Chinese characters. This sequence is then input into an encoder to generate an initial Mandarin latent representation, along with position embeddings and speaker embeddings. The duration of the accent audio is determined based on the accent-corrected sequence of Chinese characters and the accent audio. This duration, along with the initial Mandarin latent representation, is then input into a duration predictor to obtain the Mandarin audio duration. The initial Mandarin latent representation is then length-adjusted according to the Mandarin audio duration to obtain an adjusted Mandarin latent representation. The accent audio, the adjusted Mandarin latent representation, position embeddings, and speaker embeddings are then input into a decoder. The decoder modifies the accent audio based on the specific positions that need modification and outputs a Mel spectrogram. Finally, the accent-to-Mandarin audio is generated based on the Mel spectrogram.

[0019] In step S5, a corresponding Mandarin phoneme sequence is generated using a phoneme converter based on the Mandarin Chinese character sequence.

[0020] In S5, the duration of the Mandarin audio is input into the length adjuster, which dynamically adjusts the initial Mandarin hidden layer representation so that it is aligned with the accent audio in terms of duration.

[0021] In step S5, the accent-to-Mandarin audio is obtained by converting the Mel spectrogram into an audio signal using a vocoder.

[0022] II. A Chinese Dialect Accent Correction System Using Acoustic Units

[0023] The acoustic unit construction module is used to generate acoustic unit sequences corresponding to Mandarin audio and accent audio.

[0024] The speech-to-text module is used to transcribe Mandarin audio and accented audio into text, respectively.

[0025] The time alignment module is used to align the sequence of Chinese characters in Mandarin and accents with the sequence of acoustic units on the timestamp.

[0026] The text alignment module is used to modify the accented Chinese character sequence to obtain the accent-corrected Chinese character sequence.

[0027] The speech editing and correction module is used to convert accented audio into accent-to-Mandarin audio.

[0028] III. A computer device

[0029] The device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method.

[0030] This invention combines acoustic units with an accent correction model for detecting and correcting accents. Specifically, it uses acoustic units instead of phonemes to detect and correct Mandarin dialect accents. By applying acoustic units instead of conventional Mandarin phonemes for accent correction, this invention can more accurately detect and correct dialect accents. Furthermore, to fully utilize the semantic features in the speaker's audio, this invention employs the self-supervised acoustic model Wav2Vec2 to capture contextual information and semantic content in the audio.

[0031] The beneficial effects of this invention are:

[0032] The AU-ACM method proposed in this invention can detect and correct pronunciation errors that do not belong to the conventional phonemes of Chinese, giving the accent correction model higher adaptability and accuracy for dialect accents. This represents a significant improvement over most current accent correction methods. Experiments demonstrate that AU-ACM achieves good Mean Opinion Score (MOS, used to measure audio quality, higher is better) and Mel-frequency cepstral distortion (MCD, used to measure the difference between the Mel-frequency cepstral coefficient features of the corrected speech and the Mel-frequency cepstral coefficient features of the desired target speech, lower is better). AU-ACM can also correct different types of Chinese dialect accents into standard Mandarin. Extensive experiments have proven that this invention is a promising accent correction method. Attached Figure Description

[0033] Figure 1 This is a flowchart of the invention for detecting and correcting Chinese dialect accents;

[0034] Figure 2 This is a flowchart of the present invention for constructing acoustic units;

[0035] Figure 3 This is a flowchart of the present invention for speech editing and correction. Detailed Implementation

[0036] To enable those skilled in the art to better understand the present invention and to more clearly define the scope of protection of the present invention, the present invention will be described in detail below with reference to certain specific embodiments and accompanying drawings. It should be noted that the following are only some specific embodiments of the present invention, and are merely a part of the embodiments of the present invention. The specific and direct descriptions of related structures are only for the convenience of understanding the present invention, and the specific features do not necessarily or directly limit the scope of the present invention. Conventional selections and substitutions made by those skilled in the art under the guidance of the present invention, as well as reasonable arrangements and combinations of several technical features under the guidance of the present invention, should all be considered within the scope of protection of the present invention.

[0037] Figure 1This is a flowchart of a Chinese dialect accent detection and correction method according to an embodiment of the present invention. The acoustic unit construction module processes the input dialect accent audio and standard Mandarin audio, generating corresponding dialect accent acoustic unit sequences and standard Mandarin acoustic unit sequences, respectively. The text content is extracted from the dialect accent audio to form accent text. The obtained accent text and the transcribed standard Mandarin text, along with the aforementioned two acoustic unit sequences, are input into the alignment module to map the temporal relationship between Chinese characters and acoustic units, outputting timestamp information at the Chinese character and acoustic unit levels. Using the acquired timestamp information and the corresponding acoustic unit sequences, the specific time points and regions in the dialect accent audio that need adjustment are identified. With the help of the speech editing and correction module, the audio is modified at the identified locations requiring modification, resulting in an audio output that is more closely aligned with standard Mandarin.

[0038] Acoustic unit construction module. Figure 2 This is a flowchart illustrating the construction of acoustic units according to an embodiment of the present invention. Semantic vectors are extracted from two sets of audio data: one set is standard Mandarin audio, and the other set is audio with dialect accents. The extracted semantic vectors are clustered, and the continuous speech signal features are converted into a concise and discriminative discrete feature sequence, i.e., an acoustic unit sequence, where each acoustic unit corresponds to a cluster center point.

[0039] The speech-to-text and alignment module consists of an automatic speech recognition model used to identify Chinese character sequences from speech. A forced alignment tool is used to align the Chinese character sequences and acoustic unit sequences with the audio to obtain timestamps at both the Chinese character and acoustic unit levels.

[0040] Text alignment module. This module accurately maps acoustic units and assigns an editing operation to each Chinese character or acoustic unit. Characters are only left unmodified when acoustic units are identical. The text alignment module obtains the positional information that needs to be edited.

[0041] Voice editing and correction module. Figure 3This is a flowchart of speech editing and correction according to an embodiment of the present invention. A corresponding Mandarin phoneme sequence is obtained from standard Mandarin text. The phoneme sequence is converted into a hidden layer representation. Duration information is extracted from the accented text and its corresponding audio signal, and this information is used as a reference for adjusting the duration of the Mandarin audio and input into a duration predictor to obtain the duration of the Mandarin audio. The duration information is then input into a length adjuster, which dynamically adjusts the previously obtained Mandarin hidden layer representation to align its duration with the target accented speech. The adjusted hidden layer representation is combined with additional position embedding and speaker embedding, and a decoder module is responsible for decoding this composite information into a series of corrected Mel spectrograms. Using vocoder technology, the obtained Mel spectrograms are inversely converted into the actual audio signal.

[0042] The specific steps of the embodiment are as follows:

[0043] S1: Generate acoustic unit sequences corresponding to Mandarin audio and accent audio respectively through the acoustic unit construction module;

[0044] like Figure 2 As shown, S1 is specifically:

[0045] The acoustic unit construction module includes a semantic vector extraction module and a clustering model. The semantic vector extraction module extracts the corresponding semantic vectors from Mandarin audio and accent audio respectively. The clustering model clusters the semantic vectors of Mandarin and the semantic vectors of accent, so that the cluster centers obtained by clustering are the acoustic units corresponding to Mandarin / accent, thereby obtaining the acoustic unit sequence corresponding to Mandarin / accent.

[0046] The semantic vector extraction module includes a self-supervised acoustic model (Wav2Vec2) based on contrastive learning. Its input is 16kHz audio features, and the intermediate layer outputs a 1024-dimensional vector at 50Hz. The audio vector from the 8th layer is extracted. The clustering model uses the k-Means clustering algorithm (an unsupervised learning method for data classification) to obtain acoustic units, with the number of clusters k set to 4096.

[0047] S2: Based on the Mandarin acoustic unit sequence and the accent acoustic unit sequence, the Mandarin audio and the accent audio are transcribed into text by the speech-to-text module to obtain the corresponding Chinese character sequence; the speech-to-text module is an automatic speech recognition model based on byte pair encoding.

[0048] S3: After aligning the Chinese character sequences and acoustic unit sequences of Mandarin and accents with timestamps through the time alignment module, the corresponding Chinese character sequences and acoustic unit sequences containing timestamp information are obtained. The time alignment module is a Montreal ForcedAligner (MFA, an efficient speech alignment tool that uses machine learning to automatically analyze audio files and corresponding text to accurately locate speech boundaries) forced alignment tool based on GMM-HMM (Gaussian Mixture Model-Hidden Markov Model, a statistical modeling method). It aligns the Chinese character sequences and acoustic unit sequences with the audio to obtain timestamps at the Chinese character level and the acoustic unit level.

[0049] S4: Based on the Chinese character sequence and acoustic unit sequence of Mandarin and accent, as well as the obtained timestamp information, the above sequences are aligned by the text alignment module and then compared to determine the specific locations in the accent audio that need to be modified.

[0050] S4 specifically refers to:

[0051] Based on the obtained timestamp information, the acoustic unit sequence of Mandarin and the acoustic unit sequence of the accent are accurately matched. Then, an editing operation ("do not change", "insert", "replace", "delete") is assigned to each Chinese character. After comparing the Chinese character sequence of the Mandarin with the Chinese character sequence of the accent, the specific position in the accent audio that needs to be modified is obtained.

[0052] The text alignment module specifically employs the Needleman-Wunsch algorithm (a global sequence alignment strategy that can systematically evaluate and minimize the total number of editing operations required to transform one sequence into another, covering basic operation types such as insertion, replacement, and deletion).

[0053] S5: Based on the specific locations in the accented audio that need modification, the voice editing and correction module is used to edit the accented audio and convert it into accent-to-Mandarin audio.

[0054] like Figure 3 As shown, S5 specifically refers to:

[0055] Based on the Chinese character sequence of Mandarin, a phoneme converter generates a Mandarin phoneme sequence. This sequence is then input into an encoder to generate an initial Mandarin latent representation, along with position embeddings and speaker embeddings. The duration of the accent audio is then determined based on the accent-corrected Chinese character sequence and the accent audio. This duration, along with the initial Mandarin latent representation, is then input into a duration predictor to obtain the Mandarin audio duration. The initial Mandarin latent representation is then length-adjusted according to the Mandarin audio duration to obtain an adjusted Mandarin latent representation. Finally, the accent audio, the adjusted Mandarin latent representation, position embeddings, and speaker embeddings are input into a decoder. The decoder modifies the accent audio based on the specific positions requiring modification and outputs a Mel spectrogram. Finally, the accent-to-Mandarin audio is generated based on the Mel spectrogram.

[0056] The duration of the Mandarin audio is input into the length adjuster, which dynamically adjusts the initial Mandarin hidden layer representation to align its duration with the accented audio.

[0057] Generating accent-to-Mandarin audio from a Mel spectrogram involves converting the Mel spectrogram into an audio signal using a vocoder. The vocoder is specifically a Parallel-WaveGAN (a fast and space-efficient waveform generation method using a non-distilled generative adversarial network) vocoder.

[0058] The performance of accent correction was evaluated using the KeSpeech dataset (which contains 1,542 hours of speech data with dialect accents recorded by 27,237 speakers from 34 cities in my country, including standard Mandarin and 8 sub-dialects) for training and testing.

[0059] The model settings in the speech editing and correction module are as follows: The text encoder contains a 512-dimensional trainable lookup table, three one-dimensional convolutional layers, each with a kernel size of 5 and 512 channels, and a 512-dimensional bidirectional Long Short-Term Memory (LSTM) network. The speaker encoder contains a 128-dimensional trainable lookup table. The Prenet consists of two linear layers with dimensions (80, 256) and (256, 256). The forward and backward decoders each consist of two unidirectional 1024-dimensional LSTMs and a shared (1024, 80) linear layer. The duration predictor consists of a two-layer bidirectional 512-dimensional LSTM and a (512, 1) linear layer.

[0060] In the acoustic unit construction module, the Wav2Vec2 model, which has undergone self-supervised learning on a multi-dialect accent dataset, is selected as the front-end semantic feature vector extraction model. Its input is 16kHz audio features, and the intermediate layer output is a 1024-dimensional vector at 50Hz. The audio vectors from the 8th layer are extracted, and these vectors are used to train the k-Means clustering model. For the discrete acoustic unit sequence S, the number of clusters K is set to 4096.

[0061] In the data preprocessing stage, all audio samples are first downsampled to 16kHz. During training, a batch processing strategy is used, with each batch containing 32 samples. Then, the audio is denoised using the Wave-U-Net network. When performing the Short Time Fourier Transform (STFT) on the audio, a Haining window is used with a window length of 512 and a frame shift length of 160.

[0062] To evaluate the performance of this invention in correcting Chinese dialect accents, it was trained and tested using the KeSpeech dataset. The training parameters were set as follows: the Adam optimizer was used for model training, with β1 of 0.9 and β2 of 0.98. The initial learning rate was set to 1e-4, and a learning rate decay mechanism was introduced during training, with the minimum learning rate set to 1e-5.

[0063] Accent correction performance evaluation. MOS and MCD were used as the main evaluation metrics. The AU-ACM model of this invention was compared with several typical accent correction models. Since most accent conversion models use English datasets, the KeSpeech dataset was used to train these accent correction models, and the training process was consistent with AU-ACM. The trained models (denoted by *) were used for comparative experiments.

[0064] The AU-ACM model of this invention was compared horizontally with several accent correction models based on speech conversion (FAC-PPG, PPG-GMM, Zero-Shot FAC) in terms of correction effect, and the results are shown in Table 1. MOSn is the MOS value of speech naturalness with a 95% confidence interval, and MOScs is the MOS value of speech content similarity with a 95% confidence interval. Content similarity is defined as whether the Chinese characters contained in the speech before and after correction are the same. As can be seen from Table 1, the AU-ACM model achieved the best performance in both the MCD test and the MOS tests for speech naturalness and content similarity. This reflects the effectiveness and robustness of the AU-ACM model of this invention in accent correction, as well as the rationality of using acoustic units for Chinese dialect accent correction proposed in this invention.

[0065] Table 1 compares the accent correction performance of the present invention with that of existing methods.

[0066] Model MCD↓ <![CDATA[MOS n ↑]]> <![CDATA[MOS cs ↑]]> FAC-PPG* 5.13 3.58±0.19 3.65±0.21 PPG-GMM* 5.19 3.55±0.23 3.66±0.18 Zero-Shot FAC* 4.91 3.70±0.19 3.86±0.21 AU-ACM 4.83 3.73±0.22 3.94±0.23

[0067] Finally, it should be noted that the above embodiments and descriptions are only used to illustrate the technical solutions of the present invention and not to limit it. Those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the disclosure of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the protection scope of the claims of the present invention.

Claims

1. A method for correcting Mandarin dialect accents using acoustic units, characterized in that, Includes the following steps: S1: Generate acoustic unit sequences corresponding to Mandarin audio and accent audio respectively through the acoustic unit construction module; S2: Based on the Mandarin acoustic unit sequence and the accent acoustic unit sequence, the Mandarin audio and the accent audio are transcribed into text by the speech-to-text module to obtain the corresponding Chinese character sequence; S3: Align the Chinese character sequences of Mandarin and accents with the acoustic unit sequences through the time alignment module to obtain the corresponding Chinese character sequences, acoustic unit sequences and timestamp information; S4: Based on the Chinese character sequence and acoustic unit sequence of Mandarin and accent, as well as the obtained timestamp information, the above sequences are aligned by the text alignment module and then compared to determine the specific locations in the accent audio that need to be modified. S5: Based on the specific locations in the accented audio that need modification, the voice editing and correction module is used to edit the accented audio and convert it into accent-to-Mandarin audio.

2. The method for correcting Chinese dialect accents using acoustic units according to claim 1, characterized in that, Specifically, S1 is: The acoustic unit construction module includes a semantic vector extraction module and a clustering model. The semantic vector extraction module extracts the corresponding semantic vectors from Mandarin audio and accent audio respectively. The clustering model clusters the semantic vectors of Mandarin and the semantic vectors of accent, so that the cluster centers obtained by clustering are the acoustic units corresponding to Mandarin / accent, thereby obtaining the acoustic unit sequence corresponding to Mandarin / accent.

3. The method for correcting Chinese dialect accents using acoustic units according to claim 2, characterized in that, The semantic vector extraction module includes a self-supervised acoustic model based on contrastive learning.

4. The method for correcting Chinese dialect accents using acoustic units according to claim 1, characterized in that, Specifically, S4 is: Based on the obtained timestamp information, the acoustic unit sequence of Mandarin and the acoustic unit sequence of accent are accurately matched. Then, an editing operation is assigned to each Chinese character. After comparing the Chinese character sequence of Mandarin with the Chinese character sequence of accent, the specific position in the accent audio that needs to be modified is obtained.

5. The method for correcting Chinese dialect accents using acoustic units according to claim 1, characterized in that, Specifically, S5 is: A Mandarin phoneme sequence is generated based on the sequence of Chinese characters in Mandarin. The Mandarin phoneme sequence is then input into the encoder to generate the initial Mandarin hidden layer representation, as well as the position embedding and speaker embedding. The duration of the accent audio is then determined based on the accent-corrected sequence of Chinese characters and the accent audio. Then, the accent audio duration and the initial Mandarin latent representation are input into the duration predictor to obtain the Mandarin audio duration. Next, the length of the initial Mandarin latent representation is adjusted according to the Mandarin audio duration to obtain the adjusted Mandarin latent representation. Then, the accent audio, the adjusted Mandarin latent representation, the position embedding, and the speaker embedding are input into the decoder. The decoder modifies the accent audio based on the specific positions in the accent audio that need to be modified and outputs a Mel spectrogram. Finally, the accent-to-Mandarin audio is generated based on the Mel spectrogram.

6. The method for correcting Chinese dialect accents using acoustic units according to claim 5, characterized in that, In step S5, a corresponding Mandarin phoneme sequence is generated using a phoneme converter based on the Mandarin Chinese character sequence.

7. A method for correcting Chinese dialect accents using acoustic units according to claim 5, characterized in that, In S5, the duration of the Mandarin audio is input into the length adjuster, which dynamically adjusts the initial Mandarin hidden layer representation so that it is aligned with the accent audio in terms of duration.

8. A method for correcting Chinese dialect accents using acoustic units according to claim 5, characterized in that, In step S5, the accent-to-Mandarin audio is obtained by converting the Mel spectrogram into an audio signal using a vocoder.

9. A Chinese dialect accent correction system employing acoustic units, characterized in that, include: The acoustic unit construction module is used to generate acoustic unit sequences corresponding to Mandarin audio and accent audio. The speech-to-text module is used to transcribe Mandarin audio and accented audio into text, respectively. The time alignment module is used to align the sequence of Chinese characters in Mandarin and accents with the sequence of acoustic units on the timestamp. The text alignment module is used to modify the accented Chinese character sequence to obtain the accent-corrected Chinese character sequence. The speech editing and correction module is used to convert accented audio into accent-to-Mandarin audio.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images