Chinese and Korean bilingual speech recognition and synthesis system

Abstract

The invention relates to a Chinese and Korean bilingual speech recognition and synthesis system based on dialect accent, and aims to solve the recognition and synthesis problem caused by accent and dialect difference in cross-language communication. The system adopts a modular design and comprises a preprocessing module, a feature extraction module, a model training module and a post-processing module. Through a deep learning algorithm, such as a convolutional neural network (CNN) and a recurrent neural network (RNN), and in combination with a long-short term memory network (LSTM) or a gated loop unit (GRU), the system can accurately identify Chinese and Korean voices of different accent and dialect. Meanwhile, a multi-task learning and transfer learning strategy is adopted, a voice library containing various accent and dialects is constructed, an attention mechanism and an adversarial training technology are introduced, and the generalization ability of the model for unseen accent and dialects is improved.

Description

Technical Field

[0001] The present invention relates to the field of translation systems, and in particular to a Chinese-Korean bilingual speech recognition and synthesis system. Background Art

[0002] With the deepening of globalization, the need for cross-language communication has become increasingly prominent, especially between countries with frequent exchanges in the fields of economy, culture and education. As important economies in Asia, China and South Korea have increasingly close exchanges and cooperation. Whether it is business negotiations, cultural exchanges or educational cooperation, it is necessary to overcome language barriers. Therefore, it is particularly important to develop an efficient and accurate Chinese-Korean bilingual speech recognition and synthesis system.

[0003] Although existing speech recognition technology can meet the needs of daily communication to a certain extent, it still faces great challenges in dealing with different accents and dialects. China and South Korea have vast territories and large populations, and the accents and dialects in different places vary significantly. This diversity not only increases the difficulty of speech recognition, but also limits the application of speech recognition technology in a wider range of fields. For example, in business negotiations, if the speech recognition system cannot accurately recognize the other party's accent or dialect, it may lead to misunderstandings or poor communication, which in turn affects the smooth progress of cooperation.

[0004] At the same time, for speech synthesis technology, generating natural, fluent and local speech is also a difficult point in current research. Traditional speech synthesis methods are often based on fixed speech libraries and rules, which are difficult to adapt to diverse pronunciation habits and intonation changes. The language and culture of China and South Korea are rich and diverse, and the pronunciation habits and intonation characteristics of different places are different. Therefore, how to generate speech that is both in line with local pronunciation habits and natural and fluent has become an important problem that speech synthesis technology needs to solve.

[0005] In addition, with the continuous development of artificial intelligence and machine learning technologies, advanced technologies such as deep learning have been widely used in the field of speech recognition and synthesis. However, these technologies still face many challenges when dealing with Chinese and Korean bilingual speech recognition and synthesis. For example, how to effectively extract the feature vectors of Chinese and Korean speech, how to build a deep learning model suitable for Chinese and Korean bilinguals, and how to optimize model parameters to improve recognition accuracy and the naturalness of synthesized speech.

[0006] In summary, developing a Chinese-Korean bilingual speech recognition and synthesis system that can adapt to different accents and dialects not only has important academic value, but also has broad application prospects.

[0007] In order to solve the above problems, the applicant proposes a Chinese-Korean bilingual speech recognition and synthesis system. Summary of the invention

[0008] The purpose of the present invention is to provide a Chinese-Korean bilingual speech recognition and synthesis system to solve the problems in the prior art.

[0009] To achieve the above object, the present invention provides the following technical solution: a Chinese-Korean bilingual speech recognition and synthesis system, comprising:

[0010] The preprocessing module is used to perform operations such as noise reduction and endpoint detection on the input voice signal to improve the signal quality;

[0011] The feature extraction module uses methods such as Mel-frequency cepstral coefficients (MFCC) and wavelet transform to extract the feature vector of speech;

[0012] Model training module, which uses deep learning algorithms such as convolutional neural network (CNN) and recurrent neural network (RNN), combined with long short-term memory network (LSTM) or gated recurrent unit (GRU) to improve the model's ability to learn speech features;

[0013] The post-processing module is used to convert the phoneme sequence output by the model into text and perform error correction and semantic understanding;

[0014] The speech synthesis module uses an end-to-end speech synthesis framework based on deep learning, such as the Tacotron 2 model, combined with an attention mechanism and a vocoder to generate natural and fluent Chinese-Korean bilingual speech.

[0015] Optionally, the system further includes a processing mechanism for accents and dialects in Chinese and Korean speech, specifically:

[0016] Collect voice data from different regions and groups to build a voice library containing multiple accents and dialects;

[0017] During the model training phase, an attention mechanism is introduced to enhance the model’s ability to focus on specific accent and dialect features;

[0018] Adversarial training techniques are used to improve the model's generalization ability to unseen accents and dialects.

[0019] Optionally, the system further includes a strategy for improving recognition accuracy, specifically:

[0020] Increase the diversity of training data and improve the robustness of the model through data enhancement technology;

[0021] Introducing ensemble learning methods to combine the prediction results of multiple models to improve the overall recognition accuracy;

[0022] Implement fine-grained error analysis to optimize model parameters in a targeted manner and reduce the occurrence of common errors.

[0023] Optionally, the system also includes optimization solutions for teaching and translation scenarios, specifically:

[0024] Build domain-specific vocabularies and phrase libraries to improve the system's ability to understand and express content in specific domains;

[0025] Supports multiple teaching modes, such as shadowing practice and pronunciation correction, to help learners improve their language proficiency.

[0026] Optionally, the system further includes a method for evaluating speech synthesis effect, specifically:

[0027] Adopt objective evaluation methods, such as calculating the difference between synthesized speech and real recordings, using metrics such as mean opinion score (MOS) and spectral distance;

[0028] A subjective evaluation method is used to evaluate the naturalness, fluency and intelligibility of synthesized speech through user listening tests.

[0029] Beneficial effects: 1. Enhanced the accuracy of speech recognition and synthesis

[0030] The present invention significantly improves the accuracy of speech recognition and synthesis by adopting advanced deep learning algorithms, such as convolutional neural networks (CNN), recurrent neural networks (RNN), and long short-term memory networks (LSTM) or gated recurrent units (GRU), and combining multi-task learning and transfer learning strategies. In particular, for the accent and dialect problems in Chinese and Korean speech, this system can strengthen the model's ability to focus on specific features, and further improve recognition accuracy through data enhancement technology and integrated learning methods. Experimental results show that when processing Mandarin with a Northeastern accent, the word error rate (WER) of this system is 15% lower than that of the traditional system, and when processing the Jeju Island dialect of South Korea, the sentence error rate (SER) is reduced by 20%.

[0031] 2. Improved the naturalness and fluency of speech synthesis

[0032] In terms of speech synthesis, the present invention adopts an end-to-end speech synthesis framework based on deep learning, and uses the Tacotron 2 model and its related components to generate high-quality speech waveforms directly from text. By introducing language models and stress models, the system further improves the naturalness and fluency of synthesized speech. In addition, in response to the special needs of teaching and translation scenarios, the system also optimizes the processing of professional terms and commonly used expressions, and improves the system's ability to understand and express content in specific fields. Experimental evaluation shows that the synthesized Chinese-Korean bilingual speech achieved a mean opinion score (MOS) of 4.5 points (out of 5 points), showing a high level of user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the structure of an embodiment of the present invention;

[0034] Figure 2 It is the flow chart of preprocessing module;

[0035] Figure 3 It is the flow chart of feature extraction module;

[0036] Figure 4 This is the flow chart of the model training module;

[0037] Figure 5 It is the flow chart of post-processing module;

[0038] Figure 6 This is a flowchart of Chinese and Korean speech synthesis technology;

[0039] Figure 7 Flowchart for performance evaluation and optimization. DETAILED DESCRIPTION

[0040] The following describes the preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0041] In the drawings, components with the same structure are indicated by the same numerical reference numerals, and components with similar structures or functions are indicated by similar numerical reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present invention does not limit the size and thickness of each component. In order to make the illustration clearer, the thickness of the components is appropriately exaggerated in some places in the drawings.

[0042] Example 1

[0043] The present invention relates to the field of speech recognition and synthesis technology, and in particular to a Chinese-Korean bilingual speech recognition and synthesis system based on dialect accents. The specific implementation of the present invention will be described in detail below to better understand the technical solution and effects of the present invention.

[0044] The present invention relates to the field of speech recognition and synthesis technology, and in particular to a Chinese-Korean bilingual speech recognition and synthesis system based on dialect accents. The specific implementation of the present invention will be described in detail below to better understand the technical solution and effects of the present invention.

[0045] 1. System Overview

[0046] The present invention aims to design and implement a Chinese-Korean bilingual speech recognition system that can accurately recognize different accents and dialects, and develop corresponding speech synthesis technology based on this. The system adopts a modular design, including a preprocessing module, a feature extraction module, a model training module, a post-processing module, and a speech synthesis module. By improving the recognition accuracy and the naturalness of the synthesized speech, the system aims to provide a more efficient and convenient tool for teaching and translation.

[0047] (II) Detailed description of system modules

[0048] 1. Preprocessing module

[0049] The preprocessing module is the primary link in the speech recognition system. Its main task is to perform operations such as noise reduction and endpoint detection on the input speech signal to improve the signal quality and provide a good foundation for subsequent feature extraction and model recognition.

[0050] (1) Noise reduction processing

[0051] Noise reduction is one of the important functions of the preprocessing module. In practical applications, speech signals are often interfered by various background noises, such as environmental noise, equipment noise, etc. These noises will reduce the signal-to-noise ratio of the speech signal, thereby affecting the subsequent recognition effect. Therefore, the present invention adopts advanced noise reduction algorithms, such as spectral subtraction, Wiener filtering, etc., to perform noise reduction on the input speech signal to improve the purity and clarity of the signal.

[0052] (2) Endpoint Detection

[0053] Endpoint detection refers to accurately extracting the starting point and ending point of a speech segment from an input speech signal, that is, determining the effective range of the speech signal. The accuracy of endpoint detection is crucial for subsequent feature extraction and model recognition. The present invention adopts an energy-based endpoint detection algorithm to accurately determine the starting point and ending point of a speech segment by calculating features such as the short-time energy and short-time zero-crossing rate of a speech signal, thereby avoiding interference of invalid signals on subsequent processing.

[0054] 2. Feature Extraction Module

[0055] The feature extraction module is one of the key links in the speech recognition system. Its main task is to extract information that can represent speech features from the preprocessed speech signal, such as Mel-frequency cepstral coefficients (MFCC), wavelet transform coefficients, etc. These feature information will be used for subsequent model training and recognition.

[0056] (1) Mel-frequency cepstral coefficients (MFCC)

[0057] MFCC is a speech feature parameter based on the human hearing characteristics, which can effectively reflect the spectral envelope information of speech. The present invention uses MFCC as one of the speech features, calculates the Mel spectrum of the speech signal, and performs operations such as logarithmic transformation and discrete cosine transformation on it to obtain MFCC feature vectors. These feature vectors can better reflect the spectral characteristics of speech and provide strong support for subsequent model training.

[0058] (2) Wavelet transform coefficients

[0059] Wavelet transform is a time-frequency analysis method that can decompose speech signals into wavelet components of different scales and frequencies. The present invention uses wavelet transform as another method for extracting speech features. By calculating the wavelet transform coefficients of speech signals, the distribution of speech signals at different scales and frequencies is obtained. These feature information can better reflect the time-frequency characteristics of speech and provide rich feature vectors for subsequent model recognition.

[0060] 3. Model training module

[0061] The model training module is one of the core parts of the speech recognition system. Its main task is to train a model that can accurately recognize speech signals based on the extracted feature vectors. The present invention uses deep learning algorithms, such as convolutional neural networks (CNN) and recurrent neural networks (RNN), combined with long short-term memory networks (LSTM) or gated recurrent units (GRU), to improve the model's learning ability and recognition accuracy for speech features.

[0062] (1) Convolutional Neural Network (CNN)

[0063] CNN is a neural network model with a deep structure that can automatically learn the feature representation of input data. The present invention uses CNN as part of the model training. By constructing multiple convolutional layers and pooling layers, the input feature vector is extracted and reduced layer by layer to obtain a more abstract and robust feature representation. These feature representations will be used in subsequent recurrent neural network models such as RNN or LSTM to improve the recognition ability of the model.

[0064] (2) Recurrent Neural Network (RNN) and its variants (LSTM / GRU)

[0065] RNN is a neural network model that can process sequence data, and it can capture the time dependency in sequence data. However, traditional RNN is prone to problems such as gradient vanishing or gradient exploding when processing long sequence data, which limits its effectiveness in practical applications. In order to solve this problem, the present invention adopts RNN models in variant forms such as LSTM or GRU. These models effectively alleviate problems such as gradient vanishing or gradient exploding by introducing structures such as gating mechanisms and memory units, and improve the model's ability to handle long sequence data. During the model training process, the present invention uses technical means such as back-propagation algorithms and optimizers to iteratively update and optimize model parameters to obtain a more accurate and robust model.

[0066] 4. Post-processing module

[0067] The post-processing module is the last link of the speech recognition system. Its main task is to convert the phoneme sequence output by the model into text and perform error correction and semantic understanding. The present invention uses technical means such as rule-based error correction algorithms and semantic understanding algorithms to process and optimize the phoneme sequence output by the model to improve the accuracy and reliability of recognition.

[0068] (1) Error Correction Algorithm

[0069] Error correction algorithm refers to the correction and optimization of the phoneme sequence output by the model according to language rules and grammatical knowledge. The present invention adopts a rule-based error correction algorithm to match and correct the phoneme sequence output by the model by building resources such as language models and dictionaries to reduce the occurrence of recognition errors and ambiguity.

[0070] (2) Semantic Understanding Algorithm

[0071] The semantic understanding algorithm refers to the semantic analysis and understanding of the text output by the model to extract the key information and intent. The present invention uses an algorithm based on semantic understanding to conduct in-depth analysis and understanding of the text output by the model, extract the key information and intent, and provide more accurate and reliable support for subsequent applications.

[0072] 5. Speech Synthesis Module

[0073] The speech synthesis module is another important component of the present invention, and its main task is to convert the input text information into natural and fluent speech. The present invention adopts an end-to-end speech synthesis framework based on deep learning, such as the Tacotron 2 model, combined with technical means such as attention mechanism and vocoder, to achieve high-quality conversion from text to speech.

[0074] (1) Tacotron 2 model

[0075] Tacotron 2 is an end-to-end speech synthesis model based on deep learning, which can generate high-quality speech waveforms directly from text. The present invention uses the Tacotron 2 model as the main engine for speech synthesis, and trains and optimizes the model parameters to enable it to generate natural and fluent speech close to real people's voice. During the model training process, the present invention uses a large amount of speech data and text data as training samples, and iteratively updates and optimizes the model through technical means such as back-propagation algorithms and optimizers.

[0076] (2) Attention Mechanism and Vocoder

[0077] The attention mechanism is a mechanism that can focus on important information in input data, and it plays a vital role in speech synthesis. The present invention uses the attention mechanism to strengthen the model's ability to focus on key information in the input text, thereby improving the accuracy and naturalness of speech synthesis. At the same time, the present invention also uses a vocoder (such as WaveGlow) as a back-end processing tool for speech synthesis to further optimize and beautify the generated speech waveform to make it sound more natural and smooth.

[0078] 3. ...

[0079] In order to solve the problems of accent and dialect in Chinese and Korean speech, the present invention adopts the strategies of multi-task learning and transfer learning to improve the recognition ability and adaptability of the system.

[0080] 1. Multi-task learning

[0081] Multi-task learning is a method of learning multiple related tasks simultaneously, which can improve the generalization ability of the model by sharing representations and knowledge. The present invention adopts a multi-task learning strategy to consider the recognition tasks of multiple accents and dialects simultaneously during the model training process. By constructing a speech library containing multiple accents and dialects as training data and introducing an attention mechanism to strengthen the model's ability to focus on specific accent and dialect features, the present invention improves the model's ability to process diverse speech data.

[0082] 2. Transfer Learning

[0083] Transfer learning is a method of transferring knowledge learned on one task to another task. The present invention adopts a transfer learning strategy, uses an existing speech recognition model as a basic model, and improves the recognition ability of the system by fine-tuning and optimizing the target task (i.e., Chinese-Korean bilingual speech recognition). By collecting and annotating the speech data of the target task, and using it as a training sample to fine-tune and optimize the basic model, the present invention improves the adaptability and recognition accuracy of the system to the target task.

[0084] (IV) Strategies to improve recognition accuracy

[0085] In order to further improve the recognition accuracy of the system, the present invention adopts the following strategies:

[0086] 1. Data Augmentation Techniques

[0087] Data enhancement technology is a method to improve the robustness of the model by increasing the diversity and richness of the training data. The present invention uses data enhancement techniques such as time warping and noise injection to process the training data to increase the diversity and richness of the data. Through these processing methods, the present invention improves the robustness of the model to noise and changes, thereby further improving the recognition accuracy of the system.

[0088] 2. Ensemble learning methods

[0089] The ensemble learning method is a method that combines the prediction results of multiple models to improve the overall recognition accuracy. The present invention adopts the ensemble learning method to combine and optimize multiple speech recognition models with different structures and characteristics. By integrating the prediction results of multiple models and performing operations such as voting or weighted averaging, the present invention improves the accuracy and reliability of overall recognition.

[0090] 3. Fine-grained error analysis

[0091] Fine-grained error analysis is a method for carefully analyzing and optimizing model recognition errors. The present invention adopts a fine-grained error analysis method to classify and count common errors that occur during model recognition, and optimizes model parameters and training strategies in a targeted manner to reduce the occurrence of these errors. Through these optimization measures, the present invention further improves the recognition accuracy and performance of the system.

[0092] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic features of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the attached claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any figure mark in the claims should not be regarded as limiting the claims involved.

[0093] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A Chinese-Korean bilingual speech recognition and synthesis system, characterized in that: include: The preprocessing module is used to perform operations such as noise reduction and endpoint detection on the input voice signal to improve the signal quality; The feature extraction module uses methods such as Mel-frequency cepstral coefficients (MFCC) and wavelet transform to extract the feature vector of speech; Model training module, which uses deep learning algorithms such as convolutional neural network (CNN) and recurrent neural network (RNN), combined with long short-term memory network (LSTM) or gated recurrent unit (GRU) to improve the model's ability to learn speech features; The post-processing module is used to convert the phoneme sequence output by the model into text and perform error correction and semantic understanding; The speech synthesis module uses an end-to-end speech synthesis framework based on deep learning, such as the Tacotron 2 model, combined with an attention mechanism and a vocoder to generate natural and fluent Chinese-Korean bilingual speech.

2. The Chinese-Korean bilingual speech recognition and synthesis system according to claim 1, characterized in that: The system also includes a processing mechanism for accents and dialects in Chinese and Korean speech, specifically: Collect voice data from different regions and groups to build a voice library containing multiple accents and dialects; During the model training phase, an attention mechanism is introduced to enhance the model’s ability to focus on specific accent and dialect features; Adversarial training techniques are used to improve the model's generalization ability to unseen accents and dialects.

3. The Chinese-Korean bilingual speech recognition and synthesis system according to claim 1, characterized in that: The system also includes strategies for improving recognition accuracy, specifically: Increase the diversity of training data and improve the robustness of the model through data enhancement technology; Introducing ensemble learning methods to combine the prediction results of multiple models to improve the overall recognition accuracy; Implement fine-grained error analysis to optimize model parameters in a targeted manner and reduce the occurrence of common errors.

4. The Chinese-Korean bilingual speech recognition and synthesis system according to claim 1, characterized in that: The system also includes optimization solutions for teaching and translation scenarios, specifically: Build domain-specific vocabularies and phrase libraries to improve the system's ability to understand and express content in specific domains; Supports multiple teaching modes, such as shadowing practice and pronunciation correction, to help learners improve their language proficiency.

5. The Chinese-Korean bilingual speech recognition and synthesis system according to claim 1, characterized in that: The system also includes a method for evaluating speech synthesis effect, specifically: Adopt objective evaluation methods, such as calculating the difference between synthesized speech and real recordings, using metrics such as mean opinion score (MOS) and spectral distance; A subjective evaluation method is used to evaluate the naturalness, fluency and intelligibility of synthesized speech through user listening tests.

Images