Tibetan language Anduodialect speech enhancement method suitable for specific noise and general noise

Abstract

The invention discloses a Tibetan language Andole speech enhancement method suitable for specific noise and general noise, and relates to the technical field of low-resource language speech processing. The method specifically comprises the following steps: acquiring noisy voice information; preprocessing the noisy voice information, and dividing the processed data into a training set and a test set; a noise spectrum feature input by the noise classification module is optimized based on a weighted average method of a logarithmic power spectrum domain hard threshold; constructing a noise classification task and noise suppression task fused multi-target Tibetan speech enhancement model; training and testing the built speech enhancement model by using the training set and the test set to obtain a trained speech enhancement model; and inputting the actually collected noisy speech into the trained speech enhancement model to obtain an enhanced logarithmic power spectrum. Through effective fusion of the noise classification task and the suppression task, the quality and the intelligibility of the Tibetan language Andor dialect enhancement voice are improved, the human-computer interaction of the Tibetan language Andor dialect is more perfect, and the method has a positive effect on popularization of Tibetan language informatization.

Description

Technical Field

[0001] The invention belongs to the technical field of low-resource language speech processing, and in particular relates to a Tibetan Amdo dialect speech enhancement method suitable for specific noise and general noise. Background Art

[0002] Traditional speech enhancement methods, such as spectral subtraction, Wiener filtering, and minimum mean square error (MMSE), have the advantages of low computational complexity and easy implementation. These methods show good noise suppression capabilities under stationary noise conditions. However, under non-stationary noise conditions, the performance of these methods will be significantly reduced. The development of deep learning technology has brought new ideas to the field of speech enhancement and achieved very good performance under non-stationary noise conditions.

[0003] With the use of deep learning technology in the field of speech enhancement, resource-rich languages ​​such as Chinese and English have made great progress in both basic theoretical research on speech enhancement technology and industrial transformation of research results. However, most of the areas where Tibetan speakers live are in economically underdeveloped areas. Due to factors such as the lack of research accumulation in linguistics and the lack of professional researchers, the research on Tibetan speech enhancement technology started later than that of resource-rich languages, and there are few research results. After 2013, only a few academic papers on Tibetan speech enhancement or noise reduction can be found on the Internet, and the related research is mainly concentrated in laboratories, and relatively mature practical products have not yet been formed.

[0004] In addition, Tibetan has three dialects: U-Tsang, Amdo, and Kham, and the number of people using each dialect is basically equal. However, since the area centered on Lhasa is the political, religious, economic, and cultural center of the Tibetan region, most of the current research on Tibetan speech enhancement technology is mainly focused on the U-Tsang dialect. Compared with the U-Tsang dialect, the research results on the Amdo dialect in speech enhancement are weaker, and there are fewer professional researchers. At the same time, since the three dialects have their own pronunciation characteristics, the speech enhancement model that obtains good enhancement performance on the U-Tsang dialect may not obtain the same enhancement results on the other two dialects.

[0005] Furthermore, the application scenarios of most voice products, such as voice assistants, smart speakers, and voice control systems, are generally fixed, which means that the types of noise that appear in the scenarios where these products are used are almost constant and fixed. The optimal parameters of the voice enhancement model must be different for different noise types. If you know in advance the types of noise that may appear in the application scenarios of voice products, you can optimize the model in a targeted manner to improve the noise reduction performance of the enhancement model for specific noise. At the same time, various types of sudden noise may appear at any time in the scenarios where these products are used. Therefore, these products not only need to have accurate suppression capabilities for specific noises that appear in the scenarios, but also need to have good noise reduction capabilities for other general noises. Summary of the invention

[0006] In view of the above problems, the present invention proposes a Tibetan Amdo dialect speech enhancement method suitable for specific noise and general noise, which uses the idea of ​​combining dedicated paths with shared paths and introduces prior environmental information to make the enhancement model highly sensitive and accurate to a certain type of noise. Furthermore, through the multi-objective learning framework of "classification + enhancement", the model can process unconventional noise based on the similarity or complementarity between previously trained noise types even if it encounters a noise type that has not been trained in a specific noise scene, thereby improving the noise reduction capability of general noise.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A Tibetan Amdo dialect speech enhancement method suitable for specific noise and general noise comprises the following steps:

[0009] S1: Acquire noisy speech information;

[0010] S2: preprocessing the noisy speech information to obtain preprocessed speech information, and dividing the preprocessed data into a training set and a test set;

[0011] S3: Based on the weighted average method of hard threshold in logarithmic power spectrum domain, the noise spectrum characteristics of the noise classification module input are optimized;

[0012] S4: Construct a multi-objective Tibetan speech enhancement model integrating noise classification task and noise suppression task;

[0013] S5: Use the training set and the test set to train and test the built speech enhancement model to obtain a trained speech enhancement model;

[0014] S6: Input the actually collected noisy speech into the trained speech enhancement model to obtain the enhanced logarithmic power spectrum.

[0015] In the above method, optionally, the preprocessing in S2 adopts short-time Fourier transform.

[0016] In the above method, optionally, the noise spectrum characteristic calculation formula in S3 is as follows:

[0017]

[0018] E(k,l)=α·E(k,l-1)+(1-α)·N(k,l)

[0019] else

[0020] E(k,l)=E(k,l-1)

[0021] Where N(k,l) represents the logarithmic power spectrum of the noisy speech, E(k,l) represents the estimated logarithmic power spectrum, l is the index of the time frame in the spectrum, and k is the index of the frequency unit in the spectrum.

[0022] In the above method, optionally, the speech enhancement model in S4 includes a noise suppression module and a noise classification module;

[0023] The noise classification module classifies and identifies the noise of the input model, and transmits the output of the noise classification module as the attention gating factor to the noise suppression module, guiding the noise suppression module to obtain the maximum noise suppression performance benefit;

[0024] The noise classification module contains three layers of fully connected neural network. The number of neurons in the output layer is the number of noise classifications. The Softmax function is used to convert the output of the neurons into a probability distribution. Each hidden layer consists of 1024 neurons using the rectified linear unit activation function.

[0025] The objective function in the noise classification module adopts the cross entropy function, which is expressed as follows:

[0026]

[0027] Where m represents the amount of noise, L i represents the label of the i-th noise;

[0028] The noise suppression module contains m+1 branch networks. The first m branches perform noise reduction on different noise types, while the last branch is responsible for processing all types of noise.

[0029] In the above method, optionally, during the training process of the noise suppression module, the mean square error MSE is used as the error metric, and the expression is as follows:

[0030]

[0031] in, represents the logarithmic power spectrum characteristics of the enhanced speech, and Y(k,l) represents the logarithmic power spectrum characteristics of the clean speech.

[0032] It can be seen from the above technical solution that, compared with the prior art, the present invention provides a Tibetan Amdo dialect speech enhancement method suitable for specific noise and general noise, which has the following beneficial effects:

[0033] The present invention uses a multi-objective learning framework of noise classification and noise suppression to intelligently identify the type of noise in each frame of language signal, and use this to guide the training of different branches of the noise reduction network, so that it can adopt more accurate processing strategies for different noise types. At the same time, through information sharing and collaboration between branches, the complementary advantages between different noise types are fully utilized to improve the generalization performance of the model to noise. By introducing prior noise information in advance, it is not only possible to solve the noise problem in a targeted manner and improve the speech quality and intelligibility of noisy language in a specific noise environment, but also to enhance the adaptability and generalization ability of the model to noise, providing a more effective noise suppression solution for various practical application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0035] Figure 1 A flow chart of a method for speech enhancement of the Tibetan Amdo dialect suitable for specific noise and general noise disclosed by the present invention;

[0036] Figure 2 A flow chart of a speech enhancement method disclosed in an embodiment of the present invention;

[0037] Figure 3 The present invention is a schematic diagram of the structure of the speech enhancement method disclosed in the embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0040] The present invention discloses a Tibetan Amdo dialect speech enhancement method suitable for specific noise and general noise, which consists of two modules: noise suppression and noise classification. The noise suppression module introduces a priori noise environment information to further improve the model's enhancement performance for specific noise, so that the model can more accurately identify and remove noise in a specific noise environment and retain clear speech signals. The noise classification module classifies and identifies multiple noises, and passes the output as an attention gating factor to the noise suppression module, guiding the noise suppression module to obtain the maximum noise suppression performance benefit. The specific method is as follows: refer to Figure 1 As shown, a Tibetan Amdo dialect speech enhancement method suitable for specific noise and general noise includes the following steps:

[0041] S1: Acquire noisy speech information;

[0042] S2: preprocessing the noisy speech information to obtain preprocessed speech information, and dividing the preprocessed data into a training set and a test set;

[0043] S3: Based on the weighted average method of hard threshold in logarithmic power spectrum domain, the noise spectrum characteristics of the noise classification module input are optimized;

[0044] S4: Construct a multi-objective Tibetan speech enhancement model integrating noise classification task and noise suppression task;

[0045] S5: Use the training set and the test set to train and test the built speech enhancement model to obtain a trained speech enhancement model;

[0046] S6: Input the actually collected noisy speech into the trained speech enhancement model to obtain the enhanced logarithmic power spectrum.

[0047] Furthermore, the preprocessing in S2 uses short-time Fourier transform.

[0048] Furthermore, the noise spectrum characteristic calculation formula in S3 is as follows:

[0049]

[0050] E(k,l)=α·E(k,l-1)+(1-α)·N(k,l)

[0051] else

[0052] E(k,l)=E(k,l-1)

[0053] Among them, N(k,l) represents the logarithmic power spectrum of noisy speech, E(k,l) represents the estimated logarithmic power spectrum, l is the index of the time frame in the spectrum, and k is the index of the frequency unit in the spectrum. After experimental testing, when α is set to 0.9 and β is set to 2.5, the noise spectrum estimation can more accurately reflect the actual situation of the noise. Through the above process, the noise classification module can identify the proportion of various noise components in the noisy speech frame input in the current environment, and then guide the noise suppression module to adjust the importance of different noise reduction branches. It can cleverly integrate the prior environmental information into the training and reasoning of the noise suppression module in the form of the possibility of noise existence.

[0054] Furthermore, the speech enhancement model in S4 includes a noise suppression module and a noise classification module;

[0055] The noise classification module classifies and identifies the noise of the input model, and passes the output of the noise classification module to the noise suppression module as an attention gating factor, guiding the noise suppression module to obtain the maximum noise suppression performance benefit.

[0056] Furthermore, the noise classification module includes three layers of fully connected neural network layers. The number of neurons in the output layer is the number of noise classifications. The Softmax function is used to convert the output of the neurons into a probability distribution. Each hidden layer consists of 1024 neurons using the rectified linear unit (ReLU) activation function.

[0057] Furthermore, in order to improve the classification accuracy of the noise classification module, the noise logarithmic power spectrum of each frame input can be estimated and combined with the original logarithmic power spectrum features as the input features of the model. Since the estimation of the noise spectrum is correlated with the posterior signal-to-noise ratio (Posteriori SNR) level, the objective function in the noise classification module uses the cross entropy function, which is expressed as follows:

[0058]

[0059] Where m represents the amount of noise, L i Represents the label of the i-th noise.

[0060] Furthermore, the noise suppression module contains m+1 branch networks, the first m branches perform noise reduction for different noise types, and the last branch is responsible for processing all types of noise. This structure not only retains the fine processing strategy for specific noise, but also makes full use of the complementary advantages between different noise types, improving the model's ability to process unknown noise types.

[0061] Furthermore, during the training process of the noise suppression module, the mean square error (MSE) is used as the error metric, and the expression is as follows:

[0062]

[0063] in, represents the logarithmic power spectrum characteristics of the enhanced speech, and Y(k,l) represents the logarithmic power spectrum characteristics of the clean speech.

[0064] During the training of the noise suppression module, the existence probability value of each type of noise obtained by the noise classification module is multiplied by the neurons of the first m specific branches. This approach can regulate the flow of gradients during back propagation, so that the model can be trained to specifically handle the intermediate layer branches of specific noise types. The m+1th public branch does not perform a point multiplication operation with the existence probability value of each type of noise, and its gradient propagation process remains independent. Any back propagation can effectively update the weight of the public branch, making the noise suppression module "compatible" with various common noises.

[0065] As the model training progresses, each specific branch will gradually master the denoising techniques for specific noise types, and will be able to handle the noise types they focus on more and more effectively. At the same time, the public branch adds additional abnormal noise processing capabilities to the model, enhancing the model's compatibility with multiple noise types.

[0066] Speech signals are time-series information, and the context of speech is very helpful for speech analysis, recognition, and recovery. In order to ensure that the noise suppression module can model the timing of speech signals, the middle layer of each branch network in the noise suppression module consists of m+1 LSTM layers. Through model training, LSTM will accumulate and store context information related to the input speech in its memory unit, which helps the module to enhance the information of the current frame more accurately and effectively.

[0067] In one specific embodiment, see Figure 2 As shown:

[0068] The specific steps include:

[0069] (1) Constructing a training dataset

[0070] The TIMIT public speech dataset and the Tibetan Amdo dialect dataset were selected as clean speech materials, and these speech materials were uniformly resampled to 16KHz. In order to simulate the real noise environment, 10 different noise types were carefully selected from the two noise libraries, NOISEX-92 and Non-Speech Sounds, for subsequent speech mixing processing.

[0071] The production process of the English dataset is as follows: At five different signal-to-noise ratio levels, namely 15dB, 10dB, 5dB, 0dB and -5dB, all 4620 clean speech in the TIMIT training set are processed with noise mixing using n60, n75, babble, destroyerops, factory1, and white noise to generate the English noisy training dataset. At the same signal-to-noise ratio level, 800 clean speech randomly selected from the TIMIT test set are processed with noise mixing using four noises used in the training set, n60, babble, destroyerops, and m109, and four noises not used in the training set, f16, buccaneer1, leopard, and volvo, to obtain the English matched noise test dataset and unmatched noise test dataset.

[0072] The process of making the Tibetan dataset is as follows: at five different signal-to-noise ratio levels, namely 15dB, 10dB, 5dB, 0dB and -5dB, 3500 clean speech sounds randomly selected from the Tibetan Amdo dialect dataset were processed with noise mixing using n60, n75, babble, destroyerops, factory1, and white noise to obtain the noisy training dataset of the Tibetan Amdo dialect. At the same signal-to-noise ratio level, the remaining 800 clean speech sounds in the Tibetan Amdo dialect data were processed with noise mixing using four noises used in the training set, n60, babble, destroyerops, and m109, and four noises not used in the training set, buccaneer1, f16, leopard, and volvo, to obtain the matched noise test dataset and the unmatched noise test dataset of the Tibetan Amdo dialect.

[0073] (2) Evaluation of classification performance

[0074] Reference Figure 3 , which is a schematic diagram of the structure of the speech enhancement method of an embodiment of the present invention, is composed of two parts: a noise classification module and a noise suppression module. The noise classification module plays an important role in both the back propagation and forward propagation processes. The accuracy of noise classification is directly related to the suppression efficiency of the noise suppression module for a specific noise type. Therefore, the classification performance of the noise classification module is first evaluated. The evaluation results are shown in Table 1:

[0075] Table 1 Evaluation results

[0076]

[0077] The experimental results strongly prove that the noise classification module of the embodiment of the present invention has excellent environmental noise recognition ability. By accurately identifying the noise type, the noise classification module provides the noise suppression module with More accurate and targeted In addition, the estimated The feature fusion of the logarithmic power spectrum of noise calculated and the logarithmic power spectrum of noisy speech has a significant contribution to the performance improvement of the classification model. This feature fusion method not only improves the classification accuracy of the classification model, but also enhances the generalization ability of the model. This means that the noise classification module of the embodiment of the present invention can maintain a high recognition accuracy and stability even in the face of unknown or complex noise environments.

[0078] (3) Evaluation of noise suppression performance

[0079] The evaluation of noise suppression performance is mainly analyzed and measured from the following three aspects: first, the performance contribution of the noise classification module output as the attention gating factor to the noise suppression module is evaluated; second, the performance difference between the noise suppression method guided by environmental noise classification and the current similar methods is evaluated; finally, the noise suppression performance of the enhancement method on the Tibetan dataset is evaluated.

[0080] <1> Performance evaluation of attention gating factors

[0081] The average performance of the two methods with and without attention mechanism between the noise classification module and the noise suppression module on the three objective indicators of PESQ, STOI and SI_SDR are shown in Table 2, where "No-Att" means that no attention mechanism is used between the two modules, and "Att" means that an attention gating mechanism is added between the two modules.

[0082] Table 2 Comparison of attention gating performance evaluation

[0083]

[0084] Experimental results show that the performance of the noise suppression module is successfully improved by introducing the attention gating mechanism between the two modules. The introduction of the attention gating mechanism enables the noise suppression module to dynamically adjust the model parameters for specific noise types more accurately, making the noise suppression process more efficient and targeted.

[0085] <2> Performance comparison of similar methods

[0086] The comparison results of the average noise suppression performance of the speech enhancement method of the embodiment of the present invention and other two classic environmental noise-related algorithms in various specific noise scenarios are shown in Table 3. Among them, MMSE represents the minimum mean square error speech estimator for noise classification, and OMLSA represents the optimal modified logarithmic spectral amplitude speech estimator.

[0087] Table 3 Performance comparison results of the same type of methods

[0088]

[0089] The experimental results show that in the signal-to-noise ratio range of -5 to 15dB, the speech enhancement method proposed in the present invention exhibits superior performance that surpasses the other two environmental noise-related methods. Specifically, compared with the MMSE method and the OMLSA method, the average PESQ (perceptual evaluation of speech quality) scores obtained by the speech enhancement method of the embodiment of the present invention increased by 13.76% and 9.92% respectively, and the average STOI (short-time objective intelligibility) scores obtained increased by 5.73% and 4.17% respectively. These data prove that the speech enhancement method of the embodiment of the present invention not only achieves a significant improvement in speech quality when processing speech signals under different signal-to-noise ratio conditions, but also shows obvious advantages in speech intelligibility.

[0090] <3> Performance evaluation on Tibetan dataset

[0091] The quality and intelligibility of the enhanced speech largely depend on the effective recovery of each sound unit (such as phoneme, syllable, etc.) in the speech signal. Considering the unique phoneme distribution characteristics of each language, the speech enhancement method of the embodiment of the present invention is subjected to comparative experiments on the Tibetan Amdo dialect dataset to evaluate its noise suppression performance. The experimental results are shown in Table 4.

[0092] Table 4 Performance evaluation results on the Tibetan dataset

[0093]

[0094] The experimental results show that the speech enhancement method proposed in the present invention also shows better performance than the other two classic environmental noise-related algorithms when applied to the Tibetan language dataset. This strongly proves that the speech enhancement method implemented in the present invention has good cross-language generalization ability. The speech enhancement method implemented in the present invention can effectively adapt to and process the characteristics of different languages, providing strong support for speech noise suppression in a multilingual environment.

[0095] The present invention proposes a single-channel speech enhancement solution tailored for a specific application scenario. Under the premise of knowing the type of noise interference in the application scenario, a noise classification combined with noise suppression dual-branch neural network architecture that introduces prior environmental noise is designed. This architecture cleverly integrates the two tasks of noise classification and noise suppression to achieve efficient multi-objective learning. Experimental data strongly proves that compared with traditional speech enhancement technology that does not utilize prior environmental information, the dual-branch neural network model proposed in the present invention shows significant advantages in noise suppression, highlighting that prior noise information is very effective in improving the model's suppression effect on specific noise. In addition, the language enhancement method disclosed in the present invention deeply explores the complementary characteristics between different types of noise by cleverly integrating the design concept of dedicated paths and shared paths. Not only does it significantly improve the quality and intelligibility of the enhanced speech, but it also successfully enhances the generalization ability of the model, so that it shows better adaptability and robustness when facing a diverse noise environment.

[0096] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0097] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for speech enhancement of the Tibetan Amdo dialect suitable for specific noise and general noise, characterized in that: The following steps are involved: S1: Acquire noisy speech information; S2: preprocessing the noisy speech information to obtain preprocessed speech information, and dividing the preprocessed data into a training set and a test set; S3: Based on the weighted average method of hard threshold in logarithmic power spectrum domain, the noise spectrum characteristics of the noise classification module input are optimized; S4: Construct a multi-objective Tibetan speech enhancement model integrating noise classification task and noise suppression task; S5: Use the training set and the test set to train and test the built speech enhancement model to obtain a trained speech enhancement model; S6: Input the actually collected noisy speech into the trained speech enhancement model to obtain the enhanced logarithmic power spectrum.

2. The method for speech enhancement of Tibetan Amdo dialect suitable for specific noise and general noise according to claim 1, characterized in that: The preprocessing in S2 uses short-time Fourier transform.

3. The method for speech enhancement of Tibetan Amdo dialect suitable for specific noise and general noise according to claim 1, characterized in that: The calculation formula of noise spectrum characteristics in S3 is as follows: E(k,l)=α·E(k,l-1)+(1-α)·N(k,l) else E(k,l)=E(k,l-1) Where N(k,l) represents the logarithmic power spectrum of the noisy speech, E(k,l) represents the estimated logarithmic power spectrum, l is the index of the time frame in the spectrum, and k is the index of the frequency unit in the spectrum.

4. The method for speech enhancement of Tibetan Amdo dialect suitable for specific noise and general noise according to claim 1, characterized in that: The speech enhancement model in S4 includes a noise suppression module and a noise classification module; The noise classification module classifies and identifies the noise of the input model, and transmits the output of the noise classification module as the attention gating factor to the noise suppression module, guiding the noise suppression module to obtain the maximum noise suppression performance benefit; The noise classification module contains three layers of fully connected neural network. The number of neurons in the output layer is the number of noise classifications. The Softmax function is used to convert the output of the neurons into a probability distribution. Each hidden layer consists of 1024 neurons using the rectified linear unit activation function. The objective function in the noise classification module adopts the cross entropy function, which is expressed as follows: Where m represents the amount of noise, L i represents the label of the i-th noise; The noise suppression module contains m+1 branch networks. The first m branches perform noise reduction on different noise types, while the last branch is responsible for processing all types of noise.

5. The method for speech enhancement of Tibetan Amdo dialect suitable for specific noise and general noise according to claim 1, characterized in that: During the training process of the noise suppression module, the mean square error (MSE) is used as the error metric, and the expression is as follows: in, represents the logarithmic power spectrum characteristics of the enhanced speech, and Y(k,l) represents the logarithmic power spectrum characteristics of the clean speech.

Images