A Chinese electronic laryngeal speech conversion device and method based on deep neural network

Through deep neural network technology, the voice signals of electronic laryngeal voice are extracted and reconstructed, and the problem of low naturalness and intelligibility of electronic laryngeal voice is solved, higher quality speech conversion is achieved, mechanical noise interference is reduced, and throatlessness is improved Patients' oral communication.

CN114550701BActive Publication Date: 2024-11-29DUKE KUNSHAN UNIVERSITY
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202210180441.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-25
Publication Date
2024-11-29
Estimated Expiration
2042-02-25

AI Technical Summary

Technical Problem

When existing electronic laryngeal products assist in the speech of laryngeal patients, the naturalness and intelligibility of the speech are low, and the mechanical noise interference is severe, which cannot completely solve the problem of oral communication disorders in laryngeal patients.

Method used

Using the Chinese electronic laryngeal speech conversion method based on deep neural network, the speech signal is extracted and reconstructed through the series connection of the speech recognition module, the bottleneck feature mapping module, the acoustic feature reconstruction module and the vocoder module, and the speech signal is extracted and reconstructed to improve the naturalness and ability of the electronic laryngeal speech. Understand the degree.

Benefits of technology

It significantly improves the naturalness and intelligibility of electronic laryngeal voice. Through objective and subjective evaluation verification, the data modeling ability is stronger, the voice quality is enhanced, and mechanical noise interference is reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114550701B_ABST
    Figure CN114550701B_ABST
Patent Text Reader

Abstract

The present invention discloses a Chinese electronic laryngeal speech conversion device and method based on a deep neural network, the device comprising: a speech recognition module, used to extract a linguistic feature sequence derived from the electronic laryngeal speech as the bottleneck layer feature of the electronic laryngeal speech; a bottleneck feature mapping module, used to estimate the corresponding normal speech bottleneck layer feature based on the electronic laryngeal speech bottleneck layer feature extracted from the electronic laryngeal speech, and restore the tone and other linguistic information lacking in the electronic laryngeal speech; an acoustic feature reconstruction module, used to synthesize the estimated normal bottleneck layer feature into an acoustic feature sequence corresponding to the specific speech; a vocoder module, using the Mel spectrum of the speech as the acoustic feature, for extracting the acoustic feature, and reconstructing the speech signal from the acoustic feature. The present invention uses speech conversion technology to improve or enhance the intelligibility and naturalness of the electronic laryngeal speech from the perspective of software post-processing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of intelligent speech processing, and more specifically, to a Chinese electronic laryngeal speech conversion device and method based on a deep neural network. Background Art

[0002] Speech ability is one of the foundations for people to participate in social activities, but currently many patients have become laryngealless due to laryngeal diseases such as laryngeal cancer after undergoing total laryngectomy. Such patients have lost their ability to speak autonomously due to the loss of all vocal cords, and need to rely on electronic laryngeal (EL) products to replace the vibration of the vocal cords to speak. When the patient tries to speak, the electronic laryngeal device needs to be placed near the Adam's apple of the mandible, and the mechanical vibration emitted by the device cooperates with the patient's autonomous oral movements to produce sound.

[0003] However, existing electronic laryngeal products still have two obvious shortcomings: first, unlike the naturally changing vocal cord vibration mode during healthy speech, the vibration produced by the electronic laryngeal device is single, so that the speech produced with its assistance lacks natural tone information. This causes the intelligibility and naturalness of electronic laryngeal speech to be seriously lower than normal speech; second, electronic laryngeal products rely on mechanical vibration to work, and part of the energy in this process is directly dissipated in the form of noise without being regulated by the human vocal tract. The interference of this part of mechanical noise further affects the naturalness of the electronic laryngeal speech. In summary, due to the voice quality that is significantly lower than generally expected, the existing electronic laryngeal products cannot completely solve the problem of oral communication barriers in patients without a larynx.

[0004] Speech conversion technology is a type of intelligent speech processing technology that aims to change a certain aspect of speech. In the past, studies have demonstrated the feasibility of modeling the "electronic larynx-normal speech" pairing through speech conversion technology to achieve enhanced electronic larynx speech with improved intelligibility and naturalness.

[0005] In the early stage, this type of technology considers a set of speech features obtained by decomposition through digital signal processing (DSP) technology, such as fundamental frequency (F0), spectral envelope (Spectral Envelope), aperiodicity (AP) and its derived features analysis and synthesis; then uses traditional statistical machine learning methods including Gaussian Mixture Model (GMM) and non-negative matrix factorization (NMF) to model the feature mapping relationship between electronic laryngeal and normal speech; finally, uses STRAIGHT, WORLD and other signal processing-based vocoders (Vocoder) to synthesize enhanced speech. In response to the problem of missing fundamental frequency components and lack of clear and voiced opposition in electronic laryngeal speech, some scholars have proposed to map the fundamental frequency of enhanced speech by introducing the phonemic posterior probability (PPP) feature derived from the speech recognition model. However, with the development of deep learning technology in the field of intelligent speech processing, the modeling methods based on traditional statistical machine learning and the vocoders designed based on signal processing theory have revealed their shortcomings in terms of modeling capabilities and speech quality. Summary of the invention

[0006] In view of the above technical problems, the present invention is to overcome at least one deficiency or defect of the above-mentioned existing electronic laryngeal products in terms of sound effect and the processing quality of electronic laryngeal speech enhancement based on traditional statistical learning methods; the purpose of the present invention is to provide a Chinese electronic laryngeal speech conversion method based on deep learning technology, using speech conversion technology to improve or enhance the intelligibility and naturalness of electronic laryngeal speech from the perspective of software post-processing.

[0007] The technical solution of the present invention is as follows:

[0008] According to a specific embodiment of the present invention, the present invention proposes a Chinese electronic laryngeal speech conversion device based on a deep neural network, comprising the following submodules:

[0009] A speech recognition module, used for extracting a linguistic feature sequence derived from the electronic laryngeal speech as a bottleneck layer feature;

[0010] A bottleneck feature mapping module, for estimating the corresponding normal speech bottleneck layer features based on the bottleneck layer features extracted from the electronic laryngeal speech, and predicting the tone and / or other linguistic information lacking in the electronic laryngeal speech;

[0011] An acoustic feature reconstruction module, used to synthesize the estimated normal bottleneck layer features into an acoustic feature sequence corresponding to the specific speech;

[0012] The vocoder module uses the Mel spectrum of speech as acoustic features for extracting acoustic features and reconstructing speech signals from the acoustic features.

[0013] In the above technical solution, the speech recognition module, the bottleneck feature mapping module, the acoustic feature reconstruction module and the vocoder module are independently optimized using the data they need, and form a series connection form of "electronic laryngeal speech-electronic laryngeal bottleneck layer feature-normal bottleneck layer feature-acoustic feature-enhanced speech" during the test phase, thereby completing the conversion from electronic laryngeal speech to enhanced speech with higher intelligibility and naturalness.

[0014] In the above technical solution, the speech recognition module is structurally divided into two parts: an acoustic model and a decoder, wherein the acoustic model adopts a multi-layer neural network to accept speech signal input and generate linguistic features of the speech signal; the decoder outputs a speech recognition result after receiving the linguistic feature input, wherein the linguistic feature is a supervised phoneme N-gram posterior probability or an implicit representation of a neural network without a clear structure, and the bottleneck layer feature refers to the activation information of the forward propagation of the output layer or the adjacent layer of the acoustic model.

[0015] In the above technical solution, the vocoder module includes an acoustic feature extraction submodule and a speech signal reconstruction submodule. The acoustic feature extraction submodule uses an extraction algorithm of signal pre-emphasis, short-time Fourier transform, Mel filtering, and numerical statistical processing to extract acoustic features, and the speech signal reconstruction submodule uses a neural network vocoder designed based on deep learning technology to reconstruct the speech signal.

[0016] In the above technical solution, the acoustic feature reconstruction module uses a multi-speaker speech conversion model to model multiple reconstructed timbres to generate multiple timbres.

[0017] In the above technical solution, the bottleneck feature mapping module adopts a two-stage training mode of pre-training-fine-tuning. In the pre-training stage, a software simulation method is used to convert a large-scale normal speech corpus into a simulated electronic laryngeal speech with a constant excitation signal, and the simulated electronic laryngeal-normal parallel corpus is used for pre-training; in the fine-tuning stage, the real parallel corpus obtained through recording is used for training and fine-tuning.

[0018] According to another specific embodiment of the present invention, the present invention provides a Chinese electronic laryngeal speech conversion method based on a deep neural network, which is implemented based on the above-mentioned Chinese electronic laryngeal speech conversion device and includes the following steps:

[0019] Step S1: Data collection is performed by collecting electronic corpora in a targeted manner or using existing corpora on the Internet;

[0020] Step S2: Using large-scale speech recognition corpus to train the speech recognition module;

[0021] Step S3: jointly train the bottleneck layer mapping module and the acoustic feature reconstruction module using the electronic larynx-normal speech parallel corpus;

[0022] Step S4: using the Vermeer spectrum as an acoustic feature and using the generative adversarial network technology to train the vocoder module;

[0023] Step S5: testing the speech recognition module, the bottleneck layer feature mapping module, the acoustic feature reconstruction module and the vocoder module by connecting their input and output data in series, thereby completing the conversion of the electronic laryngeal speech to the enhanced speech;

[0024] Step S6: Evaluate and test the test results.

[0025] In the above technical solution, step S2 includes:

[0026] Step S201: prepare data in the existing corpus and complete data preprocessing;

[0027] Step S202: training monophone and triphone duration models, and forcibly aligning speech and text pairs of training data;

[0028] Step S203: Use the triphone alignment information to train the acoustic model.

[0029] In the above technical solution, step S3 includes:

[0030] Step S301: Pairing the electronic laryngeal parallel corpus as training data, aligning it using dynamic time warping technology, and extracting bottleneck layer features and acoustic features respectively;

[0031] Step S302: using the aligned and parallel electronic laryngeal speech bottleneck layer feature-normal speech bottleneck layer feature pairing, and using a supervised learning method of regression to train the neural network model of the bottleneck layer mapping module until convergence;

[0032] Step S303: using the aligned and parallel normal speech bottleneck layer feature-normal speech acoustic feature pairing, and using regression and supervised learning methods to train the neural network model of the acoustic feature reconstruction module until convergence.

[0033] In the above technical solution, step S4 is as follows:

[0034] Step S401: prepare corpus data and complete data preprocessing;

[0035] Step S402: Generative adversarial learning technology is used to alternately optimize the generator and the discriminator simultaneously, wherein the optimization goal of the generator is to make the classification result of the discriminator wrong, and the optimization goal of the discriminator is to correctly distinguish between normal speech and speech reconstructed by the generator.

[0036] In the above technical solution, step S5 includes the following steps:

[0037] Step S501: Collect electronic laryngeal speech to obtain a test input speech sample;

[0038] Step S502: Calculate the MFCC features based on the input speech sample, input the features into the acoustic model neural network of the speech recognition module, and take the bottleneck layer activation information as the bottleneck layer features of the electronic laryngeal speech;

[0039] Step S503: input the bottleneck feature of the electronic laryngeal speech into the bottleneck feature mapping module, and obtain the estimated bottleneck feature of the normal speech through forward propagation;

[0040] Step S504: inputting the estimated bottleneck feature of normal speech into the acoustic feature reconstruction model, and obtaining the Mel-spectrogram feature corresponding to the enhanced speech through forward propagation;

[0041] Step S505: input the reconstructed Mel-spectrogram features into the generator neural network of the MelGAN vocoder module, and obtain the electronic laryngeal enhanced speech signal through forward propagation.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] The present invention is based on deep neural network technology, which has stronger data modeling capabilities and better enhancement effects on electronic laryngeal speech. This improvement can be demonstrated by objective and subjective evaluation experimental comparisons. In addition, through appropriate module segmentation, only the bottleneck feature mapping module of the present invention must rely on the difficult-to-obtain electronic laryngeal-normal speech parallel corpus for training, while separating the representation learning of linguistic features, the timbre and sound quality of the reconstructed speech, and the enhancement of linguistic features; allowing the corresponding modules to be trained or pre-trained on large-scale applicable data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1It is a schematic diagram of a Chinese electronic laryngeal speech conversion device according to a specific embodiment of the present invention.

[0046] Figure 2 is a schematic diagram of the speech recognition module of a specific embodiment of the present invention divided into training and testing stages;

[0047] Figure 3 is a schematic diagram of the bottleneck feature mapping module of a specific embodiment of the present invention divided into training and testing stages;

[0048] Figure 4 is a schematic diagram of the acoustic feature reconstruction module of a specific embodiment of the present invention divided into training and testing stages;

[0049] Figure 5 It is a schematic diagram of the vocoder module of a specific embodiment of the present invention divided into training and testing stages. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.

[0051] according to Figure 1 As shown, the present invention provides a Chinese electronic laryngeal speech conversion device based on a deep neural network, and the Chinese electronic laryngeal speech conversion device includes:

[0052] The speech recognition module is used to extract the linguistic feature sequence derived from the electronic laryngeal speech as the bottleneck feature (Bottleneck Feature, BNF);

[0053] A bottleneck feature mapping module, for estimating the corresponding normal speech bottleneck layer features based on the bottleneck layer features extracted from the electronic laryngeal speech, and predicting the tone and / or other linguistic information lacking in the electronic laryngeal speech;

[0054] An acoustic feature reconstruction module, used to synthesize the estimated normal bottleneck layer features into an acoustic feature sequence corresponding to the specific speech;

[0055] The vocoder module uses the Mel spectrum of speech as acoustic features for extracting acoustic features and reconstructing speech signals from the acoustic features.

[0056] The above four modules can be independently optimized using the data they need, and form a series connection of "electronic laryngeal speech-electronic laryngeal bottleneck layer characteristics-normal bottleneck layer characteristics-acoustic characteristics-enhanced speech" during the testing phase, thereby completing the conversion from electronic laryngeal speech to enhanced speech with higher intelligibility and naturalness.

[0057] In the present invention, the speech recognition module can be structurally divided into two parts: an acoustic model and a decoder. The acoustic model uses a multi-layer neural network to accept speech signal input and generate linguistic features of the speech signal; the decoder outputs the speech recognition result after receiving the linguistic feature input. The above-mentioned linguistic features are supervised phoneme N-gram posterior probabilities (Phonemic Posterior-probability / Posterior gram, PPP / PPG) or hidden representations of neural networks without a clear structure in the jointly optimized acoustic model and decoder. Depending on the specific speech recognition system, the above two submodules can be jointly optimized or trained separately. The bottleneck layer features described in the present invention refer to the activation information of the output layer or the adjacent layer of the acoustic model of the speech recognition system during forward propagation, including but not limited to the specially designed bottleneck layer activation information located before the output layer.

[0058] The bottleneck feature mapping module and the acoustic feature reconstruction module adopt a feed-forward-transformer (FFT) stack based on a self-attention mechanism and a residual connection as the model design structure of the neural network. In the training process of its supervised learning, the complete speech segment is used as the basic unit for calculating its regression loss function. The acoustic feature reconstruction module uses a multi-speaker model to model multiple reconstructed timbres to produce multiple timbres.

[0059] The vocoder module uses the Mel spectrum of speech as the acoustic feature, so the vocoder module includes an acoustic feature extraction submodule and a speech signal reconstruction submodule. The acoustic feature extraction submodule uses signal pre-emphasis (Preemphasis), short-time Fourier transform (STFT), Mel-frequency filtering, and numerical statistical processing extraction algorithms to extract acoustic features, and the speech signal reconstruction submodule uses a neural network vocoder designed based on deep learning technology to reconstruct the speech signal. Typical designs include MelGAN, HiFi-GAN, WaveRNN, etc., and other structures that achieve the same function can also be used instead.

[0060] In the specific implementation of the present invention, the above-mentioned speech recognition module, bottleneck feature mapping module, acoustic feature reconstruction module and vocoder module are optimized separately using data sets suitable for the requirements of each module during the separate training process. First, the training of the speech recognition module is carried out using large-scale, multi-speaker normal speech-transcribed text pairing data; second, the acoustic feature reconstruction and vocoder modules are trained using high-quality speech corpus, which is conducive to improving the sound quality and naturalness of the reconstructed speech; third, the bottleneck feature mapping module adopts a two-stage training mode of pre-training-fine-tuning; in the pre-training stage, the "simulated electronic larynx-normal speech" parallel corpus obtained by the method of simulating the WORLD vocoder software is used for training; only in the fine-tuning stage, the paired "electronic larynx-normal speech" parallel corpus is necessary for training. This design compresses the requirements of the entire scheme for the quantity and quality of electronic larynx parallel corpus that is not easy to obtain. On this basis, the bottleneck layer mapping module and the acoustic feature reconstruction module are jointly trained using electronic larynx-normal speech parallel corpus, which can improve the effect of electronic larynx speech enhancement of the overall system.

[0061] In a specific embodiment of the present invention, the bottleneck feature mapping module adopts a two-stage training mode of pre-training-fine-tuning. In the pre-training stage, a software simulation method based on the WORLD vocoder is used to convert a large-scale normal speech corpus into a simulated electronic laryngeal speech with a constant fundamental frequency, and the simulated electronic laryngeal-normal parallel corpus is used for pre-training; in the fine-tuning stage, the real parallel corpus obtained by recording is used for training and fine-tuning.

[0062] The software simulation method of electronic laryngeal speech based on WORLD vocoder mainly includes the following steps:

[0063] Step 1: Use the feature extraction module in the WORLD vocoder to decompose normal speech into three items: fundamental frequency, spectral features (SP), and non-periodic features;

[0064] Step 2: Given a constant F el is the typical fundamental frequency of electronic laryngeal speech, and the fundamental frequency of normal speech is set to be constant at F el ;

[0065] Step 3: Use the synthesis module of the WORLD vocoder to convert the spectral features, non-periodic features and constant F of the original audio el The fundamental frequency is synthesized to simulate electronic laryngeal speech.

[0066] More specifically, this embodiment Figure 2As shown, the overall scheme can be divided into two stages: training stage and testing stage. Among them, the completion of training is the prerequisite for testing. The training stage includes the training of the neural network model in the four main modules (including pre-training and fine-tuning stages) and the joint training of the series modules. The testing stage is the practical working stage of the present invention. In this stage, by connecting the input and output interfaces of the above four main modules in series, the electronic laryngeal speech is converted into enhanced speech with significantly improved intelligibility and naturalness.

[0067] Specifically, the present invention provides a method for converting Chinese electronic laryngeal speech based on a deep neural network, comprising the following steps:

[0068] Step S1: Data collection is performed by collecting electronic corpora in a targeted manner or using existing corpora on the Internet;

[0069] For step S1, different data sets are collected for training according to the structure and function requirements of each module neural network. The electronic throat parallel corpus is obtained through targeted collection, and the rest of the corpus uses Internet public data.

[0070] For the electronic laryngeal parallel corpus data, the source speaker and the target speaker are the same young female volunteer. The volunteer received training on the use of the electronic laryngeal before recording the electronic laryngeal voice. The electronic laryngeal device was used to assist the voice without vibrating the vocal cords to simulate the scenario of laryngeal patients relying on the electronic laryngeal device to speak. The electronic laryngeal device used is a product of Huzhou Tianchou Medical Equipment Co., Ltd., with a recording sampling rate of 16kHz. A parallel corpus of about 5 hours in length was collected, and test data was reserved.

[0071] In order to meet the requirements of the speech recognition module for data volume and coverage, the AISHELL-2 corpus containing 1,000 hours of speech and transcribed text was used for training. In order to meet the requirements of the timbre and sound quality of the vocoder module, the data of 185 people in the AISHELL-3 corpus was used for training. Both of the above corpora are public data.

[0072] like Figure 2 As shown, step S2: using a large-scale speech recognition corpus to train the speech recognition module.

[0073] The specific implementation of the speech recognition module is based on the Librispeech speech recognition template in the Kaldi Toolkit. Its acoustic model uses Mel-Filterbank Cepstrum Coefficient (MFCC) as a feature. The neural network consists of three parts: a 17-layer time-delay neural network, a 256-dimensional information bottleneck layer, and a triphone posterior probability (i.e., 3-gram PPG) output layer. In this embodiment, the activation information of the neural network in the information bottleneck layer is used as the specific bottleneck layer feature.

[0074] Step S2 includes:

[0075] Step S201: prepare data from an existing corpus and complete data preprocessing; the corpus is AISHELL-2.

[0076] Step S202: training monophone and triphone duration models, and forcibly aligning speech and text pairs of training data;

[0077] Step S203: Use the triphone alignment information to train the acoustic model.

[0078] like Figure 3 and Figure 4 As shown, step S3: use the electronic laryngeal-normal speech parallel corpus to jointly train the bottleneck layer mapping module and the acoustic feature reconstruction module.

[0079] The implementation of the bottleneck feature mapping module and the acoustic feature reconstruction module neural network in this embodiment includes a fully connected layer preprocessing network, an 8-layer forward Transformer stack, a fully connected output layer, and a post-processing layer consisting of a 5-layer convolutional neural network. A 256-dimensional modeling width is used.

[0080] Step S3 includes:

[0081] Step S301: Pair the electronic laryngeal parallel corpus as training data, align them using the Dynamic Time Warping (DTW) technology, and extract bottleneck layer features and acoustic features respectively;

[0082] Step S302: using the aligned and parallel electronic laryngeal speech bottleneck layer feature-normal speech bottleneck layer feature pairing, and using a supervised learning method of regression to train the neural network model of the bottleneck layer mapping module until convergence;

[0083] Step S303: using the aligned and parallel normal speech bottleneck layer feature-normal speech acoustic feature pairing, and using regression and supervised learning methods to train the neural network model of the acoustic feature reconstruction module until convergence.

[0084] like Figure 5 It is a schematic diagram of the vocoder module of a specific embodiment of the present invention divided into training and testing stages. Step S4 of the present invention: using the Vermeer spectrum as an acoustic feature and using the generative adversarial network technology to train the vocoder module;

[0085] The vocoder module uses the Vermeer spectrum as a specific acoustic feature. The corresponding speech signal reconstruction neural network uses MelGAN as an implementation scheme. The model consists of two parts: the generator and the discriminator, and is trained using the Generative Adversarial Network (GAN) technology. Only the generator participates in its testing phase. The generator consists of a 4-layer transposed convolutional neural network (Transposed Convolution) upsampling layer and a residual connection block (Residual Stack); and the multi-scale discriminator group consists of three convolutional neural networks working at full sampling rate, 1 / 4 sampling rate and 1 / 8 sampling rate respectively.

[0086] Step S4 is as follows:

[0087] Step S401: prepare corpus data and complete data preprocessing; specifically, the corpus is AISHELL-3.

[0088] Step S402: Generative adversarial learning technology is used to alternately optimize the generator and the discriminator simultaneously, wherein the optimization goal of the generator is to make the classification result of the discriminator wrong, and the optimization goal of the discriminator is to correctly distinguish between normal speech and speech reconstructed by the generator.

[0089] During the testing phase, only the generator in the generative adversarial network is used to convert acoustic features into higher-quality speech signals.

[0090] Step S5 of the present invention: Step S5: The speech recognition module, the bottleneck layer feature mapping module, the acoustic feature reconstruction module and the vocoder module are tested by serially connecting their input and output data to complete the conversion of electronic laryngeal speech to enhanced speech.

[0091] Step S5 includes the following steps:

[0092] Step S501: Collect electronic laryngeal speech to obtain a test input speech sample;

[0093] Step S502: Calculate the MFCC features based on the input speech sample, input the features into the acoustic model neural network of the speech recognition module, and take the bottleneck layer activation information as the bottleneck layer features of the electronic laryngeal speech;

[0094] Step S503: input the bottleneck feature of the electronic laryngeal speech into the bottleneck feature mapping module, and obtain the estimated bottleneck feature of the normal speech through forward propagation;

[0095] Step S504: inputting the estimated bottleneck feature of normal speech into the acoustic feature reconstruction model, and obtaining the Mel-spectrogram feature corresponding to the enhanced speech through forward propagation;

[0096] Step S505: input the reconstructed Mel-spectrogram features into the generator neural network of the MelGAN vocoder module, and obtain the electronic laryngeal enhanced speech signal through forward propagation.

[0097] Step S6: Evaluate and test the test results.

[0098] In the test result evaluation experiment, the evaluation scale includes the naturalness and intelligibility of the enhanced speech, and the experiment includes two parts: objective evaluation and subjective evaluation. In order to compare with the effect of the conversion system described in this embodiment, unconverted electronic laryngeal speech and converted speech obtained by statistical machine learning methods such as Gaussian Mixture Model (GMM) are introduced for evaluation.

[0099] Objective evaluation: In this experiment, the objective evaluation uses Mel-Cepstral Distortion (MCD) as the similarity between the audio enhanced by the conversion system and the ideal target audio, thereby measuring the naturalness of the generated speech. This parameter is calculated using the following formula:

[0100]

[0101] in and They are the d-th order target speech cepstrum parameter and the d-th order enhanced speech cepstrum parameter. The larger the MCD value, the more serious the distortion between the two audio segments and the lower the similarity between the two. On the contrary, the greater the audio similarity, the unit is decibel (dB).

[0102] The objective evaluation uses a speech recognition system to recognize the text content of the converted speech, and evaluates the intelligibility of the converted speech by calculating the character error rate (CER) between the recognition result and the actual text content.

[0103] Subjective evaluation:

[0104] In the subjective evaluation, 15 volunteers were invited to listen to a series of recorded or converted voices and rate the naturalness, intelligibility and similarity to the reference normal voice based on their subjective judgment. The scale is: 5-very good, 4-good, 3-average, 2-poor, 1-very poor. Four types of voice samples participated in this subjective scoring experiment: 1) source electronic laryngeal voice;

[0105] 2) Enhanced speech based on statistical machine learning methods; 3) Enhanced speech based on the method described in the present invention; 4) Normal speech of the target speaker (reference speech).

[0106] Experimental results:

[0107] Objective evaluation results:

[0108] The MCDs calculated for the original electronic laryngeal speech, the speech enhanced by the GMM method, the speech enhanced by the system described in this embodiment, and the target normal speech are shown in Table 1. From Table 1, it can be seen that the system of this embodiment can reduce the distortion rate of the electronic laryngeal speech and the target speech by 5.915 dB. It is generally believed that MCD is correlated with human auditory perception, which can prove that the naturalness of the electronic laryngeal speech by the system of this embodiment is significantly improved compared with the GMM system.

[0109] Table 1

[0110]

[0111] The character error rate calculation results of the original electronic laryngeal speech, the speech enhanced by the GMM system, the speech enhanced by the system described in this embodiment, the speech enhanced by the system described in this embodiment without the bottleneck feature mapping module, and the normal speech are shown in Table 2. The trend of the character error rate in the table is roughly the same as that in Table 1. It is easy to see that the system described in this embodiment can significantly improve (+11.95%) the chance of the electronic laryngeal speech being correctly recognized by the normal speech recognition system, that is, it significantly improves the intelligibility of the speech. In addition, by comparing the experimental results of the system of the present invention with and without the bottleneck feature mapping module, it can be explained that the design of the bottleneck layer feature mapping and acoustic feature reconstruction splitting proposed by the present invention can effectively improve the intelligibility of the enhanced speech.

[0112] Table 2

[0113]

[0114] Subjective evaluation results:

[0115] Table 3 shows the subjective scoring results (mean values) of volunteers for electronic laryngeal speech, GMM system, the system and normal speech samples. It is easy to see that in the subjective evaluation, the system described in this embodiment has achieved significant improvements in naturalness, intelligibility and similarity compared to the unprocessed electronic laryngeal speech and the results enhanced by the traditional GMM system.

[0116] Table 3

[0117]

[0118] It should be noted that the same or similar reference numerals correspond to the same or similar components.

[0119] The description of the positional relationship in the accompanying drawings is only for illustrative purposes and cannot be understood as a limitation of this patent; the above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above-mentioned specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be arbitrarily combined with each other.

Claims

1. A Chinese electronic laryngeal speech conversion device based on deep neural network, characterized in that: Includes the following submodules: A speech recognition module is used to extract a linguistic feature sequence derived from the electronic laryngeal speech as a bottleneck layer feature; A bottleneck feature mapping module is used to estimate the corresponding normal speech bottleneck layer features based on the bottleneck layer features extracted from the electronic glottal speech, and predict the tones and / or other linguistic information lacking in the electronic glottal speech; An acoustic feature reconstruction module, used to synthesize the estimated normal bottleneck layer features into an acoustic feature sequence corresponding to the specific speech; The vocoder module uses the Mel spectrum of speech as acoustic features to extract acoustic features and reconstruct the speech signal from the acoustic features.

2. The Chinese electronic laryngeal speech conversion device according to claim 1, characterized in that: The speech recognition module, the bottleneck feature mapping module, the acoustic feature reconstruction module, and the vocoder module are independently optimized using the data they need. During the testing phase, a series of "electronic laryngeal speech-electronic laryngeal bottleneck layer features-normal bottleneck layer features-acoustic features-enhanced speech" is formed, thereby completing the conversion from electronic laryngeal speech to enhanced speech.

3. The Chinese electronic laryngeal speech conversion device according to claim 1, characterized in that: The speech recognition module is structurally divided into two parts: an acoustic model and a decoder. The acoustic model uses a multi-layer neural network to accept speech signal input and generate linguistic features of the speech signal. The decoder outputs a speech recognition result after receiving the linguistic feature input. The linguistic feature is a supervised phoneme N-gram posterior probability or an implicit representation of a neural network without a clear structure. The bottleneck layer feature refers to the activation information of the forward propagation of the output layer or a similar layer of the acoustic model.

4. The Chinese electronic laryngeal speech conversion device according to claim 1, characterized in that: The vocoder module includes an acoustic feature extraction submodule and a speech signal reconstruction submodule. The acoustic feature extraction submodule uses an extraction algorithm of signal pre-emphasis, short-time Fourier transform, Mel filtering, and numerical statistical processing to extract acoustic features. The speech signal reconstruction submodule uses a neural network vocoder designed based on deep learning technology to reconstruct the speech signal.

5. The Chinese electronic laryngeal speech conversion device according to claim 1, characterized in that: The acoustic feature reconstruction module uses a multi-speaker speech conversion model to model multiple reconstructed timbres to generate multiple timbres.

6. The Chinese electronic laryngeal speech conversion device according to claim 1, characterized in that: The bottleneck feature mapping module adopts a two-stage training mode of pre-training and fine-tuning. In the pre-training stage, a software simulation method is used to convert a large amount of normal speech corpus into simulated electronic laryngeal speech with a constant excitation signal, and pre-training is performed using the simulated electronic laryngeal-normal parallel corpus. During the fine-tuning phase, real parallel corpus obtained through recording is used for training and fine-tuning.

7. A method for converting Chinese electronic laryngeal speech based on a deep neural network, implemented based on the Chinese electronic laryngeal speech conversion device according to any one of claims 1 to 6, characterized in that: The steps include: Step S1: Data collection is performed by specifically collecting electronic laryngeal parallel corpora or using existing corpora on the Internet; Step S2: Use large-scale speech recognition corpus to train the speech recognition module; Step S3: Jointly train the bottleneck layer mapping module and the acoustic feature reconstruction module using the electronic larynx-normal speech parallel corpus; Step S4: Using the Vermeer spectrum as the acoustic feature, the vocoder module is trained using the generative adversarial network technology; Step S5: testing the speech recognition module, the bottleneck layer feature mapping module, the acoustic feature reconstruction module, and the vocoder module by serially connecting their input and output data to complete the conversion of the electronic laryngeal speech to the enhanced speech; Step S6: Evaluate and test the test results.

8. The Chinese electronic laryngeal speech conversion method according to claim 7, characterized in that: The step S2 comprises: Step S201: prepare data from the existing corpus and complete data preprocessing; Step S202: training monophone and triphone duration models, and forcibly aligning speech and text pairs of training data; Step S203: Use the triphone alignment information to train the acoustic model.

9. The Chinese electronic laryngeal speech conversion method according to claim 7, characterized in that: The step S3 comprises: Step S301: Pairing the electronic laryngeal parallel corpus as training data, aligning it using dynamic time warping technology, and extracting bottleneck layer features and acoustic features respectively; Step S302: using the aligned and parallel electronic laryngeal speech bottleneck layer feature and normal speech bottleneck layer feature pairing, using a supervised learning method with regression to train the neural network model of the bottleneck layer mapping module until convergence; Step S303: Using the aligned and parallel normal speech bottleneck layer feature-normal speech acoustic feature pairing, a neural network model of the acoustic feature reconstruction module is trained using regression and supervised learning methods until convergence.

10. The Chinese electronic laryngeal speech conversion method according to claim 7, characterized in that: The step S4 is as follows: Step S401: prepare corpus data and complete data preprocessing; Step S402: Generative adversarial learning technology is used to alternately optimize the generator and the discriminator simultaneously, where the optimization goal of the generator is to make the discriminator classification result wrong, and the optimization goal of the discriminator is to correctly distinguish normal speech from the generator-reconstructed speech.

11. The Chinese electronic laryngeal speech conversion method according to claim 7, characterized in that: The step S5 comprises the following steps: Step S501: collecting electronic laryngeal speech to obtain a test input speech sample; Step S502: Calculate the MFCC features based on the input speech sample, input the features into the acoustic model neural network of the speech recognition module, and take the bottleneck layer activation information as the bottleneck layer features of the electronic laryngeal speech; Step S503: Inputting the bottleneck features of the electronic laryngeal speech into the bottleneck feature mapping module, and obtaining the estimated bottleneck features of normal speech through forward propagation; Step S504: Input the estimated bottleneck features of normal speech into the acoustic feature reconstruction model, and obtain the Mel-spectrogram features corresponding to the enhanced speech through forward propagation; Step S505: input the reconstructed mel-spectrogram features into the generator neural network of the vocoder module, and obtain the electronic laryngeal enhanced speech signal through forward propagation.

Citation Information

Patent Citations

  • Computer system for assisting spoken language learning

    CN101551947A

  • Chinese electrolarynx speech conversion method

    CN106448673A

  • An intelligent micro camera array endoscopic imaging system

    CN109068035A

  • Electrolaryngeal speech enhancement for telephony

    WO2001059758A1