Mandarin nasal obstruction voice signal processing method, experimental system and experimental method

Abstract

The invention relates to a mandarin nasal obstruction voice signal processing method and experimental system and method, and the processing method comprises the following steps: S1, collecting a mandarin nasal obstruction voice signal, detecting vowel voice frames in the nasal obstruction voice signal, and carrying out the classification detection of the vowel voice frames, determining whether the vowel voice frame belongs to one of 16 types of nasal vowel voice frames including n and ng, thereby detecting the nasal vowel voice frame, and determining the type of the nasal vowel voice frame, namely determining which type of the 16 types of nasal vowel voice frames the nasal vowel voice frame belongs to; the 16 types of nasal vowel voice frames containing n and ng refer to voice frames of eight front nasal vowels an, ian, uan, uan, en, uen, in and un and voice frames of eight rear nasal vowels ang, iang, uang, eng, ueng, ing, ong and ion; and S2, performing nasal obstruction voice recovery processing on the nasal vowel voice frame, and not performing processing on other voice frames. According to the invention, efficient processing of nasal obstruction voice can be realized.

Description

Technical Field

[0001] The invention relates to a sound signal processing technology, and in particular to a method for processing nasal congestion speech signals in Mandarin.

[0002] The invention also relates to an experimental system and an experimental method, which are used for teaching in multimedia technology, digital signal processing and other courses offered in vocational colleges and universities through speech signal processing experiments. Background Art

[0003] Colds, fevers, and seasonal allergic rhinitis are common diseases. In these cases, people often experience nasal congestion and runny nose, which leads to nasal speech, that is, the voice is muffled and difficult to hear clearly. For teachers, it often affects the effect of students' listening and leads to a decline in teaching quality. Similarly, for other people who rely on language as a job, it is also a problem that needs to be solved urgently, such as announcers, network anchors, etc.

[0004] In the existing technology, "Research on Feature Extraction and Recognition of the Voice of Cold Patients" mainly extracted 16-dimensional features including resonance peaks F1, F2, F3 and 12-dimensional Mel cepstral coefficients, and used trained neural networks to distinguish between cold and non-cold voices. However, it only involves the recognition of cold voices and does not provide a method for restoring normal pronunciation from cold voices. Summary of the invention

[0005] The purpose of the present invention is to restore the nasal blocked speech to normal pronunciation by processing the nasal blocked speech.

[0006] On the other hand, the purpose of the present invention also involves applying the processing method to courses such as multimedia technology and digital signal processing in higher vocational and university teaching, so as to facilitate students to learn, understand and master the principles and practices of multimedia technology and speech signal processing technology.

[0007] To this end, the present invention proposes a method for processing a Mandarin nasal congestion speech signal, comprising the following steps:

[0008] Step S1, collecting a Mandarin nasal congestion speech signal, detecting a vowel speech frame in the nasal congestion speech signal, and classifying and detecting the vowel speech frame to determine whether the vowel speech frame belongs to one of 16 types of nasal vowel speech frames including n and ng, thereby detecting a nasal vowel speech frame and determining the category of the nasal vowel speech frame, that is, determining which category of the 16 types of nasal vowel speech frames the nasal vowel speech frame belongs to;

[0009] The above-mentioned 16 types of nasal finals speech frames including n and ng refer to the speech frames of the 8 front nasal finals an, ian, uan, üan, en, uen, in, ün and the speech frames of the 8 back nasal finals ang, iang, uang, eng, ueng, ing, ong, iong;

[0010] Step S2, performing nasal congestion speech recovery processing on the nasal vowel speech frame, and not performing processing on other speech frames;

[0011] The nasal congestion speech recovery process in step S2 specifically includes the following steps:

[0012] Step S21, adjusting the low-frequency spectrum amplitude spectrum of the nasal vowel speech frame to obtain the low-frequency adjusted nasal vowel speech signal amplitude spectrum;

[0013] Step S22, reconstructing the high-frequency amplitude spectrum of the nasal vowel speech frame according to the amplitude spectrum of the nasal vowel speech signal after the low-frequency adjustment obtained in step S21, and obtaining the amplitude spectrum after high and low frequency restoration;

[0014] Step S23, combining the phase spectrum of the nasal vowel speech frame based on the high and low frequency restored amplitude spectrum obtained in step S22 to obtain a restored spectrum, and obtaining a restored speech frame time domain signal according to the restored spectrum.

[0015] In another technical solution, the step S21 of adjusting the low-frequency spectrum amplitude spectrum of the nasal vowel speech frame to obtain the low-frequency adjusted nasal vowel speech signal amplitude spectrum specifically includes:

[0016] The first low-frequency part of the amplitude spectrum of the nasal vowels n and ng in the nasal speech signal is multiplied by the low-frequency adjustment vector to obtain the adjusted spectrum, and then the rest of the amplitude spectrum of the nasal vowel speech frame is superimposed to obtain the low-frequency adjusted amplitude spectrum.

[0017] In another technical solution, the first low-frequency part in step S21 is specifically to select the frequency range of the low-frequency adjustment to be 0 to 5000 Hz;

[0018] The low-frequency adjustment vector in step S21 is outputted through a trained low-frequency adjustment neural network, and different low-frequency adjustment neural network models are trained for different nasal finals, and 16 low-frequency adjustment neural network models are trained for the nasal finals an, ian, uan, üan, en, uen, in, ün, ang, iang, uang, eng, ueng, ing, ong, iong respectively;

[0019] According to the category of the nasal final speech frame determined in step S1, the corresponding model is determined from 16 neural network models, the amplitude spectrum of the nasal final speech frame is input into the low-frequency adjustment neural network model of the corresponding final category, and a low-frequency adjustment vector is output. The low-frequency adjustment vector is multiplied by the first low-frequency part of the amplitude spectrum of the nasal final speech frame, and then the rest of the amplitude spectrum of the nasal final speech frame is superimposed to obtain the amplitude spectrum after low-frequency adjustment.

[0020] In another technical solution, in step 22, the second low-frequency part of the amplitude spectrum of the nasal final speech signal after low-frequency adjustment obtained in step 21 is multiplied by the high-frequency reconstruction vector to obtain the reconstructed amplitude spectrum, and then according to the fundamental frequency, while maintaining the harmonic structure, the reconstructed amplitude spectrum is shifted to the high-frequency part, and superimposed with the amplitude spectrum of the nasal final speech signal after low-frequency adjustment obtained in step 21, so as to obtain the amplitude spectrum after high and low frequency restoration.

[0021] In another technical solution, the high-frequency reconstruction vector in step S22 is output through a trained high-frequency reconstruction neural network, and different high-frequency reconstruction neural network models are trained for different nasal finals, and 16 high-frequency reconstruction neural network models are trained for the nasal finals an, ian, uan, üan, en, uen, in, ün, ang, iang, uang, eng, ueng, ing, ong, and iong respectively;

[0022] According to the category of the nasal final speech frame determined in step S1, the corresponding model is determined from 16 high-frequency reconstruction neural network models, and the low-frequency adjusted nasal final speech signal amplitude spectrum of the nasal final speech frame output in step S21 is input into the high-frequency reconstruction neural network model of the corresponding final category, and a high-frequency reconstruction vector is output.

[0023] In another technical solution, the step S1 includes:

[0024] S11, using VAD endpoint detection algorithm to identify the starting frame of a sentence;

[0025] S12: After identifying the start frame, determine whether the current frame is a final frame or a non-final frame.

[0026] S12, performing classification detection based on the GMM-HMM model on the vowel speech frame frame by frame to determine whether the vowel speech frame belongs to one of the 16 types of nasal vowel speech frames including n and ng:

[0027] Extract 13-dimensional MFCC features for each final speech frame as the observation vector of the current frame, obtain the multi-frame MFCC feature vectors from the start frame of the sentence to the end of the current frame as the observation vector sequence, substitute the observation vector sequence into the trained GMM-HMM model, and obtain the result of which nasal final the current frame belongs to;

[0028] S13. When it is detected that the current frame is a frame containing nasal finals n or ng, the nasal congestion speech recovery process described in the aforementioned step S2 is performed.

[0029] In another technical solution, after the start frame is detected, when N frames are continuously detected to contain nasal finals of n and ng, the nasal speech recovery process described in step S2 is performed on the current frame and subsequent frames, where N is a positive integer greater than or equal to 3.

[0030] In another technical solution, for the first M nasal final frames after the starting frame in a sentence, M is a positive integer greater than or equal to 5, and the low-frequency adjustment vector in step S21 is multiplied by a coefficient to obtain a new low-frequency adjustment vector for nasal congestion speech processing; the high-frequency reconstruction vector in step S22 is multiplied by a coefficient to obtain a new high-frequency reconstruction vector for nasal congestion speech processing; the coefficient is less than 1 and increases frame by frame for the first M frames.

[0031] An experimental teaching system for realizing the aforementioned nasal congestion speech signal processing method is also proposed, the system comprising: a DSP development board, an audio acquisition circuit, and an audio output circuit;

[0032] The DSP development board includes a processor and an integrated development environment. The DSP development board allows students to download computer programs to the development board for testing and verification. The computer program can implement the aforementioned nasal congestion speech signal processing method. The sound collected by the audio acquisition circuit is processed in real time by the DSP development board, directly amplified and output through the audio output circuit, or input into a PC for storage.

[0033] An experimental teaching method for the experimental teaching system is also proposed, including:

[0034] DSP preparation includes connecting the DSP development board to the computer and configuring the corresponding connection options and parameters, including selecting the correct interface connection method, setting serial port communication parameters, etc.

[0035] Students open the integrated development environment, create a new project, and use the integrated development environment to write a signal processing computer program code, which implements the aforementioned method for processing Mandarin nasal congestion speech signals. Using the debugging function provided by the development environment, the code is executed step by step, variables and memory status are viewed, and performance analysis and optimization are performed. After debugging is completed, the computer program code is compiled and downloaded to the DSP development board for testing and verification. The sound collected by the audio acquisition circuit is processed in real time by the DSP development board, directly amplified and output through the audio output circuit, or input into a PC for storage;

[0036] The DSP development board stores a high-frequency reconstruction vector lookup table and a low-frequency adjustment vector lookup table.

[0037] Technical effect: The contribution of the present invention lies in applying the recognition of nasal finals containing n and ng to the restoration of cold-induced nasal congestion speech. By combining low-frequency restoration and high-frequency reconstruction, better nasal sound restoration is achieved, and real-time processing is realized through frame-by-frame classification recognition and processing. Description of the Drawings

[0038] Figure 1A Is the spectrogram of the normal pronunciation of "si"

[0039] Figure 1B Is the spectrogram of the nasal congestion speech "si"

[0040] Figure 2A Is the spectrogram of the normal pronunciation of "san"

[0041] Figure 2B Is the spectrogram of the nasal congestion speech "san"

[0042] Figure 2C Is the amplitude spectrogram of the normal pronunciation of "san"

[0043] Figure 2D Is the amplitude spectrogram of the nasal congestion speech "san"

[0044] Figure 3 Schematic diagram of the GMM-HMM speech recognition model

[0045] Figure 4 Is the structural diagram of the speech frame in the present invention

[0046] Figure 5 Is the flowchart of the processing method of the first embodiment of the present invention

[0047] Figure 6 Is the flowchart of the processing method of step S2 in the second embodiment of the present invention Detailed Embodiments

[0048]

First Embodiment

[0049] Cold-induced nasal congestion does not affect all types of syllables, but only affects the pronunciation of some types of syllables, that is, those syllables that are pronounced with air passing through the nose.

[0050] Figure 1A Is the spectrogram of the normal pronunciation of "si" in Mandarin, Figure 1B Is the spectrogram of the nasal congestion pronunciation of "si". Judging from the spectrograms of the two, the normal pronunciation of "si" in Mandarin and the nasal congestion pronunciation do not differ much, and there is no obvious distinguishable difference in terms of hearing. This is because the pronunciation of "si" in Mandarin consists of two parts. The first part is the initial consonant "s (si)", and the second part is the final "i (yi)", where:

[0051] The initial consonant "s(hiss)" is a voiceless consonant, which does not involve vocal cord vibration and cavity resonance. The pronunciation process does not require the participation of nasal airflow. Nasal congestion does not have a significant effect on the pronunciation of the initial consonant "s(hiss)".

[0052] The final "i (yi)" is a vowel. A cold and nasal congestion will affect the nasal resonance of the final "i". However, during the pronunciation of the final i, most of the airflow flows out through the mouth and does not rely too much on the nasal cavity. As a result, nasal congestion does not have a significant effect on the pronunciation of the final "i (yi)". Figure 1A , 1B From the spectrum of the final “i(yi)” in the second half, the resonance peak structure of the final “i” has only changed slightly.

[0053] According to the paper "Types of Nasal Initials in Chinese Dialects" (author Li Mingxing) in the Journal of Chinese Phonetics, nasal sounds appear more frequently in the pronunciation of syllables such as m, n, η, etc. in Chinese initials. Combined with the observation of experimental data, it can be seen that colds and nasal congestion have a greater impact on syllables such as m, n, η, especially the initial consonant "m". This is because "m" is a closed-mouth sound, and there is no airflow through the mouth when it is pronounced, and it relies entirely on nasal ventilation to achieve pronunciation. When the nose is blocked, especially when the nasal cavity is completely blocked and can only breathe through the mouth, the air cannot flow out of the nasal cavity, which will make the "m" sound extremely difficult to pronounce or even impossible to pronounce.

[0054] In summary, the influence of a stuffy nose caused by a cold on different syllables is different, some syllables are greatly affected, some are slightly affected or even not affected at all. Based on this phonetic feature, the present invention proposes the following first embodiment.

[0055] Figure 5 This is a flow chart of a processing method according to a first embodiment of the present invention. The method comprises the following steps:

[0056] Step S1, collect Mandarin nasal congestion speech signal, detect the vowel voice frame in the nasal congestion speech signal, and classify and detect the vowel voice frame to identify the nasal vowel voice frame; the nasal vowel voice frame refers to the nasal vowel voice frame containing n and ng, that is, the voice frames of 8 front nasal vowels an, ian, uan, üan, en, uen, in, ün and the voice frames of 8 back nasal vowels ang, iang, uang, eng, ueng, ing, ong, iong. Syllable / phoneme / initial / final classification detection is a mature existing technology in the field of speech recognition. For example, the mixed Gaussian hidden Markov algorithm mentioned in the paper "Design and Implementation of Embedded Syllable Recognition System" (author: Wang Xiaolong) of Huazhong University of Science and Technology can be used to realize the recognition of vowel syllables, and the technology in "Application of Vector Quantization Technology and Hidden Markov Model Method in Vowel Recognition" (author: Wu Jianxiong, Journal of Shanghai Jiaotong University) can also be used to recognize nasal vowels containing n and ng. The specific method will not be repeated in this invention.

[0057] Step S2, perform nasal speech recovery processing on the nasal final speech frame, and do not perform processing on other speech frames. In this first embodiment, the present invention realizes that different processing methods should be used for different categories of syllables, and syllables with little or no impact are not processed, and only syllables with greater impact (i.e., nasal finals) are processed, so as to ensure the recovery effect of nasal speech and avoid inappropriate processing of syllables with little / no impact, which leads to deterioration of sound quality; in addition, the algorithm complexity can be reduced and the processing efficiency can be improved. The low algorithm complexity also makes it easy for students to understand, thereby facilitating teaching.

[0058] The pronunciation of each word in Mandarin consists of an initial consonant and a final consonant. Figure 1A , Figure 1B , Figure 2A , Figure 2B From the spectrogram, we can see that the duration of the initial consonant part accounts for a very small proportion, which is 1 / 3 or even less of the entire Chinese character pronunciation duration, and its importance is relatively low; moreover, for voiceless initial consonants such as s, b, d, and g, their pronunciation uses less nasal airflow, so nasal congestion has little effect on these initial consonants, and when performing nasal congestion speech processing, there is no need to process these syllables.

[0059] The present invention only performs nasal speech recovery processing on the nasal finals speech frames, that is, only processes the nasal finals speech frames containing n and ng, that is, the speech frames of the 8 front nasal finals an, ian, uan, üan, en, uen, in, ün and the speech frames of the 8 back nasal finals ang, iang, uang, eng, ueng, ing, ong, iong.

[0060] Although the sound of the initial consonant "m" is most affected by a cold and nasal congestion, since the duration of the initial consonant accounts for a relatively short proportion, its importance is relatively low, and the processing difficulty is relatively high. Therefore, in order to achieve a compromise between the restoration processing effect and the algorithm complexity, the present invention does not process the initial consonant "m".

[0061]

Second Embodiment

[0062] Figure 6 It is a flowchart of the nasal congestion speech restoration processing in step S2, which specifically includes the following steps:

[0063] Step S21: Adjust the low-frequency spectrum amplitude spectrum of the nasal vowel speech frame to obtain the amplitude spectrum of the nasal vowel speech signal after low-frequency adjustment;

[0064] Step S22: Reconstruct the high-frequency amplitude spectrum of the nasal vowel speech frame according to the amplitude spectrum of the nasal vowel speech signal after low-frequency adjustment obtained in step S21 to obtain the amplitude spectrum after high-low frequency restoration;

[0065] Step S23: Combine the phase spectrum of the nasal vowel speech frame on the basis of the amplitude spectrum after high-low frequency restoration obtained in step S22 to obtain the restored spectrum, and obtain the restored speech frame time-domain signal according to the restored spectrum;

[0066] The pronunciation of "three" in Mandarin contains the nasal vowel "an". Figure 2A-2D Shows the spectrogram and amplitude spectrum diagram of the vowel "an" in the pronunciation of "three (san)". Figure 2A Is the spectrogram of the normal pronunciation of "three", Figure 2B Is the spectrogram of the nasal congestion speech of "three", Figure 2C Is the amplitude spectrum diagram of the normal pronunciation of "three", Figure 2D Is the amplitude spectrum diagram of the nasal congestion speech of "three". It can be found by comparison that:

[0067] The normal pronunciation has obvious high-frequency components in the high-frequency band between 6500 - 8000 Hz, and formants are formed in this part (such as Figure 2A , Figure 2C ). In the low-frequency band part, the peak position of the spectral amplitude is at the position of the first harmonic.

[0068] After a cold and nasal congestion, the spectral structure of the vowel an has changed significantly. In the high-frequency part, the high-frequency components between 6500 - 8000 Hz have almost disappeared. In the low-frequency part, the spectral amplitude has been distorted, and the peak position of the spectral amplitude has changed to the position of the fourth harmonic.

[0069] It can be seen that compared with normal speech, due to the change of the resonance structure, the amplitude spectrum envelope of the low-frequency components of the nasal congestion speech has been distorted, and the high-frequency component segment is missing.

[0070] To this end, step S2 uses a combination of adjusting the low-frequency spectrum envelope and reconstructing the high-frequency spectrum to process the nasal finals containing n and ng, achieving a better nasal congestion speech recovery effect.

[0071] Step S21, adjusting the low-frequency spectrum amplitude spectrum of the nasal vowel speech frame to obtain the low-frequency adjusted nasal vowel speech signal amplitude spectrum.

[0072] The low frequency part is selected as 0-5000 Hz, which is called the first low frequency part. 0-5000 Hz contains the main energy of the low frequency component. In step S21, the first low frequency part of the amplitude spectrum of the speech frame containing n and ng nasal vowels in the nasal speech signal is multiplied by the low frequency adjustment vector Low (1: k) to obtain the adjusted amplitude spectrum, thereby adjusting the first low frequency part.

[0073] The above low-frequency adjustment vector Low(1:k) is output through the trained low-frequency adjustment neural network. Different low-frequency adjustment neural networks are trained for different nasal vowels, for example, the neural network model C_an is trained for the vowel an, the neural network model C_ang is trained for the vowel ang, and so on. 16 low-frequency adjustment neural network models are trained to output low-frequency adjustment vectors, namely C_an, C_ian, C_uan, C_üan, C_en, C_uen, C_in, C_ün, C_ang, C_iang, C_uang, C_eng, C_ueng, C_ing, C_ong, C_iong.

[0074] Determine the corresponding low-frequency adjustment neural network model (for example, C_an) according to the category of the nasal vowel speech frame (speech frame) determined in step S1 (which one of the nasal vowels belongs to: an, ian, uan, üan, en, uen, in, ün, ang, iang, uang, eng, ueng, ing, ong, iong).

[0075] The amplitude spectrum F(1:1024) of the nasal final speech frame containing n and ng in the nasal congestion speech signal is input into the low-frequency adjustment neural network model (such as C_an) of the corresponding final, and the low-frequency adjustment vector Low(1:k) is output.

[0076] The low-frequency adjustment vector Low(1:k) is multiplied by the first low-frequency part F(1:k) of the amplitude spectrum F(1:1024) of the nasal vowel speech frame, and then combined with the remaining part F((k+1):1024) of F(1:1024) to obtain the low-frequency adjusted amplitude spectrum F_S21(1:1024).

[0077] When training the neural network, a large number of training samples from different people and different vowels are collected, and the low-frequency amplitude spectrum envelope of the normal nasal vowel speech frame is divided point by point by the low-frequency amplitude spectrum envelope of the cold nasal congestion speech frame, so as to obtain the low-frequency adjustment vector sample for training. In one example, the low-frequency amplitude spectrum envelope is the low-frequency amplitude spectrum envelope of the 1-5000Hz part. The low-frequency adjustment vector is expressed as Low(1:k)=[A1, A2, A3...Ak], k is the number of FFT frequency points of the first low-frequency part, and A1, A2...Ak are the magnification factors of each frequency point. For example, when the sampling frequency is 16KHz, the first low-frequency part is selected as 1-5000Hz, and the number of FFT sampling points is 1024, then k is 320. The low-frequency adjustment vector samples used for the above training are used as the output of the neural network, and the amplitude spectrum envelope of the cold and nasal congestion speech frame is used as the input of the neural network, and 16 low-frequency adjustment neural network models (i.e., C_an, C_ian, C_uan, C_üan, C_en, C_uen, C_in, C_ün, C_ang, C_iang, C_uang, C_eng, C_ueng, C_ing, C_ong, C_iong) are trained, which are suitable for cold speech with different nasal finals.

[0078] In this embodiment, different types of nasal sounds are passed through different types of neural network models to output different low-frequency adjustment vectors, so that the algorithm exhibits good adaptability.

[0079] Step S2 also includes step S22, reconstructing the high frequency amplitude spectrum of the nasal vowel speech frame;

[0080] Since the high frequency part is almost completely lost, it cannot be restored by envelope adjustment. Figure 2B , Figure 2D As can be seen, the high-frequency missing part is mainly located above 6KHz. Since the finals have good harmonic characteristics, the high-frequency part and the first low-frequency part have similar harmonic characteristics. According to this characteristic, the present invention adopts the method of low-frequency compensation high-frequency to achieve high-frequency recovery and reconstruction.

[0081] In step 22, the second low-frequency part of the amplitude spectrum of the nasal final speech signal after low-frequency adjustment obtained in step 21 is multiplied by the high-frequency reconstruction vector to obtain the reconstructed amplitude spectrum, and then according to the fundamental frequency, while maintaining the harmonic structure, the reconstructed amplitude spectrum is shifted to the high-frequency part, and superimposed with the amplitude spectrum F_S21 (1:1024) of the nasal final speech signal after low-frequency adjustment obtained in step 21, so as to obtain the amplitude spectrum after high and low frequency restoration.

[0082] In order to achieve the correct reconstruction of the high-frequency harmonic structure without destroying the harmonic structure of the high-frequency part, it is necessary to find the harmonic frequency point in the high-frequency range and then determine the reconstruction frequency band. For example, when the fundamental frequency is 300Hz, at 7000Hz, select a 1000Hz range roughly centered on the harmonic frequency point 7000Hz as the high-frequency band part to be reconstructed, that is, 6600Hz to 7600Hz, a total of 1000Hz as the high-frequency part to be reconstructed, and the starting frequency 6600Hz must be an integer multiple of the fundamental frequency to comply with the harmonic law. Thus, the low-frequency 0-1000Hz part (i.e., the second low-frequency part) of the amplitude spectrum of the nasal vowel speech signal after low-frequency adjustment is used to restore the amplitude spectrum of 6600Hz to 7600Hz.

[0083] The high-frequency reconstruction vector High (1: n) is a series of amplification coefficients High = [A1, A2, A3 ... An], and the value of n is related to the frequency range, sampling rate, and FFT point number. For example, when the reconstruction frequency range is selected to be 1000Hz and the sampling rate is 16KHz, then n = 64. Accordingly, the 0-1000Hz frequency range of the amplitude spectrum F_S21 (1: 1024) of the nasal vowel speech signal after low-frequency adjustment obtained in step 21, that is, the amplitude of each frequency point of the amplitude spectrum F_S21 (1: 64) of the 1-64 frequency points in the amplitude spectrum, is multiplied by the corresponding amplification coefficient in the high-frequency reconstruction vector High (1: 64), thereby obtaining the reconstructed amplitude spectrum F_S21_low (1: n). The reconstructed amplitude spectrum is migrated to the high-frequency reconstruction frequency band of 6400-7400 Hz, and superimposed with the low-frequency adjusted nasal vowel speech signal amplitude spectrum F_S21 (1:1024) obtained in step 21, so as to obtain the amplitude spectrum after high and low frequency restoration.

[0084] The high-frequency reconstruction vector High[1:n] is output through the trained high-frequency reconstruction neural network. Different high-frequency reconstruction neural networks are trained for different nasal finals. For example, the neural network model N_an is trained for the final an, and the neural network model N_ang is trained for the final ang, and so on. 16 high-frequency reconstruction neural network models are trained to output high-frequency reconstruction vectors, namely N_an, N_ian, N_uan, N_üan, N_en, N_uen, N_in, N_ün, N_ang, N_iang, N_uang, N_eng, N_ueng, N_ing, N_ong, N_iong.

[0085] Determine the corresponding high-frequency reconstruction neural network model (for example, N_an) according to the category of the nasal vowel speech frame (speech frame) determined in step S1 (which one of the nasal vowels belongs to: an, ian, uan, üan, en, uen, in, ün, ang, iang, uang, eng, ueng, ing, ong, iong).

[0086] The amplitude spectrum F(1:1024) of the nasal final speech frame containing n and ng in the nasal speech signal is input into the neural network model of the corresponding final (for example, N_an), and the high-frequency reconstruction vector High(1:n) is output.

[0087] When training the neural network, a large number of training samples of different people and different vowels are collected, and the amplitude spectrum envelope within the reconstruction range of the normal nasal vowel speech frame (for example, the 6400-7400 Hz band with a bandwidth of 1000 Hz near 7000 Hz determined based on the fundamental frequency of 300 Hz) is divided point by point by the low-frequency amplitude spectrum envelope of 1-1000 Hz of the cold nasal congestion speech frame, thereby obtaining the high-frequency reconstruction vector samples for training. The above-mentioned high-frequency reconstruction vector samples for training are used as the output of the neural network, and the amplitude spectrum envelope of the normal speech frame is used as the input of the neural network. 16 high-frequency reconstruction neural network models (i.e., N_an, N_ian, N_uan, N_üan, N_en, N_uen, N_in, N_ün, N_ang, N_iang, N_uang, N_eng, N_ueng, N_ing, N_ong, N_iong) are obtained through training, so as to be suitable for the cold speech of different nasal vowels.

[0088] In this embodiment, different types of nasal sounds output different high-frequency reconstruction vectors after passing through different types of neural network models, so that the algorithm exhibits good adaptability.

[0089] Step S23, combining the phase spectrum of the nasal vowel speech frame based on the restored amplitude spectrum of the high and low frequencies to obtain a restored spectrum, and obtaining a restored speech frame time domain signal according to the restored spectrum.

[0090] The result of step S22 output is the complete amplitude spectrum after high-frequency reconstruction and low-frequency recovery. In order to restore the complete spectrum, a phase spectrum is also required. The present invention directly combines the phase spectrum of the nasal vowel speech frame with the complete amplitude spectrum to obtain the complete spectrum after high-frequency reconstruction and low-frequency recovery. The time domain signal of the normal speech frame can then be restored through IFFT.

[0091] The present invention adopts the high-frequency reconstruction process of adjusting the amplitude spectrum of the low-frequency spectrum combined with the low-frequency compensation of the high-frequency, thereby achieving good recovery of nasal congestion speech.

[0092] [Third embodiment]

[0093] In this embodiment, real-time processing is considered. In some scenarios of nasal congestion speech recovery, the requirements for real-time processing are high, such as teachers giving lectures, announcers broadcasting, and anchors broadcasting. How to restore normal speech signals in real time is another technical problem to be solved by the present invention.

[0094] The premise of real-time processing is the real-time frame-by-frame detection of nasal finals containing n and ng. In order to complete the real-time frame-by-frame detection of nasal finals, the speech recognition technology based on the GMM-HMM model is used in step S1 of the present invention to complete the classification detection of nasal finals.

[0095] like Figure 3 As shown in the figure, the speech recognition technology based on the GMM-HMM model uses the acoustic model + lexicon model + language model to realize speech recognition. Among them, the acoustic model can complete the function of determining which nasal vowel the current frame belongs to. The lexicon model combines the phonemes into the pronunciation of the word, and the language model predicts the corresponding text based on the pronunciation.

[0096] The specific steps for completing the classification detection of nasal vowels using speech recognition technology based on the GMM-HMM model are as follows:

[0097] S11, using VAD endpoint detection algorithm to identify the starting frame of a sentence;

[0098] S12: After identifying the start frame, determine whether the current frame is a final frame or a non-final frame.

[0099] Since the present invention only processes the nasal finals containing n and ng, such finals (such as Figure 4 ) compared to the initial consonants (e.g. Figure 4 Therefore, the present invention firstly uses the harmonic characteristics to determine whether the current frame is a final speech frame or a non-final frame.

[0100] S12, performing classification detection based on the GMM-HMM model on the final speech frame frame by frame to determine whether the final speech frame belongs to one of the 16 types of nasal final speech frames including n and ng, specifically including:

[0101] For each vowel speech frame (10-20ms), 13-dimensional MFCC features are extracted as the observation vector of the current frame, and multi-frame MFCC feature vectors from the start frame of the sentence to the end of the current frame are obtained as the observation vector sequence. The observation vector sequence is substituted into the trained GMM-HMM model to obtain the result of which nasal vowel the current frame belongs to (refer to the paper of Xidian University, "Research and Implementation of Non-Specific Speaker Chinese Continuous Digital Speech Recognition System_Gao Chaohuang.pdf").

[0102] As shown in the figure above, when the V1 frame is detected as a final, the observation vectors of the C1, C2...Cn, and V1 frames of the activated speech are substituted into the GMM-HMM model as a whole to determine whether the V1 frame is a frame containing n or ng finals; when the V2 frame is detected as a final frame, the observation vectors of the C1, C2...Cn, V1, and V2 frames of the activated speech are substituted into the GMM-HMM model as a whole again to determine whether the V1 frame is a frame containing n or ng finals. This classification and judgment method of substituting the model frame by frame improves the real-time performance of syllable detection.

[0103] S13. When it is detected that the current frame is a frame containing nasal finals n or ng, execute the aforementioned step S2 to perform nasal congestion speech recovery processing on the nasal final speech frame.

[0104] [Fourth embodiment]

[0105] While improving real-time performance, the above-mentioned embodiments may reduce the accuracy of syllable judgment. For the HMM hidden Markov model, the more observations there are, the more accurate the classification is. Therefore, for the nasal finals of the first few frames, there may be a problem of misclassification, resulting in the restoration processing of frames that do not contain nasal finals such as n and ng, thus causing auditory degradation problems.

[0106] In order to reduce the auditory deterioration caused by inaccurate syllable detection, the present invention adopts the following two methods.

[0107] First, the tail detection of syllable detection is adopted. Since the duration of the finals occupies a long time in each word, only when multiple frames (for example, N frames, N is preferably a positive integer greater than or equal to 3) containing n and ng nasal finals are detected continuously after the starting frame in a sentence, for example, when 10 consecutive final frames V1, V2, V3, ... V10 are all nasal final frames, the 11th frame is considered to be a real nasal final speech frame, and the recovery process in step S2 is performed on the 11th frame and subsequent nasal final speech frames, thereby improving the processing accuracy.

[0108] Secondly, a fade-in processing technique is adopted. For the first few nasal vowel frames (for example, the first M frames, M is preferably a positive integer greater than or equal to 5) after the initial frame in a sentence, the judgment accuracy of nasal vowel is not high, for the speech frames of these nasal vowels, a recovery processing mode with low intervention degree is adopted, i.e., the low-frequency adjustment vector in step S2 and the high-frequency reconstruction vector need to be multiplied by a coefficient, and then multiplied with the first and second low-frequency parts of the amplitude spectrum of the nasal vowel speech frame, and this coefficient increases frame by frame, for example, for the first nasal vowel frame detected at the initial frame, this coefficient is 0.2, and for the second nasal vowel frame, this coefficient is 0.4, and so on. If 5 consecutive frames are all judged to contain n, ng nasal vowels, the processing intensity is promoted to 1. If non-nasal vowel frames appear in these 5 frames, step S2 is not executed, and the nasal vowel processing is stopped. This mode of gradually improving the processing degree avoids the mishandling caused by the inaccurate judgment of the first few frames and brings sound quality degradation.

[0109] [Fifth embodiment]

[0110] Computer majors in colleges and universities often offer courses such as multimedia technology and signal processing. Speech signal processing algorithms can be used as teaching experiments for students to learn and practice the principles of digital signal processing.

[0111] To this end, this patent also proposes an experimental teaching system and experimental method for nasal sound processing technology. The system includes a DSP development board, an audio acquisition circuit, and an audio output circuit. DSP preparation work includes connecting the DSP development board to a computer and configuring corresponding connection options and parameters. This includes selecting the correct interface connection method, setting serial port communication parameters, etc. Among them, the DSP development board includes a processor and an integrated development environment. Students open the integrated development environment, create a new project, and use the integrated development environment to write the signal processing algorithm code in the present invention to implement the processing method of the cold and nasal congestion speech of the aforementioned scheme. Use the debugging function provided by the development environment to step by step execute the code, view variables and memory status, and perform performance analysis and optimization. After debugging is completed, compile and download the code to the DSP development board for testing and verification. The sound collected by the audio acquisition circuit is processed in real time by the DSP development board, directly amplified and output through the audio output circuit, or input into a PC for storage.

[0112] In order to reduce the difficulty of neural network programming teaching in the present invention, a table lookup method is used to realize the simulation of the neural network model. That is, according to the training samples, a table lookup table is pre-prepared in advance, and the corresponding relationship between the amplitude spectrum F_Normal (1:1024) of the nasal vowel speech signal under normal conditions without nasal congestion and the high-frequency reconstruction vector is pre-prepared in the lookup table by clustering; and then the corresponding relationship between the amplitude spectrum F_Abnormal (1:1024) of the nasal vowel speech signal under nasal congestion and the low-frequency adjustment vector is established. For example, the following lookup table is pre-established and stored in the development board:

[0113] High frequency reconstruction vector lookup table of vowel an

[0114] Note: The second low frequency part is 1000Hz, sampling rate = 16KHz, n = 64)

[0115] 1 F_Normal_1(1:1024) High_AN_1[1:n] 2 F_Normal_2(1:1024) High_AN_2[1:n] 3 F_Normal_3(1:1024) High_AN_3[1:n] 4 F_Normal_4(1:1024) High_AN_4[1:n] ... ...... ......

[0116] Low frequency adjustment vector lookup table of vowel an

[0117] Note: The first low frequency part is 1~5000Hz, sampling rate = 16KH, k = 320)

[0118] 1 F_Abnormal_1(1:1024) Low_AN_1[1:k] 2 F_Abnormal_2(1:1024) Low_AN_2[1:k] 3 F_Abnormal_3(1:1024) Low_AN_3[1:k] 4 F_Abnormal_4(1:1024) Low_AN_4[1:k] ... ...... ......

[0119] By analogy, a corresponding high-frequency reconstruction vector and a low-frequency adjustment vector lookup table are established for each of the 16 nasal vowels. When processing a nasal congestion speech signal, when it is determined that the current speech frame is a syllable an in Mandarin, a lookup and comparison is performed in the low-frequency adjustment vector lookup table of the syllable an to find the amplitude spectrum sample with the closest Euclidean distance to the amplitude spectrum of the current nasal congestion n, ng nasal vowel frame, such as F_Abnormal_3 (1:1024), thereby outputting the corresponding low-frequency adjustment amplitude spectrum Low_AN_3 (1:320) to adjust the first low-frequency part of the amplitude spectrum, and then using the low-frequency adjusted nasal vowel speech signal amplitude spectrum F_S21 (1:1024) to look up and compare in the high-frequency reconstruction vector lookup table of the syllable an to find the amplitude spectrum of the nasal vowel speech signal with the current low-frequency adjustment The amplitude spectrum sample with the closest Euclidean distance, such as F_Normal_5(1:1024), is output to obtain the corresponding high-frequency reconstruction vector High_AN_5(1:64), and the second low-frequency part F_S21(1:64) of the amplitude spectrum F_S21 of the nasal vowel speech signal after low-frequency adjustment is multiplied by the high-frequency reconstruction vector High_AN_5(1:64) to obtain the reconstructed amplitude spectrum F_S21_low(1:n). When the fundamental frequency is 300Hz, the reconstructed amplitude spectrum is moved to the harmonic spectrum range near 7000Hz, that is, the range of 6600-7600Hz, and superimposed with the amplitude spectrum F_S21(1:1024) of the nasal vowel speech signal after low-frequency adjustment to obtain the amplitude spectrum after high and low frequency recovery, and then the amplitude spectrum is combined with the nasal speech frame phase spectrum to restore the original spectrum.

[0120] This embodiment can achieve the purpose of teaching DSP programming and development technology and speech signal processing technology, and the use of the lookup table can also reduce the difficulty of students' learning.

Claims

1. A method for processing Mandarin nasal congestion speech signals, characterized in that: The following steps are involved: Step S1, collecting a Mandarin nasal congestion speech signal, detecting a vowel speech frame in the nasal congestion speech signal, and classifying and detecting the vowel speech frame to determine whether the vowel speech frame belongs to one of 16 types of nasal vowel speech frames including n and ng, thereby detecting a nasal vowel speech frame and determining the category of the nasal vowel speech frame, that is, determining which category of the 16 types of nasal vowel speech frames the nasal vowel speech frame belongs to; The above-mentioned 16 types of nasal finals speech frames including n and ng refer to the speech frames of the 8 front nasal finals an, ian, uan, üan, en, uen, in, ün and the speech frames of the 8 back nasal finals ang, iang, uang, eng, ueng, ing, ong, iong; Step S2, performing nasal congestion speech recovery processing on the nasal vowel speech frame, and not performing processing on other speech frames; The nasal congestion speech recovery process in step S2 specifically includes the following steps: Step S21, adjusting the low-frequency spectrum amplitude spectrum of the nasal vowel speech frame to obtain the low-frequency adjusted nasal vowel speech signal amplitude spectrum; Step S22, reconstructing the high-frequency amplitude spectrum of the nasal vowel speech frame according to the amplitude spectrum of the nasal vowel speech signal after the low-frequency adjustment obtained in step S21, and obtaining the amplitude spectrum after high and low frequency restoration; Step S23, combining the phase spectrum of the nasal vowel speech frame based on the high and low frequency restored amplitude spectrum obtained in step S22 to obtain a restored spectrum, and obtaining a restored speech frame time domain signal according to the restored spectrum.

2. The method according to claim 1, characterized in that The step S21 of adjusting the low-frequency spectrum amplitude spectrum of the nasal vowel speech frame to obtain the low-frequency adjusted nasal vowel speech signal amplitude spectrum specifically includes: The first low-frequency part of the amplitude spectrum of the nasal vowels n and ng in the nasal speech signal is multiplied by the low-frequency adjustment vector to obtain the adjusted spectrum, and then the rest of the amplitude spectrum of the nasal vowel speech frame is superimposed to obtain the low-frequency adjusted amplitude spectrum.

3. The method according to claim 2, characterized in that The first low-frequency part in step S21 is specifically to select the frequency range of the low-frequency adjustment to be 0 to 5000 Hz; The low-frequency adjustment vector in step S21 is outputted through a trained low-frequency adjustment neural network, and different low-frequency adjustment neural network models are trained for different nasal finals, and 16 low-frequency adjustment neural network models are trained for the nasal finals an, ian, uan, üan, en, uen, in, ün, ang, iang, uang, eng, ueng, ing, ong, iong respectively; According to the category of the nasal final speech frame determined in step S1, the corresponding model is determined from 16 neural network models, the amplitude spectrum of the nasal final speech frame is input into the low-frequency adjustment neural network model of the corresponding final category, and a low-frequency adjustment vector is output. The low-frequency adjustment vector is multiplied by the first low-frequency part of the amplitude spectrum of the nasal final speech frame, and then the rest of the amplitude spectrum of the nasal final speech frame is superimposed to obtain the amplitude spectrum after low-frequency adjustment.

4. The method according to claim 1, characterized in that In step 22, the second low-frequency part of the amplitude spectrum of the nasal final speech signal after low-frequency adjustment obtained in step 21 is multiplied by the high-frequency reconstruction vector to obtain the reconstructed amplitude spectrum, and then according to the fundamental frequency, while maintaining the harmonic structure, the reconstructed amplitude spectrum is shifted to the high-frequency part and superimposed with the amplitude spectrum of the nasal final speech signal after low-frequency adjustment obtained in step 21, so as to obtain the amplitude spectrum after high and low frequency restoration.

5. The method according to claim 4, characterized in that The high-frequency reconstruction vector in step S22 is outputted through a trained high-frequency reconstruction neural network, and different high-frequency reconstruction neural network models are trained for different nasal finals, and 16 high-frequency reconstruction neural network models are trained for the nasal finals an, ian, uan, üan, en, uen, in, ün, ang, iang, uang, eng, ueng, ing, ong, iong respectively; According to the category of the nasal final speech frame determined in step S1, the corresponding model is determined from 16 high-frequency reconstruction neural network models, and the low-frequency adjusted nasal final speech signal amplitude spectrum of the nasal final speech frame output in step S21 is input into the high-frequency reconstruction neural network model of the corresponding final category, and a high-frequency reconstruction vector is output.

6. The method according to any one of claims 2 to 5, characterized in that: The step S1 comprises: S11, using VAD endpoint detection algorithm to identify the starting frame of a sentence; S12: After identifying the start frame, determine whether the current frame is a final frame or a non-final frame. S12, performing classification detection based on the GMM-HMM model on the vowel speech frame frame by frame to determine whether the vowel speech frame belongs to one of the 16 types of nasal vowel speech frames including n and ng: Extract 13-dimensional MFCC features for each final speech frame as the observation vector of the current frame, obtain the multi-frame MFCC feature vectors from the start frame of the sentence to the end of the current frame as the observation vector sequence, substitute the observation vector sequence into the trained GMM-HMM model, and obtain the result of which nasal final the current frame belongs to; S13. When it is detected that the current frame is a frame containing nasal finals n or ng, the nasal congestion speech recovery process described in the aforementioned step S2 is performed.

7. The method according to claim 6, characterized in that After the start frame is detected, when N frames are continuously detected to contain nasal finals of n and ng, the nasal speech recovery process described in step S2 is performed on the current frame and subsequent frames, where N is a positive integer greater than or equal to 3.

8. The method according to claim 6, characterized in that For the first M nasal final frames after the starting frame in a sentence, M is a positive integer greater than or equal to 5. The low-frequency adjustment vector in step S21 is multiplied by a coefficient to obtain a new low-frequency adjustment vector for nasal congestion speech processing; the high-frequency reconstruction vector in step S22 is multiplied by a coefficient to obtain a new high-frequency reconstruction vector for nasal congestion speech processing; the coefficient is less than 1 and increases frame by frame for the first M frames.

9. An experimental teaching system for implementing the nasal congestion speech signal processing method according to claims 1-8, the system comprising: DSP development board, as well as audio acquisition circuit and audio output circuit; The DSP development board includes a processor and an integrated development environment. The DSP development board allows students to download computer programs to the development board for testing and verification. The computer program can implement the nasal congestion speech signal processing method as described in claims 1-8. The sound collected by the audio acquisition circuit is processed in real time by the DSP development board, directly amplified and output through the audio output circuit, or input into a PC for storage.

10. An experimental teaching method for the experimental teaching system according to claim 9, the method comprising: DSP preparation includes connecting the DSP development board to the computer and configuring the corresponding connection options and parameters, including selecting the correct interface connection method, setting serial port communication parameters, etc. The student opens the integrated development environment, creates a new project, and uses the integrated development environment to write a signal processing computer program code, which implements the method for processing Mandarin nasal congestion speech signals described in any one of claims 1 to 8. The student uses the debugging function provided by the development environment to gradually execute the code, view variables and memory status, and perform performance analysis and optimization. After debugging is completed, the student compiles and downloads the computer program code to the DSP development board for testing and verification. The sound collected by the audio acquisition circuit is processed in real time by the DSP development board, directly amplified and output through the audio output circuit, or input into a PC for storage; The DSP development board stores a high-frequency reconstruction vector lookup table and a low-frequency adjustment vector lookup table.

Images