A method, system and device for converting fuzzy whispered speech

Abstract

This invention belongs to the field of speech signal processing, specifically relating to a method, system, and apparatus for converting ambiguous ear speech. The conversion method includes the following steps: S1: Designing an improved model for ear speech recognition based on classical generative adversarial networks. S2: Preparing clear ear speech corpora and ambiguous ear speech corpora to form a first training set, a second training set, and a test set. S3: Designing an audio synthesis module. S4: Training the improved model using the training and test sets. S5: Recognizing ear speech using the ear speech conversion model, then concatenating the recognized meta-audio and converting it into the desired complete audio of normal sound. The conversion system includes: a sample acquisition module, a preprocessing module, a signal optimization module, a backbone network, an audio synthesis module, a training organization module, and a post-processing module. This invention solves the problem of insufficient accuracy in recognizing ambiguous ear speech in existing ear speech conversion methods.

Description

Technical Field

[0001] The invention belongs to the field of speech signal processing, and in particular relates to a method, system and device for converting fuzzy whisper speech. Background Art

[0002] Speech communication is a unique way of communication that distinguishes humans from other animals. Normal human voice production mainly requires the abdominal cavity, vocal cords, airflow, and oral cavity to work together. The main components of the audio signal contained in speech can be roughly divided into voiced sounds produced by vocal cord vibration and unvoiced sounds produced by airflow. Whispering is a special way of speech communication. For example, whispering between people or whispering in a quiet environment mainly relies on audio produced by non-vocal cord vibration. This way of sound production that mainly contains unvoiced sound components produces whispering.

[0003] Speech that contains information is called language. People's language types are affected by regions and customs. The languages ​​of people in different countries and regions are not interoperable, which brings obstacles to communication between people. Speech recognition technology can recognize different human voices and convert them into a unified language or text, thereby reducing the cost of communication between people. With the continuous development of speech recognition technology, small languages ​​and dialects can also be accurately recognized. Whispering, as a special form of voice communication, plays a special role among some people whose vocal cords are damaged due to diseases or genetic factors; whispering allows these people to communicate normally. Therefore, the development of special whispering speech recognition technology plays an important role in improving the communication efficiency of these people.

[0004] The pronunciation principle of whispered speech is that the airflow passes through the half-open glottis, causing the vocal cavity to produce a sound signal. This pronunciation method has low energy. Compared with natural speech, this type of audio usually lacks fundamental frequency signals, and has large differences in intelligibility and naturalness. Therefore, it is difficult to be recognized by conventional speech recognition systems. In order to realize whispered speech conversion, it is necessary to develop special speech recognition tools for it. At present, technicians have developed whispered speech conversion methods based on different principles. These technical solutions can produce better recognition effects in whispered speech recognition, and the recognition efficiency and accuracy can basically meet the application requirements.

[0005] However, conventional whispered speech recognition tools can usually only recognize clear whispered speech. For some people who are damaged by diseases, if the vocal cords, larynx or other joints related to speech production are partially damaged; the audio generated may not be purely airflow-generated, nor does it include intermittent vocal cord noise. These signals will interfere with whispered speech recognition, and thus the whispered speech containing complex components will become fuzzy whispered speech. At the same time, environmental noise during the audio collection process may also affect the final speech recognition effect. It can be seen that the existing speech recognition schemes still have the problem of insufficient recognition accuracy when recognizing fuzzy whispered speech with complex components. Summary of the invention

[0006] In order to solve the problem of insufficient recognition accuracy of ambiguous whispered speech in the existing whispered speech conversion method, the present invention provides a conversion method, system and device for ambiguous whispered speech.

[0007] The present invention is implemented by the following technical solutions:

[0008] A method for converting fuzzy whispered speech, the method comprising the following steps:

[0009] S1: Design an improved model for whispered speech recognition based on the classic generative adversarial network. The improvements to the generative adversarial network are as follows:

[0010] S11: Add a Mel filter component as a preprocessing module in the generative adversarial network. The preprocessing module is used to convert the actual sampling frequency f of the audio signal s Convert to Mel frequency f m ; The output of the preprocessing module is used as the input of the generative adversarial network.

[0011] S12: A signal optimization module is connected before the signal preprocessing module; the signal optimization module is used to first perform frame processing on the input signal according to a preset period, and then perform windowing processing on the framed signal using a Blackman window function.

[0012] S13: In the generator of the generative adversarial network, the DTM algorithm and the self-attention module are used simultaneously to perform feature alignment on the generated normal sound. The DTM algorithm is used to align the explicit features of the speech, and the self-attention module is used to align the invisible features of the speech.

[0013] S14: Add a similarity metric module to the discriminator, and the similarity metric module serves as the second loss function of the discriminator during the training process.

[0014] S2: Prepare a number of clear whispered speech data and fuzzy whispered speech data; respectively form a first training set containing only clear whispered speech data, a second training set containing only fuzzy whispered speech data, and a test set.

[0015] S3: Design an audio synthesis module, which is used to superimpose fluctuating noise with a frequency spectrum ranging from 50-230 Hz on the input original audio signal.

[0016] S4: Use the training set and test set to train the improved model. The training process is as follows:

[0017] S41: The training process is divided into two stages, and the learning rates of the two stages are set to 0.0001-0.0002. The regularized hingeloss function is set as the loss function of the first stage of the generative adversarial network, and in the second stage, the second loss function is introduced into the discriminator.

[0018] S42: inputting the original audio in the first training set into the audio synthesis module, superimposing random fluctuation noise, and then inputting into the improved model for the first stage of training;

[0019] S43: Inputting the ambiguous whispered speech in the second training set into the improved model for the second stage of training.

[0020] S44: After the training is completed, the improved model is tested using the test set, and the following decisions are made based on the test results:

[0021] (1) When the test results meet the preset performance requirements, the parameters of the improved model in the training phase are saved to obtain the desired whisper-to-speech conversion model.

[0022] (2) When the test results meet the preset performance requirements, new corpus is collected to update the training set and test set, and the improved model is retrained based on the updated data set.

[0023] S5: The whispered speech conversion model is used to recognize the audio of the whispered speech to be converted, and then the recognized multiple mel-frequency vowel audios are concatenated and converted into the complete audio of the required normal sound.

[0024] As a further improvement of the present invention, in step S11, the preprocessing process of the signal preprocessing module is as follows:

[0025]

[0026] In the above formula, m represents the original whispered speech signal; f s represents the sampling frequency of the original signal; N represents the number of Fourier transform points; f h and f l Respectively represent the maximum and minimum values ​​of the preset filter frequency; F mel (f s) represents the Mel frequency of the original signal; f(m) represents the center frequency of the Mel spectrum of the original signal; M represents the number of filters.

[0027] As a further improvement of the present invention, in the framing process of step S12, the entire audio signal is divided into a plurality of continuous speech frames according to a period of 20-50 ms, and an overlap of 5-10% is set between each adjacent speech frame.

[0028] As a further improvement of the present invention, the definition of the Blackman window function in step S12 is as follows:

[0029]

[0030] In the above formula, n represents the total length of the window function, n∈N; M represents the effective length of the preset window function.

[0031] As a further improvement of the present invention, in step S14, the expression of the second loss function is as follows:

[0032]

[0033] In the above formula, D(·) represents the discriminator, M l represents the dimension of the discriminator; C(x) represents the samples in the second training set.

[0034] As a further improvement of the present invention, in step S41, in the first stage of the training process, the loss function of the discriminator is:

[0035] L D1 =E[max(0,MD(y))]+E[max(0,MD(G(z))];

[0036] In the above formula, E represents the expected value of the distribution function; D(y) represents the output of the discriminator; G(z) represents the output of the generator; M is a preset boundary vector; max(a,b) represents a maximum value function.

[0037] The loss function of the generator is:

[0038] L G1 =-E[D(G(z))].

[0039] As a further improvement of the present invention, in the second stage of the training process, the loss function of the discriminator is:

[0040] L D2 =L D1 +λL SIM ;

[0041] In the above formula, λ represents the coordination coefficient when the hingeloss function and the self-attention module are jointly trained.

[0042] The loss function of the generator is:

[0043] L G2 =L G1 .

[0044] As a further improvement of the present invention, in step S44, the preset performance requirement indicators include short-time objective intelligibility STOI, speech quality perception evaluation PESQ, and Mel spectrum distance MCD or log spectrum distance LSD. Among them, STOI is required to be no less than 0.5; PESQ is required to be greater than 1.0; MCD and LSD are respectively less than 7.0 and 1.3.

[0045] The present invention also includes a conversion system for fuzzy whispered speech, which is used to recognize and convert the collected whispered speech containing voiced sound and environmental noise to generate the corresponding normal sound audio. The conversion system includes: a sample collection module, a preprocessing module, a signal optimization module, a backbone network, an audio synthesis module, a training organization module, and a post-processing module.

[0046] The sample collection module is used to collect sample data in the training phase or the application phase, and classifies and saves the collected sample data according to clear whisper speech and fuzzy whisper speech in the training phase to obtain a first training set and a second training set.

[0047] The signal optimization module includes a framing unit and a windowing unit. The signal optimization module is used to perform framing and windowing processing on the input audio signal in sequence. The framing unit is used to divide the entire audio signal into multiple speech frames and ensure that adjacent speech frame signals contain overlapping parts in data. The windowing unit uses the Blackman window function as the processing function.

[0048] The preprocessing module is used to preprocess the input audio signal, thereby converting the actual sampling frequency of the audio signal into Mel frequency.

[0049] The backbone network adopts the classic generative adversarial network, which includes a generator and a discriminator. The generator of the generative adversarial network adopts both the DTM algorithm and the self-attention module, which align the generated speech segments based on explicit features and implicit features respectively. A similarity measurement module is added to the discriminator. The similarity measurement module is used to evaluate the similarity of speech signals from different sources and serves as the second loss function of the discriminator during the training process. The backbone network after training is used for fuzzy ear speech recognition. In the recognition stage, the fuzzy ear speech data to be recognized in the sample acquisition module is directly processed by the signal optimization module and the preprocessing module and then input into the backbone network; the backbone network outputs the recognized continuous mel-spectrogram-based meta-audio.

[0050] The audio synthesis module is used to degrade the whispered speech signal by superimposing fluctuating noise with a frequency spectrum range of 50-230 Hz on the whispered speech signal, and the signal after superimposing the fluctuating noise is used as the input sample of the first stage of the training process.

[0051] The training organization module is used to generate training plans and conduct two-stage training on the backbone network. In the first stage, the clear whispered speech data in the first training set is first input into the audio synthesis module for degradation processing. The degraded sample data is processed by the preprocessing module and the signal optimization module in turn, and finally used for backbone network training. In the second stage, the fuzzy whispered speech data in the second training set is processed by the signal optimization module and the preprocessing module in turn, and finally used for backbone network training. At the same time, the regularized hingeloss function is used as the loss function in the first stage, and the hingeloss function and the second loss function of the self-attention module are used in the second stage to jointly train the generator; until the trained backbone network meets the requirements of the preset training indicators.

[0052] The post-processing module includes a splicing unit and a conversion unit. The splicing unit is used to splice the continuous primitive audios output by the backbone network in the recognition stage into a complete audio signal according to the overlapping parts in the spectrum. The spliced ​​audio signal is a Mel audio signal. The conversion unit is used to convert the Mel audio signal into an audio signal of normal speech.

[0053] The present invention also includes a device for converting fuzzy-whispering speech, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The conversion device is deployed in accordance with the sample acquisition module, preprocessing module, signal optimization module, postprocessing module, and trained backbone network in the aforementioned fuzzy-whispering speech conversion system. When the processor executes the program, the steps of the aforementioned method for converting fuzzy-whispering speech are implemented.

[0054] The technical solution provided by the present invention has the following beneficial effects:

[0055] The present invention designs a new speech recognition network model based on the pronunciation mechanism and audio characteristics of ambiguous whispered speech. The speech recognition network model takes a generative adversarial network as the basic network, and introduces a self-attention module and a similarity measurement module into the generative adversarial network. The similarity measurement module improves the learning ability of the generative adversarial network for the implicit features in the ambiguous whispered speech during the training stage, and the self-attention module can adaptively perform invisible time alignment on the whispered speech and the normal sound during the model training process.

[0056] In the training stage of the whispered speech recognition network, in order to improve the learning effect of the network model on the audio features in the fuzzy whispered speech, the present invention adopts a two-stage training scheme to gradually train the model. During the training process, the clear whispered speech is first degraded as the training sample of the first stage, and then the fuzzy whispered speech is used for the second stage training. The use of this training scheme can not only make the network model converge quickly and reduce the number of iterations in the training stage; it can also increase the generalization of the network model, so that the network model can show the best recognition effect for various special fuzzy whispered speech, and improve the accuracy of whispered speech conversion. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 The present invention is a flowchart of a method for converting ambiguous whispered speech provided in Embodiment 1 of the present invention.

[0058] Figure 2 This is a flow chart of the network model training and application process in Example 1 of the present invention.

[0059] Figure 3 Schematic diagram of the overall framework of the whispered speech recognition network provided in Example 1 of the present invention.

[0060] Figure 4 This is a schematic diagram of a model of a system for converting ambiguous whispered speech provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0062] Example 1

[0063] Fuzzy whispered speech has the characteristics of normal vocal cord pronunciation and whispered speech airflow pronunciation. At the same time, the audio signal also contains many different types of random noise. Therefore, it is much more difficult to recognize than clear whispered speech and various normal speech.

[0064] In view of the special properties of the fuzzy whisper speech component, this embodiment designs a new speech recognition model based on the generative adversarial network, and optimizes and adjusts the training process of the network model according to the type of speech to be recognized. The generative adversarial network is used to automatically learn the various features contained in the fuzzy whisper speech, and then establish a mapping relationship between the fuzzy whisper speech and the normal sound.

[0065] like Figure 1 and Figure 2 As shown, the method for converting fuzzy whispered speech provided in this embodiment includes the following steps:

[0066] S1: Design an improved model for whispered speech recognition based on the classic generative adversarial network. The improvements to the generative adversarial network are as follows:

[0067] S11: Add a Mel filter component as a preprocessing module in the generative adversarial network. The preprocessing module is used to convert the actual sampling frequency f of the audio signal s Convert to Mel frequency f m . And the output of the preprocessing module is used as the input of the generative adversarial network.

[0068] Among them, the preprocessing process of the signal preprocessing module is as follows:

[0069]

[0070] In the above formula, m represents the original whispered speech signal; f s represents the sampling frequency of the original signal; N represents the number of Fourier transform points; f h and f l Respectively represent the maximum and minimum values ​​of the preset filter frequency; F mel (f s ) represents the Mel frequency of the original signal; f(m) represents the center frequency of the Mel spectrum of the original signal; M represents the number of filters.

[0071] There are obvious differences in the pronunciation mechanism of whispered speech and normal sound. The excitation source of whispered speech is noise, so whispered speech is completely non-vibrating whether it is the voiced segment or the unvoiced end. Therefore, from the perspective of acoustic characteristics, the audio signal component of whispered speech usually has no fundamental frequency component, or has less fundamental frequency component, and the signal-to-noise ratio and energy are relatively low compared to normal sound. In addition, a large amount of airflow is required when whispered speech is pronounced. Considering that people have limited lung capacity, the speed of whispered speech will be significantly slower than normal sound, which causes the single tone length of whispered speech to be longer than normal sound. In addition, there are also differences in the vocal tract transfer function of whispered speech and normal sound, which causes the bandwidth and position of the resonance peak in the audio signal to shift. Therefore, in order to more accurately analyze the characteristics of whispered speech and facilitate the network model to learn deeper information from it, this embodiment first converts the whispered speech from the normal sampling frequency to the Mel spectrum.

[0072] S12: A signal optimization module is connected before the signal preprocessing module; the signal optimization module is used to first perform frame processing on the input signal according to a preset period, and then perform windowing processing on the framed signal using a Blackman window function.

[0073] During the framing process, the entire audio signal is divided into multiple continuous speech frames according to a period of 20-50 ms, and a 5-10% overlap is set between each adjacent speech frame.

[0074] The definition of the Blackman window function used in the windowing process is as follows:

[0075]

[0076] In the above formula, n represents the total length of the window function, n∈N; M represents the effective length of the preset window function.

[0077] Taking into account that a longer spectrum contains too much information, the network model may not be able to learn the spectral features of each syllable or sentence during feature learning. Therefore, this embodiment uses frame processing to appropriately adjust the duration of a single audio input into the network model. The principle of adjustment is mainly to keep it consistent with the syllable length of the whispered speech pronunciation process, to ensure that the characteristics of the single syllable itself are not destroyed, and to include the characteristics of as many continuous syllables as possible.

[0078] In particular, in order to facilitate the testing of the normal sound generated by the model in the later stage of recognition and the splicing of continuous audio signals, this embodiment also reserves a part for each audio unit that overlaps with the previous audio and the next audio in the frame processing. Since the frame processing operates on the time domain of the signal, from the time domain point of view, the overlapping part accounts for about 5-10% of the entire audio cycle.

[0079] In this embodiment, the Blackman window function is used to perform windowing processing on the signal. During the windowing process, the resolution broadening of the signal can be enhanced while maintaining the peak sidelobe ratio, so that the integral sidelobe ratio is effectively reduced, thereby effectively eliminating the influence of invalid noise in the audio signal and improving the network model's ability to learn implicit features in speech.

[0080] S13: In the generator of the generative adversarial network, the DTM algorithm and the self-attention module are used simultaneously to perform feature alignment on the generated normal sound. The DTM algorithm is used to align the explicit features of the speech, and the self-attention module is used to align the invisible features of the speech.

[0081] S14: Add a similarity metric module to the discriminator, and the similarity metric module serves as the second loss function of the discriminator during the training process.

[0082] In the learning process of the generative adversarial network, the generator mainly learns the relationship between the source data and the generated data through continuous training, and establishes a mapping between the two in the hidden feature space. Then the discriminator uses the abstract structure determined by the hidden feature space to distinguish the difference between the real data and the generated data. The similarity is an indicator used to measure the similarity between the two types of data. Therefore, this embodiment introduces a similarity measurement module for calculating similarity into the training process of the network model as another loss function adopted; it can be used in the training process to jointly train the generative adversarial network.

[0083] The expression of the second loss function is as follows:

[0084]

[0085] In the above formula, D(·) represents the discriminator, M l represents the dimension of the discriminator; C(x) represents the samples in the second training set.

[0086] S2: Prepare a number of clear whispered speech data and fuzzy whispered speech data; respectively form a first training set containing only clear whispered speech data, a second training set containing only fuzzy whispered speech data, and a test set.

[0087] This embodiment intends to train the network model in two stages so that the network model gradually learns the difference between the fuzzy whisper speech material and the conventional clear whisper speech material. Therefore, the clear whisper speech and the fuzzy whisper speech are collected as the data sets used in the training stage.

[0088] S3: Design an audio synthesis module, which is used to superimpose fluctuating noise with a frequency spectrum ranging from 50-230 Hz on the input original audio signal.

[0089] In the network model training process involved in this embodiment, the audio synthesis module is first used to degrade the clear whispered speech, and the degraded signal is used to train the network model, which can significantly enhance the generalization of the model and improve the model's recognition ability for the whispered speech signal after superimposed interference. In fact, the complexity of the degraded signal is still lower than that of the real fuzzy whispered speech. Therefore, this embodiment performs the second stage of training on the basis of the first stage of training until the network model can establish a relatively reliable mapping between the fuzzy whispered speech and the normal sound.

[0090] The solution provided in this embodiment selects the best from various types of random noises and selects the noise signal that is most effective in improving the model training efficiency. The fluctuating noise with a spectrum range of 50-230Hz is most similar to the interference model contained in the fuzzy whispering speech, so using this noise type in the first stage can enable the network model to learn the main information in the fuzzy whispering speech. When the audio signal after the random noise signal is degraded is used for the first stage training, the training set data scale of the first stage training and the second stage training is approximately (2-2.5):1.

[0091] S4: The improved model is trained using the training set and the test set. In the training stage of the network model of this embodiment, considering the different training objectives at different stages, different loss functions are set for the training tasks at different stages in order to improve the training effect of the model. In the first stage, only the hingeloss function is used as the loss function for model optimization, while in the second stage, the similarity function is further introduced for joint training.

[0092] Specifically, the training process of the network model is as follows:

[0093] S41: The training process is divided into two stages, and the learning rates of the two stages are set to 0.0001-0.0002. The regularized hingeloss function is set as the loss function of the first stage of the generative adversarial network, and in the second stage, the second loss function is introduced into the discriminator.

[0094] In the first stage of the training process, the loss function of the discriminator is:

[0095] L D1 =E[max(0,MD(y))]+E[max(0,MD(G(z))];

[0096] In the above formula, E represents the expected value of the distribution function; D(y) represents the output of the discriminator; G(z) represents the output of the generator; M is a preset boundary vector; max(a,b) represents a maximum value function.

[0097] The loss function of the generator is:

[0098] L G1 =-E[D(G(z))].

[0099] In the second stage of the training process, the loss function of the discriminator is:

[0100] L D2 =L D1 +λL SIM ;

[0101] In the above formula, λ represents the coordination coefficient when the hingeloss function and the self-attention module are jointly trained.

[0102] The loss function of the generator is:

[0103] L G2 =L G1 .

[0104] S42: inputting the original audio in the first training set into the audio synthesis module, superimposing random fluctuation noise, and then inputting into the improved model for the first stage of training;

[0105] S43: Inputting the ambiguous whispered speech in the second training set into the improved model for the second stage of training.

[0106] S44: After training, the improved model is tested using the test set, and the following decisions are made based on the test results:

[0107] (1) When the test results meet the preset performance requirements, the parameters of the improved model in the training phase are saved to obtain the desired whisper-to-speech conversion model.

[0108] (2) When the test results meet the preset performance requirements, new corpus is collected to update the training set and test set, and the improved model is retrained based on the updated data set.

[0109] The training process of generating adversarial networks is roughly as follows Figure 3 shown.

[0110] S5: The whispered speech conversion model is used to recognize the audio of the whispered speech to be converted, and then the recognized multiple mel-frequency vowel audios are concatenated and converted into the complete audio of the required normal sound.

[0111] In particular, the performance requirement indicators preset in this embodiment include short-time objective intelligibility STOI, speech quality perception evaluation PESQ, and Mel spectrum distance MCD or log spectrum distance LSD. Among them, the STOI indicator is used to objectively evaluate the intelligibility of the converted speech, and its value range is between 0-1. The closer the value is to 1, the higher the intelligibility of the converted speech and the better the conversion system. PESQ is used to objectively evaluate the quality of the converted speech. Its value range is between 0.5-4.5. The larger the value, the higher the quality of the converted speech and the closer it is to the normal sound. MCD and LSD are two commonly used evaluation indicators for calculating the performance of the converted speech spectrum. The lower the value, the higher the spectral similarity between the generated normal sound and the reference sound.

[0112] For the whisper-to-speech conversion model designed in this embodiment, STOI is required to be no less than 0.5; PESQ is required to be greater than 1.0; at the same time, MCD and LSD are respectively less than 7.0 and 1.3; this embodiment selects STOI and PESQ as the main indicators for evaluating model performance, and at least one of MCD and LSD is used as an auxiliary indicator. After training, at least one of the main indicators and the auxiliary indicators must meet the requirements.

[0113] Performance Testing

[0114] In this example, in order to prove the effectiveness of the experimental results, a control experiment is set up to compare the traditional speech conversion methods GMM, BLSTM, Cycle-GAN and Sequence-to-Sequence with this implementation. The experimental data set of the above methods is the same as the data set used in this example, and the DTW algorithm is used to perform speech time alignment processing on the whispered speech and the generated sound. The objective evaluation indicators of this example method and the control group on the dual-end indicators are shown in Table 1 below:

[0115] Table 1: Results of the two-end evaluation indicators of normal sounds generated by each comparison model

[0116]

[0117] As can be seen from Table 1, this example method has achieved good results in various dual-end indicators in the task of converting whispered speech to normal speech. However, due to the particularity of whispered speech pronunciation, whispered speech is usually pronounced slower than normal speech, resulting in a longer time series of whispered speech than normal speech. When calculating the dual-end indicator value, a reference speech needs to be provided, and the speech is forced to be aligned using the DTW algorithm, resulting in speech distortion. In order to more accurately evaluate the conversion effect of whispered speech, this example also uses the p.563 single-end indicator for evaluation. The P.563 algorithm does not require forced alignment of speech and can simulate the audience's evaluation of speech quality. In the single-end indicator evaluation, GMM, BLSTM, Cycle-GAN and Sequence-to-Sequence are compared respectively, and the results are shown in Table 2 below.

[0118] Table 2: Evaluation results of normal tone single-ended indicators generated by each comparison model

[0119]

[0120] As can be seen from Table 2, the single-end index of this example method exceeds the traditional voice conversion method and achieves the highest score. The evaluation result on the p.563 index is very close to the result of normal sound. The experimental results indirectly reflect that our attention mechanism and similarity loss have greater advantages than the traditional DTW loss and can restore the characteristics of whispered speech to the greatest extent.

[0121] Example 2

[0122] This embodiment provides a system for converting fuzzy whispered speech, which is used to recognize and convert the collected whispered speech containing voiced sound and environmental noise by using the fuzzy whispered speech conversion method in Example 1, and generate the corresponding normal sound audio. Figure 4As shown, the whisper-to-speech conversion system includes: a sample collection module, a preprocessing module, a signal optimization module, a backbone network, an audio synthesis module, a training organization module, and a post-processing module.

[0123] The sample collection module is used to collect sample data in the training phase or the application phase, and classifies and saves the collected sample data according to clear whisper speech and fuzzy whisper speech in the training phase to obtain a first training set and a second training set.

[0124] The signal optimization module includes a framing unit and a windowing unit. The signal optimization module is used to perform framing and windowing processing on the input audio signal in sequence. The framing unit is used to divide the entire audio signal into multiple speech frames and ensure that adjacent speech frame signals contain overlapping parts in data. The windowing unit uses the Blackman window function as the processing function.

[0125] The preprocessing module is used to preprocess the input audio signal, thereby converting the actual sampling frequency of the audio signal into Mel frequency.

[0126] The backbone network adopts the classic generative adversarial network, which includes a generator and a discriminator. The generator of the generative adversarial network adopts both the DTM algorithm and the self-attention module, which align the generated speech segments based on explicit features and implicit features respectively. A similarity measurement module is added to the discriminator. The similarity measurement module is used to evaluate the similarity of speech signals from different sources and serves as the second loss function of the discriminator during the training process. The backbone network after training is used for fuzzy ear speech recognition. In the recognition stage, the fuzzy ear speech data to be recognized in the sample acquisition module is directly processed by the signal optimization module and the preprocessing module and then input into the backbone network; the backbone network outputs the recognized continuous mel-spectrogram-based meta-audio.

[0127] The audio synthesis module is used to degrade the whispered speech signal by superimposing fluctuating noise with a frequency spectrum range of 50-230 Hz on the whispered speech signal, and the signal after superimposing the fluctuating noise is used as the input sample of the first stage of the training process.

[0128] The training organization module is used to generate training plans and conduct two-stage training on the backbone network. In the first stage, the clear whispered speech data in the first training set is first input into the audio synthesis module for degradation processing. The degraded sample data is processed by the preprocessing module and the signal optimization module in turn, and finally used for backbone network training. In the second stage, the fuzzy whispered speech data in the second training set is processed by the signal optimization module and the preprocessing module in turn, and finally used for backbone network training. At the same time, the regularized hingeloss function is used as the loss function in the first stage, and the hingeloss function and the second loss function of the self-attention module are used in the second stage to jointly train the generator; until the trained backbone network meets the requirements of the preset training indicators.

[0129] The post-processing module includes a splicing unit and a conversion unit. The splicing unit is used to splice the continuous primitive audios output by the backbone network in the recognition stage into a complete audio signal according to the overlapping parts in the spectrum. The spliced ​​audio signal is a Mel audio signal. The conversion unit is used to convert the Mel audio signal into an audio signal of normal speech.

[0130] Example 3

[0131] On the basis of Embodiments 1 and 2, this embodiment further provides a device for converting fuzzy whisper speech, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The conversion device is deployed in accordance with the sample acquisition module, preprocessing module, signal optimization module, post-processing module, and trained backbone network of the fuzzy whisper speech conversion system in Embodiment 2. When the processor executes the program, the steps of the method for converting fuzzy whisper speech in Embodiment 1 are implemented.

[0132] The computer device may be a smart phone, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server or cabinet server (including an independent server or a server cluster composed of multiple servers) that can execute programs. The computer device of this embodiment includes at least but is not limited to: a memory and a processor that can communicate with each other through a system bus.

[0133] In this embodiment, the memory (i.e., readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory may also be an external storage device of a computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Of course, the memory may also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory is generally used to store an operating system and various application software installed on the computer device. In addition, the memory may also be used to temporarily store various types of data that have been output or are to be output.

[0134] The processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor is generally used to control the overall operation of a computer device. In this embodiment, the processor is used to run the program code stored in the memory or process data to implement the processing steps of converting the fuzzy whispered speech into normal sound as in Embodiment 1.

[0135] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for converting fuzzy whispered speech, characterized in that: The steps include: S1: Design an improved model for whispered speech recognition based on the classic generative adversarial network. The improvements of the generative adversarial network are as follows: S11: Add a Mel filter component as a preprocessing module in the generative adversarial network, and the preprocessing module is used to convert the actual sampling frequency f of the audio signal s Convert to Mel frequency f m ;The output of the preprocessing module is used as the input of the generative adversarial network; S12: A signal optimization module is connected before the preprocessing module; the signal optimization module is used to first perform frame processing on the input signal according to a preset period, and then perform windowing processing on the framed signal using a Blackman window function; S13: In the generator of the generative adversarial network, the DTM algorithm and the self-attention module are used simultaneously to perform feature alignment on the generated normal sound; wherein the DTM algorithm is used to align the explicit features of the speech, and the self-attention module is used to align the invisible features of the speech; S14: adding a similarity metric module to the discriminator, and the similarity metric module serves as the second loss function of the discriminator during the training process; S2: prepare a number of clear whispered speech data and fuzzy whispered speech data; respectively form a first training set containing only clear whispered speech data, a second training set containing only fuzzy whispered speech data, and a test set; S3: design an audio synthesis module, wherein the audio synthesis module is used to superimpose a fluctuating noise with a frequency spectrum ranging from 50 to 230 Hz on an input original audio signal; S4: The improved model is trained using the training set and the test set. The training process is as follows: S41: The training process is divided into two stages, and the learning rates of the two stages are set to 0.0001-0.0002; the regularized hingeloss function is set as the loss function of the first stage of the generative adversarial network, and in the second stage, the second loss function is introduced into the discriminator; S42: inputting the original audio in the first training set into the audio synthesis module, superimposing random fluctuation noise, and then inputting it into the improved model for the first stage of training; S43: inputting the ambiguous whispered speech in the second training set into the improved model for second stage training; S44: After the training is completed, the improved model is tested using the test set, and the following decision is made based on the test result: (1) When the test results meet the preset performance requirements, the parameters of the improved model in the training phase are saved. Obtaining the desired whisper-speech conversion model; (2) When the test results meet the preset performance requirements, new corpus is collected to update the training set and the test set, and the improved model is retrained based on the updated data set; S5: using the whispered speech conversion model to recognize the audio of the whispered speech to be converted, and then concatenating the recognized multiple mel-frequency vowel audios and converting them into the required complete audio of the normal sound.

2. The method for converting fuzzy whispered speech as claimed in claim 1, characterized in that: In step S11, the preprocessing process of the preprocessing module is as follows: In the above formula, m represents the original whispered speech signal; f s represents the sampling frequency of the original signal; N represents the number of Fourier transform points; f h and f l Respectively represent the maximum and minimum values ​​of the preset filter frequency; F mel (f s ) represents the Mel frequency of the original signal; f(m) represents the center frequency of the Mel spectrum of the original signal; M represents the number of filters.

3. The method for converting ambiguous whispered speech according to claim 1, characterized in that: In the framing process of step S12, the entire audio signal is divided into a plurality of continuous speech frames according to a period of 20-50 ms, and an overlap of 5-10% is set between each adjacent speech frame.

4. The method for converting fuzzy whispered speech as claimed in claim 1, characterized in that: The Blackman window function in step S12 is defined as follows: In the above formula, n represents the total length of the window function, n∈N; M represents the effective length of the preset window function.

5. The method for converting fuzzy whispered speech as claimed in claim 1, characterized in that: In step S14, the expression of the second loss function is as follows: In the above formula, D(·) represents the discriminator, M l represents the dimension of the discriminator; C(x) represents the samples in the second training set.

6. The method for converting ambiguous whispered speech as claimed in claim 5, characterized in that: In step S41, in the first stage of the training process, the loss function of the discriminator is: L D1 =E[max(0,M-D(y))]+E[max(0,M-D(G(z))] In the above formula, E represents the expected value of the distribution function; D(y) represents the output of the discriminator; G(z) represents the output of the generator; M is a preset boundary vector; max(a,b) represents a maximum value function; The loss function of the generator is: L G1 =-E[D(G(z))]。 7. The method for converting vague whispered speech according to claim 6, characterized in that: In the second stage of the training process, the loss function of the discriminator is: THE D2 =L D1 +λL SIM In the above formula, λ represents the coordination coefficient when the hingeloss function and the self-attention module are jointly trained; The loss function of the generator is: L G2 =L G1 .

8. The method for converting ambiguous whispered speech as claimed in claim 1, characterized in that: In step S44, the preset performance requirement indicators include short-time objective intelligibility STOI, speech quality perception evaluation PESQ, and Mel spectrum distance MCD or log spectrum distance LSD; Among them, STOI is required to be no less than 0.5; PESQ is required to be greater than 1.0; MCD and LSD are required to be lower than 7.0 and 1.3 respectively.

9. A system for converting fuzzy whispered sounds, which is used to recognize and convert collected whispered sounds containing voiced sounds and environmental noise to generate corresponding normal sound audio; characterized in that: The conversion system comprises: A sample collection module, which is used to collect sample data in a training phase or an application phase, and classify and save the collected sample data according to clear whispered speech and fuzzy whispered speech in the training phase to obtain a first training set and a second training set; A signal optimization module, comprising a framing unit and a windowing unit; the signal optimization module is used to perform framing processing and windowing processing on the input audio signal in sequence; the framing unit is used to divide the entire audio signal into multiple voice frames, and ensure that adjacent voice frame signals contain overlapping parts in data; the windowing unit uses a Blackman window function as a processing function; A preprocessing module, which is used to preprocess the audio signal output by the signal optimization module, and then convert the actual sampling frequency of the audio signal into Mel frequency; The backbone network adopts a classic generative adversarial network, which includes a generator and a discriminator; the generator of the generative adversarial network adopts both a DTM algorithm and a self-attention module, which respectively perform feature alignment on the generated speech segments based on explicit features and implicit features; a similarity measurement module is added to the discriminator, and the similarity measurement module is used to evaluate the similarity of speech signals from different sources and serves as the second loss function of the discriminator during the training process; the backbone network after training is used for fuzzy ear speech recognition, and in the recognition stage, the fuzzy ear speech data to be recognized in the sample acquisition module is directly processed by the signal optimization module and the preprocessing module and then input into the backbone network; the backbone network outputs each recognized continuous mel-spectrogram-based vowel audio; An audio synthesis module is used to degrade the whispered speech signal by superimposing fluctuating noise with a frequency spectrum ranging from 50 to 230 Hz on the whispered speech signal, and the signal after superimposing the fluctuating noise is used as an input sample for the first stage of the training process; A training organization module, which is used to generate a training plan and perform two-stage training on the backbone network; in the first stage, the clear whispered speech data in the first training set is first input into the audio synthesis module for degradation processing, and the degraded sample data is processed by the preprocessing module and the signal optimization module in turn, and finally used for backbone network training; in the second stage, the fuzzy whispered speech data in the second training set is processed by the signal optimization module and the preprocessing module in turn, and finally used for backbone network training; at the same time, in the first stage, the regularized hingeloss function is used as the loss function, and in the second stage, the hingeloss function and the second loss function of the self-attention module are used to jointly train the generator; until the trained backbone network meets the requirements of the preset training indicators; The post-processing module includes a splicing unit and a conversion unit; the splicing unit is used to splice the continuous primitive audios output by the backbone network in the recognition stage into a complete audio signal according to the overlapping parts in the spectrum, and the spliced ​​audio signal is a Mel audio signal; the conversion unit is used to convert the Mel audio signal into an audio signal of normal speech.

10. A device for converting fuzzy whispered speech, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The conversion device is deployed in accordance with the sample acquisition module, preprocessing module, signal optimization module, post-processing module and trained backbone network in the conversion system of ambiguous whispering speech as claimed in claim 9; and when the processor executes the program, the steps of the conversion method of ambiguous whispering speech as claimed in any one of claims 1 to 8 are implemented.

Images