Multi-modal speech enhancement algorithm based on semi-supervised learning pre-training

Abstract

The invention provides a multi-modal speech enhancement algorithm based on semi-supervised learning pre-training. The multi-modal speech enhancement algorithm comprises the following steps: S1, taking an extracted noisy speech feature vector # imgabs0 # as the input of a noisy speech GCRBM; s2, de-noising the noisy voice # imgabs 1 # by using a traditional spectral subtraction method to obtain a de-noised voice # imgabs 2 #; s3, taking the estimated de-noised speech feature vector # imgabs3 # as the input of the pure speech GCRBM; s4, taking the extracted video signal feature vector # imgabs4 # as the input of the video GCRBM; s5, optimizing parameters of the multi-mode speech enhancement network; s6, during testing, noise-containing speech features are input to the multi-mode speech enhancement network; according to the method, the voice features and the mouth features are extracted and input into the multi-mode voice enhancement network, and after parameter optimization of the multi-mode voice enhancement network is carried out, the noisy voice features are input into the multi-mode voice enhancement network to obtain the enhanced voice signals, so that the voice annotation cost is reduced, and the intelligibility of voice enhancement is improved.

Description

Technical Field

[0001] The present invention relates to a speech enhancement algorithm, in particular to a multimodal speech enhancement algorithm based on semi-supervised learning pre-training, and belongs to the technical field of speech enhancement. Background Art

[0002] Speech enhancement aims to improve the quality and clarity of speech signals by eliminating background noise, reverberation, distortion and other interference in speech signals. Currently, there are two main types of speech enhancement algorithms based on label learning: one is a speech enhancement algorithm based on supervised learning, and the other is a speech enhancement algorithm based on unsupervised learning. Although speech enhancement based on supervised learning has achieved significant performance improvements, it also has some obvious shortcomings: a large amount of paired pure speech and noisy speech data is required for training, and the labeling cost is high, especially when collecting data in a variety of noise environments. The model usually performs well in training data, but the performance may drop significantly in unseen noise types or new scenarios. In an environment with extremely low signal-to-noise ratio, the model may not be able to distinguish between noise and speech, resulting in partial removal of speech content. The noise and speech signals in the actual environment may change dynamically, but the training data of the supervised learning model is static, and it is difficult to flexibly adapt to these changes. Human speech perception not only relies on hearing, but also involves visual factors. The brain can combine auditory signals with visual information, especially the movement and shape of the lips. These visual clues are called visemes, which convey important information about the content of the language. The integration of multimodal information can be reflected by the McGurk effect. When we hear "ba" and see the mouth shape of "ga", we usually mishear it as "da". In a noisy environment, especially in a situation where many people are talking (i.e., the "cocktail party effect"), lip reading is crucial to improving speech comprehension. Visual information helps us distinguish words with similar pronunciations, especially in a noisy environment. Through multimodal methods, it can promote the enhancement of speech intelligibility. Even in a clear speech environment, visual information can help identify phonemes with similar acoustic features, further improving speech intelligibility. To this end, a multimodal speech enhancement algorithm based on semi-supervised learning pre-training is proposed. Summary of the invention

[0003] In view of this, the present invention provides a multimodal speech enhancement algorithm based on semi-supervised learning pre-training to solve or alleviate the technical problems of high annotation cost and low speech enhancement intelligibility in the prior art, and at least provides a beneficial option.

[0004] The technical solution of the present invention is implemented as follows: a multimodal speech enhancement algorithm based on semi-supervised learning pre-training comprises the following steps: S1, extract the noisy speech feature vector As the input of noisy speech GCRBM; S2, using traditional spectral subtraction to detect noisy speech Denoise to get denoised speech ; S3, the estimated denoised speech feature vector As the input of clean speech GCRBM; S4, extract the video signal feature vector As input to the video GCRBM; S5. Optimizing the parameters of the multimodal speech enhancement network; S6. During testing, the noisy speech features are input into the multimodal speech enhancement network.

[0005] Further preferably, in said S1, the noisy speech feature vector ,in is the frame number, is the sample sequence, Indicates that the sample comes from the unlabeled training set.

[0006] Further preferably, in the CD method, by maximizing the likelihood function Estimating parameters of GCRBM for noisy speech , and , for noisy speech GCRBM, for is the connection matrix between the input layer and the hidden layer, for The connection matrix from the input layer to the hidden layer at for When the input layer and The connection matrix of the input layer when .

[0007] More preferably,

[0008] In the formula is the hidden layer sample of GCRBM for noisy speech.

[0009] Further preferably, in S3, in the CD method, by maximizing the likelihood function Estimating clean speech GCRBM parameters , and , for pure speech GCRBM, for is the connection matrix between the input layer and the hidden layer, for The connection matrix from the input layer to the hidden layer at for When the input layer and The connection matrix of the input layer when .

[0010] More preferably,

[0011] In the formula is the hidden layer sample of pure speech GCRBM.

[0012] Further preferably, in S4, for the video GCRBM, by maximizing the likelihood function in the CD method Estimating parameters of video GCRBM , and , for video GCRBM, for is the connection matrix between the input layer and the hidden layer, for The connection matrix from the input layer to the hidden layer at for When the input layer and The connection matrix of the input layer when .

[0013] More preferably,

[0014] In the formula is the hidden layer sample of video GCRBM.

[0015] Further preferably, in said S5, after pre-training, the noisy speech and the corresponding video sequence Input into noisy speech GCRBM and video GCRBM respectively, and minimize the following objective function: (1) The following parameters of the multimodal speech enhancement network are optimized , and , , , , and and , where Indicates noisy speech The corresponding pure voice, Indicates that the sample comes from the labeled training set, Indicates the sample number, Indicates a frame signal.

[0016] Further preferably, in said S6, the proposed noisy speech feature vector Input the multimodal speech enhancement network to obtain the feature vector of the estimated speech , Indicates that the sample comes from the test set, and finally the feature vector Transform to spectrogram.

[0017] The embodiment of the present invention has the following advantages due to the adoption of the above technical solution: First, the present invention extracts speech features and mouth features, inputs them into a multimodal speech enhancement network, and after optimizing the parameters of the multimodal speech enhancement network, inputs the noisy speech features into the multimodal speech enhancement network to obtain an enhanced speech signal, thereby reducing the speech annotation cost and improving the intelligibility of speech enhancement; 2. After extracting the mouth features, the present invention compares the color histograms of adjacent frames. When the difference exceeds a preset threshold, it can be determined as a shot switch. Face detection based on HOG features is performed on each frame of the image to extract possible face areas. In order to track consistency in the time dimension, a KLT tracker is used to correlate the face detection results across frames to ensure that the facial position of the same person can remain consistent between different frames. Based on face positioning, a regression tree ensemble method is used to extract key points from a sparse subset of pixel intensities in the image, especially to accurately locate the speaker's mouth.

[0018] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present invention will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A flowchart of the steps of the multimodal speech enhancement algorithm of the present invention; Figure 2 It is a multimodal speech enhancement network diagram in the multimodal speech enhancement algorithm of the present invention. DETAILED DESCRIPTION

[0021] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present invention. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0022] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0023] like Figure 1 and 2 As shown, the embodiment of the present invention provides a multimodal speech enhancement algorithm based on semi-supervised learning pre-training, comprising the following steps: S1, extract the noisy speech feature vector As the input of noisy speech GCRBM; S2, using traditional spectral subtraction to detect noisy speech Denoise to get denoised speech ; S3, the estimated denoised speech feature vector As the input of clean speech GCRBM; S4, extract the video signal feature vector As input to the video GCRBM; S5. Optimizing the parameters of the multimodal speech enhancement network; S6. During testing, the noisy speech features are input into the multimodal speech enhancement network; The specific implementation is as follows: (1) Given three sets: A, B, C, A and B are training sets, C is the test set, where A = {XA, ZA}, XA represents the noisy speech signal, ZA represents the corresponding video signal, B = {XB, ZB, YB}, XB represents the noisy speech signal, ZB represents the corresponding video signal, YB represents the clean speech corresponding to XB, C = {XC}, XC represents the noisy speech signal; (2) For the noisy speech in set XA, extract the 161-dimensional short-time Fourier transform (STFT) amplitude spectrum, and the number of feature bits is , the convolution kernel is used to perform convolution operation on the amplitude spectrum feature to obtain the convolution layer, and each convolution layer uses 16 convolution kernels of size , 32 in size , 64 in size , 128 in size , and 256 sizes are The convolution kernels of these convolution kernels have step sizes of (1, 2), so that five convolution layers can be obtained, and the feature dimensions are , , , , , and then reshape the features of the last layer to obtain the feature dimension The eigenvector of , is the duration of noisy speech; (3) If Figure 2 In the multimodal speech enhancement network shown in dimensional noisy speech feature vector Input noisy speech into GCRBM, by maximizing the likelihood function Estimating parameters of GCRBM for noisy speech , and ; (4) Using spectral subtraction on the noisy speech XA, we obtain the estimated speech YA after denoising. (5) Extract the 161-dimensional short-time Fourier transform (STFT) amplitude spectrum for the estimated speech YA, and the feature dimension is , the convolution kernel is used to perform convolution operation on the amplitude spectrum feature to obtain the convolution layer, and each convolution layer uses 16 convolution kernels of size , 32 in size , 64 in size , 128 in size , and 256 sizes are The convolution kernels of these convolution kernels have step sizes of (1, 2), so that five convolution layers can be obtained, and the feature dimensions are , , , , , and then reshape the features of the last layer, and the feature dimension is , and recorded as ; 6) If Figure 2 In the multimodal speech enhancement network shown in Noisy Speech Input into the clean speech GCRBM, by maximizing the likelihood function Estimating parameters of clean speech GCRBM , and ; (7) For the video sequence ZA, the shot boundary is detected by calculating the color histogram difference between consecutive frames. Specifically, by comparing the color histograms of adjacent frames, when the difference exceeds a preset threshold, it can be determined as a shot switch. Then, face detection based on HOG (Histogram of Oriented Gradients) features is performed on each frame to extract possible face areas. In order to track consistently in the temporal dimension, the KLT tracker is used to correlate the face detection results across frames to ensure that the facial position of the same person can remain consistent between different frames. Based on face localization, the regression tree ensemble method is used to extract key points from a sparse subset of pixel intensities in the image, especially to accurately locate the speaker's mouth. (8) The mouth image captured in step (1) is input into Resnet34 after audio and video synchronization to obtain a 1000-dimensional feature vector ; (9) Video signal As the input of the video GCRBM, the input of the GCRBM is obtained by maximizing the likelihood function Estimating parameters of video GCRBM , and , where the hidden layer dimensions of noisy speech GCRBM, clean speech GCRBM and video GCRBM are all 800; (10) Apply the method in step (5) to XB and YB to obtain The eigenvectors of and ; (11) Apply the method of steps (7) and (8) to ZB to obtain the video sequence vector ; (12) During training, the noisy speech and the corresponding video sequence The noisy speech GCRBM and video GCRBM in the multimodal speech enhancement network are input respectively, and the following objective function is minimized: (2) After optimization, the parameters are obtained and , , , , and , , and , , The optimization algorithm uses the Adam optimizer, the initial learning rate is set to 0.003, and the weight decay is set to 0.01. The feature vector corresponding to the pure speech ; (13) During the test, the feature vector is extracted from XC using the method in step (5) , and enter into Figure 1 The multimodal speech enhancement network of , Indicates that the sample comes from the test set; (14) Yes Deconvolution operation is used, and each deconvolution layer uses 128 sizes of , 64 in size , 32 in size , 16 sizes are , and 1 size The convolution kernels of these convolution kernels have step sizes of (1, 2), so that five deconvolution layers can be obtained, and the feature dimensions are , , , , , and then reshape the features of the last layer, and the feature dimension is ; (15) Yes Performing short-time inverse Fourier transform on the feature vector of can obtain the enhanced speech signal.

[0024] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of various changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multimodal speech enhancement algorithm based on semi-supervised learning pre-training, characterized in that: The following steps are involved: S1, extract the noisy speech feature vector As the input of noisy speech GCRBM; S2, using traditional spectral subtraction to detect noisy speech Denoise to get denoised speech ; S3, the estimated denoised speech feature vector As the input of clean speech GCRBM; S4, extract the video signal feature vector As input to the video GCRBM; S5. Optimizing the parameters of the multimodal speech enhancement network; S6. During testing, the noisy speech features are input into the multimodal speech enhancement network.

2. The multimodal speech enhancement algorithm based on semi-supervised learning pre-training according to claim 1, characterized in that: In S1, the noisy speech feature vector ,in is the frame number, is the sample sequence, Indicates that the sample comes from the unlabeled training set.

3. The multimodal speech enhancement algorithm based on semi-supervised learning pre-training according to claim 1, characterized in that: In the CD method, by maximizing the likelihood function Estimating parameters of GCRBM for noisy speech , and , for noisy speech GCRBM, for is the connection matrix between the input layer and the hidden layer, for The connection matrix from the input layer to the hidden layer at for When the input layer and The connection matrix of the input layer when .

4. The multimodal speech enhancement algorithm based on semi-supervised learning pre-training according to claim 3, characterized in that: ； In the formula is the hidden layer sample of GCRBM for noisy speech.

5. The multimodal speech enhancement algorithm based on semi-supervised learning pre-training according to claim 1, characterized in that: In S3, in the CD method, by maximizing the likelihood function Estimating clean speech GCRBM parameters , and , for pure speech GCRBM, for is the connection matrix between the input layer and the hidden layer, for The connection matrix from the input layer to the hidden layer at for When the input layer and The connection matrix of the input layer when .

6. The multimodal speech enhancement algorithm based on semi-supervised learning pre-training according to claim 5, characterized in that: ； In the formula is the hidden layer sample of pure speech GCRBM.

7. The multimodal speech enhancement algorithm based on semi-supervised learning pre-training according to claim 1, characterized in that: In S4, for the video GCRBM, the CD method is used to maximize the likelihood function Estimating parameters of video GCRBM , and , for video GCRBM, for is the connection matrix between the input layer and the hidden layer, for The connection matrix from the input layer to the hidden layer at for When the input layer and The connection matrix of the input layer when .

8. The multimodal speech enhancement algorithm based on semi-supervised learning pre-training according to claim 7, characterized in that: ； In the formula is the hidden layer sample of video GCRBM.

9. The multimodal speech enhancement algorithm based on semi-supervised learning pre-training according to claim 1, characterized in that: In S5, after pre-training, the noisy speech and the corresponding video sequence Input into noisy speech GCRBM and video GCRBM respectively, and minimize the following objective function: （1） The following parameters of the multimodal speech enhancement network are optimized , and , , , , and , , and , , , where Indicates noisy speech The corresponding pure voice, Indicates that the sample comes from the labeled training set, Indicates the sample number, Indicates a frame signal.

10. The multimodal speech enhancement algorithm based on semi-supervised learning pre-training according to claim 1, characterized in that: In S6, the proposed noisy speech feature vector Input the multimodal speech enhancement network to obtain the feature vector of the estimated speech , Indicates that the sample comes from the test set, and finally the feature vector Transform to spectrogram.

Images