Multi-scale based multimodal time-domain speech separation method

Abstract

This multi-scale, multimodal time-domain speech separation method, involving the fields of artificial intelligence, audio signal processing, and computer vision, aims to provide analysis and assistance for mixed audio separation and enhancement. It includes the following steps: S1: data preprocessing, including mixed audio generation and lip image sequence extraction; S2: audio signal feature encoding; S3: lip region feature encoding and lip movement feature calculation; S4: multi-scale feature extraction of speaker features and lip movement features; S5: learning the detailed mapping between lip movement and speech; S6: fusion learning of relevant features such as audio, lip image sequences, lip movement, and speaker, and generation of an audio weight matrix; S7: treating steps S4-S6 as a speech separation module and repeating steps S4-S6; S8: separating the mixed audio according to the weight matrix and decoding the resulting audio signal. This method achieves end-to-end blind source speech separation from multimodal data, independent of the speaker.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence, audio signal processing, and computer vision, and in particular to a multi-scale multi-modal time-domain speech separation method for multi-modal speech separation and speech enhancement. Background Art

[0002] In a noisy environment, multiple sounds from multiple sound sources coexist, but not all sound source information needs to be paid attention to at the same time. The mutual interference of multiple sound sources will greatly reduce the amount of information people can receive. Therefore, separating different clean sound sources from mixed audio is a problem worth studying, namely the "cocktail party problem." For humans, it is relatively simple to distinguish and focus on the sound of a specific sound source in a mixed environment. However, for machines, how to quickly and accurately distinguish based on the sound characteristics of different sound sources is a challenge. The sound source information contained in sound source separation includes multiple categories, mainly speech, musical instrument sounds, environmental sounds, etc. Sound source separation is divided into different subtasks according to the different types of sound sources. Speech separation is one of the sub-fields of sound source separation.

[0003] In the past, speech separation focused on pure audio separation. However, with the development of the internet, multimedia support in smart devices, and the development of streaming media application platforms, video has become a primary component of information media. Video consists of image sequences and audio, so compared to audio, video data also contains visual information. For speech separation, pronunciation is closely related to lip movements and facial expressions during speech. Therefore, the inclusion of visual information facilitates speech separation and improves its accuracy.

[0004] The methods used in pure audio speech separation tasks generally include principal component analysis (ICA), beam-based, clustering, permutation invariant training (PIT) models, and the fusion of deep neural networks such as long short-term memory networks (LSTM), convolutional neural networks (CNN), and generative adversarial networks (GAN).

[0005] Audiovisual speech separation has garnered widespread attention since 2015. Researchers have combined visual and audio information into network models for training to achieve speech enhancement. Experimental results demonstrate the effectiveness of incorporating visual information into speech tasks, while also improving the accuracy of speech separation. With the rapid development of deep learning in recent years, deep neural network models have become the most widely used method for multimodal speech separation. Audio signals are processed separately for amplitude and phase, while lip image sequences are incorporated. Deep network models are then used to predict the amplitude and phase of the target signal. Some research combines speech separation with image reconstruction and facial and voice feature aggregation, using multi-task learning to improve speech separation.

[0006] However, current speech separation methods still face challenges. First, the inclusion of visual information increases network size and computational cost. Second, most current speech separation methods require prior knowledge of the number of speakers in the audio mixture, which determines the amount of audio that needs to be reconstructed at the network's output layer. Third, some current methods are highly speaker-dependent, requiring models to be trained for specific speakers or extracting features from pre-recorded audio of the speaker as prior knowledge, which significantly limits the model's generalization capabilities. Summary of the Invention

[0007] The purpose of this invention is to address the problem of speech separation by providing a multi-scale, multimodal, time-domain speech separation method based on end-to-end audiovisual fusion that is independent of prior knowledge of speakers. This method employs a deep neural network to address the multimodal speech separation task, combining vision and hearing. Using video data as a carrier, the method assists in audio signal separation based on the correlation between lip movements and speech. Furthermore, without requiring pre-recorded speech from the relevant speakers in the mixed audio or knowing the number of speakers, the method achieves speaker-independent, end-to-end, audiovisual, and multimodal speech separation.

[0008] The present invention comprises the following steps:

[0009] 1) Data preprocessing, including mixed audio generation and lip region extraction from video image sequences;

[0010] 2) Encode the one-dimensional single-channel audio signal features to generate an audio feature vector;

[0011] 3) Encode the lip region features to generate a feature vector, and generate lip motion features based on the inter-frame differences of the lip image sequence features;

[0012] 4) Multi-scale feature extraction of speaker characteristics and lip movement characteristics through pooling and skip connections, and speaker prediction based on speaker characteristics during training;

[0013] 5) Segment the video and learn the detailed mapping between lip movements and speech;

[0014] 6) Fusing audio features, lip image sequence features, speaker features, lip movement features, and the mapping relationship between lip movement and speech with feature learning to generate a weight matrix;

[0015] 7) Consider steps 4) to 6) as a speech separation module, repeat steps 4) to 6), and use the weight matrix output by the previous module as the input of the next module to update the audio features;

[0016] 8) Separate the corresponding parts from the mixed audio according to the weight matrix, and decode and restore the separated audio signal.

[0017] In step 1), the data used for generating the mixed audio is a training set of 800 randomly selected video clips of different people from the public dataset VoxCeleb2 (Chung JS, Nagrani A, Zisserman A. Voxceleb2: Deepspeaker recognition [J]. arXiv preprint arXiv: 1806.05622, 2018). Each speaker has a unique ID, the video frame rate is 25 fps, and the audio sampling rate is 16 kHz. Two audio segments are randomly selected and mixed at a random signal-to-noise ratio of -10 dB to 10 dB to generate mixed audio data.

[0018] The lip region is extracted by converting the video image into a grayscale image and scaling it to a size of 224*224, and taking the middle portion of 112*112 as the representation of the lip region.

[0019] In step 2), the one-dimensional single-channel audio signal feature encoding is performed. For the training audio data, the first n seconds are taken, with a total of 16000*n sampling points (n=3 in the experiment), and a 256-dimensional feature vector is generated by convolution. The experiment is conducted when the convolution kernel size L is 40 and the convolution step size is L / 2.

[0020] In step 3), the lip area feature encoding is performed by extracting features of the lip area using a pre-trained speech recognition task model (T. Afouras, JS Chung, A. Senior, O. Vinyals and A. Zisserman. Deep Audio-Visual Speech Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 12, pp. 8717-8727, 1 Dec. 2022, doi: 10.1109 / TPAMI.2018.2889052) to generate a 512-dimensional feature vector, which is then finally converted into a 256-channel feature representation through one-dimensional convolution;

[0021] The lip movement feature is calculated from the lip area feature. The size of the lip area feature calculated by the pre-trained speech recognition task model is 512*25n (where n represents the number of seconds of the video). The first frame feature of the lip area is retained, and the difference between the second frame feature and the first frame feature, the difference between the third frame feature and the second frame feature are calculated... and so on, until all inter-frame difference calculations of the entire video are completed, and all difference results are spliced ​​with the first frame feature in the second dimension to obtain the movement feature representation of the lip area.

[0022] In step 4), the speaker features include auditory information and visual information, and the speaker representation is obtained by fusing the audio features with the lip image sequence features;

[0023] The multi-scale feature extraction of speaker features and lip movement features is performed using two multi-layer downsampling network structures that do not share weight parameters to extract speaker features and lip movement features respectively. The network structure of each part is composed of multiple pooling modules in series. The multi-layer pooling continuously reduces the feature map to retain key information and weaken unimportant information. At the same time, skip connections are used to ensure information integrity and avoid information bias caused by pooling.

[0024] The result of speaker ID prediction based on speaker characteristics during the training process is used as part of the first stage training objective function to speed up the model fitting to the data.

[0025] In step 5), the detailed mapping between lip movements and speech is learned by segmenting the video to better learn the detailed information in the video clips. The video is divided into 3 small clips, that is, the corresponding audio features and lip movement features are divided into three segments, which are fused separately; the fused feature segments are subjected to multiple convolutions, batch normalization, and activation, and finally a feature weight matrix of 0 to 1 is obtained through the Sigmoid function, and different weights are assigned to different feature points in the segment to form an attention mechanism; finally, the fusion feature input by the module is multiplied by the weight matrix to obtain the final segment fusion feature.

[0026] In step 6), the learning of the fused features is to input the fused features into a temporal convolutional network and output a mask matrix with a threshold range of 0 to 1.

[0027] In step 7), steps 4) to 6) are regarded as a speech separation module, and steps 4) to 6) are repeated. The weight matrix output by the previous module is used as the input of the next module, and multiplied with the features of the original mixed audio as the audio feature representation in the current module.

[0028] In step 8), the corresponding parts are separated from the mixed audio according to the weight matrix, and the separated audio signals are restored by decoding. This is achieved by multiplying the mask matrix obtained in step 6) with the original mixed audio, thereby separating the individual audio data of a certain speaker, and reconstructing the audio signal through a decoder.

[0029] The principle of the present invention is described below:

[0030] T1. The method used in the present invention adopts a network structure of "encoder-separation network-decoder". First, the audio and lip image sequences are encoded respectively to generate feature vectors. The inter-frame differences are calculated according to the features of the lip image sequence to obtain the feature vectors of the lip movements. The audio features are fused with the features of the lip image sequence to obtain the speaker features. The speaker features, audio features, and lip movement features are input into the separation network. The mask matrix is ​​output through model inference and multiplied with the mixed audio to obtain the separated audio matrix. Finally, the decoder audio is reconstructed to obtain the separated audio signal.

[0031] T2. The original audio input in the present invention is represented by an audio signal in the time domain. In the Conv-TasNet method (Y.Luo and N.Mesgarani.Conv-TasNet:Surpassing Ideal Time–FrequencyMagnitude Masking for Speech Separation.IEEE / ACM Transactions on Audio, Speech, and Language Processing,vol.27,no.8,pp.1256-1266,Aug.2019,doi:10.1109 / TASLP.2019.2915167), experiments have shown that the experimental results of using time-domain audio signals as input are better than the method of representing audio with time-frequency masks in terms of both objective indicators and subjective evaluation.

[0032] T3, Speaker Feature Learning and Speaker Prediction Module: Multi-layer average pooling can be used to obtain global key features of speaker features, but this will lead to information loss. Therefore, multi-scale jump connections are used to obtain key information of feature maps at different scales to avoid information omission and loss.

[0033] T4, Inter-frame Lip Movement Variation Module: Since each utterance in a sentence is directly related to a lip movement, learning the mapping between speech and lip movement is crucial for better audio separation. Inter-frame movement representations are obtained by inter-frame differences in video image sequence features. This serves as input to the Inter-frame Movement Variation Module, which also uses multi-scale fusion based on average pooling for feature learning.

[0034] T5. Feature segmentation fusion module based on attention mechanism: Since the input data in the speaker representation module is the fusion feature of the complete speech and image sequence, the key information extracted during the average pooling process is the global mean. However, the longer the time interval in the segment, the lower the correlation. Therefore, the global mean cannot well represent local information, which will also greatly increase the information omission rate. Therefore, considering segmenting the video, it is planned to divide the complete video into three segments and fuse the audio features of the three segments with the lip movement features respectively. After extracting the fusion features through convolution and normalization, the fusion features are weighted by the Sigmoid activation function to form an attention weight matrix. The segmented fusion features are then multiplied by the weight matrix to obtain the segmented fusion features.

[0035] Compared with the prior art, the present invention has the following outstanding advantages:

[0036] 1. Most existing speech separation methods require prior knowledge of the number of speakers in the mixed audio, which determines the number of audio signals that need to be reconstructed at the network's output layer. This method utilizes the concept of speech enhancement to achieve speech separation. This method treats the audio related to non-target speakers in the mixed audio as noise, then separates the target speaker's speech by denoising the mixed audio. The method outputs one audio signal at a time through inference. Therefore, the method does not require prior knowledge of the number of speakers in the mixed audio.

[0037] 2. Some existing speech separation methods are highly speaker-dependent, requiring models to be trained for specific speakers or extracting features from pre-recorded audio of the speaker as prior knowledge. This significantly limits the model's generalization capabilities. The method of the present invention can perform speech separation tasks on untrained audio data, demonstrating strong generalization capabilities.

[0038] 3. Regarding the data used, this invention uses time-domain waveform signals as audio representations, eliminating the need for conversion to a time-frequency spectrum through Short-Time Fourier Transformation or other transformations. During decoding, there is no need for inverse Short-Time Fourier Transformation or other methods on the generated separated audio spectrograms, thus achieving an end-to-end speech separation method. Furthermore, compared to traditional audio-only speech separation, this invention incorporates visual information as an aid, improving the quality and accuracy of speech separation. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is the overall architecture diagram of the model.

[0040] Figure 2 Schematic diagram of the network structure of the multi-scale feature extraction module for speaker features and lip movement features.

[0041] Figure 3 Schematic diagram of the network structure of the detail mapping learning module between lip movements and speech.

[0042] in, Indicates that two tensors are concatenated according to a certain dimension. represents the point-wise addition of two tensors, represents the Sigmoid function, Represents the point-wise multiplication of two tensors.

[0043] Figure 4 This is the changing trend of the loss function value on the validation set when using one-stage training and two-stage training.

[0044] Figure 5 Ablation experiment of the model. DETAILED DESCRIPTION

[0045] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described with reference to the following embodiments in conjunction with the accompanying drawings.

[0046] The embodiment of the present invention includes the following steps:

[0047] S1. Preprocessing the data, including mixed audio generation and lip region extraction in video image sequences;

[0048] S2. Perform feature encoding on the one-dimensional audio signal to generate a two-dimensional feature vector;

[0049] S3. Use the pre-trained model to encode the features of the lip area, generate a two-dimensional feature vector, and generate lip motion features based on the inter-frame differences of the lip image sequence features;

[0050] S4. Perform multi-scale feature extraction on speaker features and lip movement features through pooling and skip connections, and perform speaker prediction based on speaker features during training.

[0051] S5. Segment the video and learn the detailed mapping between lip movements and speech;

[0052] S6, fusing and learning the audio features, lip image sequence features, speaker features, lip movement features, and the mapping relationship between lip movement and speech to generate a weight matrix;

[0053] S7, treating steps S4 to S6 as a speech separation module, repeating steps S4 to S6, and using the weight matrix output by the previous module as the input of the next module to update the audio features;

[0054] S8. Separate the corresponding parts from the mixed audio according to the weight matrix, and perform decoding to restore the separated audio signals.

[0055] Figure 1 A schematic diagram of the overall architecture of the model of an embodiment of the present invention is given.

[0056] In step S1, the data used is from the public dataset VoxCeleb2. 800 randomly selected video clips of different people are used as training and validation sets. Each speaker has a unique ID. The video clips in the training and validation sets do not overlap. The video frame rate is 25fps, and the audio sampling rate is 16kHz. Two randomly selected audio clips are mixed at a random signal-to-noise ratio ranging from -10dB to 10dB to generate mixed audio data.

[0057] The lip region is extracted by converting the video image into a grayscale image and scaling it to a size of 224*224, and taking the middle portion of 112*112 as the representation of the lip region.

[0058] In step S2, the audio signal is feature encoded. For the training audio data, the first 3 seconds, a total of 48,000 sampling points, are taken, and a 256-dimensional feature vector is generated through convolution. Experiments are performed with a convolution kernel size L of 40 and a convolution step size of L / 2 to generate a feature map of (B, 256, 2399).

[0059] In step S3, the lip area is feature encoded and the pre-trained speech recognition task model is used to extract features from the lip area of ​​the original video to generate a 512-dimensional feature vector. Since the video frame rate is 25fps and the video length used is 3s, the first 75 feature points are taken and a 256-channel feature representation is finally generated through one-dimensional convolution. The feature size is (B, 256, 75);

[0060] The lip action feature is calculated from the lip area feature. The size of the lip area feature calculated by the pre-trained speech recognition task model is 512*75. The first frame feature of the lip area is retained, and the difference between the second frame feature and the first frame feature is calculated, and the difference between the third frame feature and the second frame feature is calculated... and so on, until all inter-frame difference calculations of the entire video are completed, and all difference results are spliced ​​with the first frame feature in the second dimension to obtain the action feature representation of the lip area, with a size of (B, 512, 75).

[0061] In step S4, the speaker features include auditory information and visual information. The speaker representation is obtained by fusing the audio features with the lip image sequence features, where the audio feature size is (B, 256, 2399). The lip image sequence features are expanded to (B, 256, 2399) and then fused with the audio features in the first dimension to obtain a speaker feature representation of size (B, 512, 2399).

[0062] Multi-scale feature extraction is performed on speaker features and lip movement features. Two multi-layer downsampling network structures that do not share weight parameters extract speaker features and lip movement features respectively. Figure 2 This is a diagram of the network structure of the multi-scale feature extraction module. Each part of the network structure consists of multiple pooling modules connected in series. Each module has the structure of Conv1D-BN-PReLU-AvgPool. Through multi-layer pooling, the feature map is continuously reduced to retain key information and weaken the unimportant information. At the same time, the output of each module is added to the output of the previous pooling modules. This skip connection method ensures information integrity and avoids information bias caused by pooling.

[0063] During the training process, the output results of the downsampling network for speaker features and lip movement features are fused point by point by adding them up. Then, based on the fused features, a linear transformation is used to map them to 800 IDs to represent the probability of each speaker. The ID corresponding to the maximum probability is used as the speaker prediction result, and it is checked against the real speaker ID corresponding to the video. This is used as part of the first stage training objective function to accelerate the model's fitting of the data.

[0064] Since speaker prediction is similar to a classification task, the cross entropy loss between the predicted result and the actual true result is calculated as an indication of how close the two are. The smaller the value, the closer the two are. Therefore, the cross entropy loss function is used as part of the objective function in the training process. The cross entropy loss is calculated as follows:

[0065]

[0066] Where B represents the number of samples in the training process, s corresponds to the 800 speaker IDs in the training set and the validation set respectively, and T is Indicates whether the real speaker corresponding to the i-th sample is s. If so, the value is 1, otherwise 0. is It represents the probability that the predicted result of the i-th sample is speaker s.

[0067] In step S5, in order to better learn the detailed information in the video clip, the video is segmented to learn the detailed mapping between lip movements and speech. Figure 3 A schematic diagram of the network structure of the detailed mapping learning module between lip movements and speech is given. First, the audio features and lip movement features are expanded to 2400 in the second dimension and then divided into three segments. For each segment, the first dimension, i.e., the feature channel, is fused to obtain a fused feature vector of (B, 512, 400). The fused feature segments are subjected to multiple one-dimensional convolutions, batch normalization, and PReLU activations. Finally, a 0-1 feature weight matrix is ​​obtained using the Sigmoid function. Different weights are assigned to different feature points in the segment to form an attention mechanism. Finally, the fused features input to the module are multiplied by the weight matrix to obtain the final segment fused features. The number of channels of the final fused features is then reduced to 256 through convolution, and the segment fused features are spliced ​​according to the time dimension.

[0068] In step S6, since the speaker features contain the visual information of audio features and lip image sequence features, and the segment fusion features also contain the visual information of audio features and lip movement features, the segment fusion features and the speaker features are fused by point-by-point addition. At the same time, they are fused and spliced ​​with the predicted audio mask results of the previous module and the original lip image sequence features in the first dimension.

[0069] Learning fused features involves inputting the fused features into a temporal convolutional network, which outputs a mask matrix with a threshold range of 0 to 1. The temporal convolutional network consists of multiple temporal modules: Conv1D(1x1)-PReLU-gLN-Conv1D(Depthwise)-PReLU-gLN-Conv1D(1x1). Conv1D(Depthwise) convolutions use channel-based dilated convolutions, where each output channel is convolved on only one input channel, and the dilation coefficient of the dilated convolution increases by a factor of 2 with the index of the temporal module. gLN represents global layer normalization.

[0070] In step S7, steps S4-S6 are considered as a speech separation module. Steps S4-S6 are repeated, and the weight matrix output by the previous module is used as the input of the next module. This matrix is ​​multiplied by the features of the original mixed audio and then used as the audio feature representation in the current module. For the first speech separation module, the original features of the mixed audio are used as input.

[0071] In step S8, the speech separation is performed by multiplying the mask matrix obtained in S6 with the original mixed audio to separate the individual audio data of one of the speakers, and reconstructing the audio signal through the decoder. The similarity between the separated audio signal and the real clean audio is measured by calculating the scale-invariant signal-to-noise ratio (SI-SNR) (Y.Luo and N.Mesgarani.Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation.IEEE / ACM Transactions on Audio, Speech, and LanguageProcessing, vol.27, no.8, pp.1256-1266, Aug.2019, doi: 10.1109 / TASLP.2019.2915167) between the two in the time domain. The larger the value, the more similar the two are, and the opposite value is used as part of the objective function. The calculation method of SI-SNR is as follows:

[0072]

[0073]

[0074]

[0075] Among them, S represents the real clean speech, represents the separated speech of a certain speaker. The objective function in the training process is defined as follows:

[0076] L SISNR = -SISNR, (5)

[0077] L=αL sISNR +βL CrossEntropy , (6)

[0078] In this example, Adam was used as the optimizer for 100 training rounds. The initial learning rate ε = 0.001, and the loss function was used as the metric in the validation set. When this value did not decrease for three consecutive rounds, the learning rate was reduced by half, that is, ε = 0.5ε. When the metric -SISNR did not decrease for six consecutive rounds, the model had converged and training was terminated early.

[0079] The training process is carried out in two stages. In the first stage, the loss function L = αL sISNR +βL CrossEntropv In the middle, α is 1, β is 0.1, and β is adjusted to 0 in the second stage. Figure 4 The changes in the loss function values ​​on the validation set when using a single-stage training approach versus a two-stage training approach are shown. Adding the cross-entropy loss of speaker prediction results to the first-stage loss function accelerates model convergence. However, in the middle and late stages of training, the limited number of speakers in the sample can lead to overfitting of the model to the speakers, which manifests as an increasing loss function value on the validation set with increasing iterations. Furthermore, since speaker prediction is not the primary task, the coefficient of the cross-entropy loss of speaker prediction results in the loss function is adjusted to 0 in the second stage to achieve better performance on the speech separation task. In this embodiment, the coefficient β in the loss function is adjusted to 0 after 40 training iterations.

[0080] Figure 5 The change in the loss function value on the test set at each iteration of the training process is shown. The loss function used on the test set is similar to that used on the training and validation sets, with coefficients α and β equal to 1 and 0, respectively. The figure shows the baseline of the MuSE model, module 1 represents the multi-scale speaker feature extraction module, module 2 represents the lip movement feature extraction module, and module 3 represents the mapping of speech and lip movement details.

[0081] In this embodiment, the signal-to-distortion ratio (SDR), signal-to-interference ratio (SIR), and signal-to-artifact ratio (SAR) are used (E. Vincent, R. Gribonval, and C. Fevotte. Performance measurement in blind audio source separation. IEEE Transactions on Audio, Speech, and Language Processing, vol. 14, no. 4, pp. 1462-1469, July 2006, doi: 10.1109 / TSA.2005.858005), which are defined as follows:

[0082]

[0083]

[0084]

[0085]

[0086]

[0087] Among them, S t Corresponding to the target audio part in the mixed audio, S m represents the real and clean target speech, Represents the separated target speaker’s speech; E i Corresponding to the non-target audio part in the mixed audio, similarly, S n Other clean audio representing non-target speakers, represents the audio of the separated non-target speaker; E n Corresponding to the noise part in the mixed audio, including the random noise from -10dB to 10dB randomly added when generating the mixed audio and other noise during the device acquisition process; E a The corresponding systematic errors and distortions are generated during the calculation process. The larger the values ​​of the three indicators, the better the speech separation effect.

[0088] In addition, perceptual evaluation of speech quality (PESQ) (Rix, AW, Beerends, JG, Hollier, MP, and Hekstra, AP. Perceptual evaluation of speech quality (PESQ)-a new methodfor speech quality assessment of telephone networks and codec.2001 IEEE international conference on acoustics, speech, and signal processing. Proceedings (Cat. No.01CH37221). IEEE, 2001, 2:749-752) and short-time objective intelligibility (STOI) (CH Taal, RC Endriks, R. Heusdens and J. Jensen. A short-time objective intelligibility measure for time-frequency weighted noisy speech.2010 IEEE International Conference on Acoustics, Speech and Signal Processing, 2010, pp. 4214-4217, doi:10.1109 / ICASSP.2010.5495701) is an objective evaluation metric and a commonly used indicator for speech quality assessment. The perceptual assessment of speech quality has a threshold of -0.5 to 4.5 and objectively evaluates audio clarity, volume, latency, interference, and loss. Short-term objective intelligibility, with a threshold of 0 to 1, assesses the intelligibility of sentences in the audio. The magnitude of the results of both is directly proportional to the quality of the separation.

[0089] Table 1 gives the evaluation results of the speech separation effect of the model after adding different modules.

[0090] Table 1

[0091]

[0092] The baseline represents the MuSE model (Z. Pan, R. Tao, C. Xu and H. Li. Muse: Multi-Modal target speaker extraction with visual cues. ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 6678-6682, doi: 10.1109 / ICASSP39728.2021.9414023). Module 1 represents the multi-scale speaker feature extraction module, Module 2 represents the lip movement feature extraction module, and Module 3 represents the mapping of speech and lip movement details. As can be seen from the table, each module improves the results of the speech separation task.

Claims

1. A multi-scale multimodal time-domain speech separation method, characterized by The following steps are involved: 1) Data preprocessing, including mixed audio generation and lip region extraction from video image sequences; 2) Feature encoding of the one-dimensional single-channel audio signal to generate a feature vector; 3) Use the pre-trained model to encode the lip region to generate a feature vector, and generate lip motion features based on the inter-frame differences in the lip image sequence features; 4) Multi-scale feature extraction of speaker features and lip movement features is performed through pooling and skip connections, and speaker ID prediction is performed based on speaker features during training; 5) Segment the video to learn the detailed mapping between lip movements and speech; The detailed mapping between lip movements and speech is learned by segmenting the video to better learn the detailed information in the video clips. The video is divided into three small clips, that is, the corresponding audio features and lip movement features are divided into three segments and fused separately; the fused feature segments are subjected to multiple convolutions, batch normalization, and activation, and finally a feature weight matrix of 0-1 is obtained through the Sigmoid function. Different weights are assigned to different feature points in the segment to form an attention mechanism; finally, the fused features input by the module are multiplied by the weight matrix to obtain the final segment fusion features; 6) Fusing and learning the audio features, lip image sequence features, speaker features, lip movement features, and the mapping relationship between lip movement and speech to generate a weight matrix; 7) Consider steps 4) to 6) as a speech separation module, repeat steps 4) to 6), and use the weight matrix output by the previous module as the input of the next module to update the audio features; 8) Separate the corresponding parts from the mixed audio according to the weight matrix, and decode and restore them to obtain the separated audio signal.

2. The multi-scale multimodal time-domain speech separation method according to claim 1, characterized in that In step 1), the data used for generating the mixed audio is a training set of 800 randomly selected video clips of different people from the public dataset VoxCeleb2. Each speaker has a unique ID, the video frame rate is 25fps, and the audio sampling rate is 16kHz. Two audio clips are randomly selected and mixed at a random signal-to-noise ratio of -10dB to 10dB to generate mixed audio data. The lip region is extracted by converting the video image into a grayscale image and scaling it to a size of 224*224, and taking the middle portion of 112*112 as the representation of the lip region.

3. The multi-scale multimodal time-domain speech separation method according to claim 1, characterized in that In step 2), the audio signal is feature encoded. For the training audio data, the first n seconds are taken, with a total of 16000*n sampling points, and a 256-dimensional feature vector is generated through convolution.

4. The multi-scale multimodal time-domain speech separation method according to claim 1, characterized in that In step 3), the lip area is feature encoded, and features of the lip area are extracted using a pre-trained speech recognition task model to generate a 512-dimensional feature vector, which is then finally converted into a 256-channel feature representation through one-dimensional convolution. The lip movement feature is calculated from the lip area feature. The size of the lip area feature calculated by the pre-trained speech recognition task model is 512*25n, where n represents the number of seconds of the video. The first frame feature of the lip area is retained, and the difference between the second frame feature and the first frame feature, the difference between the third frame feature and the second frame feature are calculated... and so on, until all inter-frame difference calculations of the entire video are completed, and all difference results are spliced ​​with the first frame feature in the second dimension to obtain the movement feature representation of the lip area.

5. The multi-scale multimodal time-domain speech separation method according to claim 1, characterized in that In step 4), the speaker features include auditory information and visual information, and the speaker representation is obtained by fusing the audio features with the lip image sequence features; The multi-scale feature extraction of speaker features and lip movement features is performed by two multi-layer downsampling network structures that do not share weight parameters to extract speaker features and lip movement features respectively. The network structure of each part is composed of multiple pooling modules connected in series. Multi-layer pooling is used to continuously reduce the feature map to retain key information and weaken unimportant information. At the same time, jump connections are used to ensure information integrity and avoid information bias caused by pooling.

6. The multi-scale multimodal time-domain speech separation method according to claim 1, characterized in that In step 4), the result of speaker ID prediction based on speaker features during the training process is used as part of the first-stage training objective function to accelerate the model's fitting of the data.

7. The multi-scale multimodal time-domain speech separation method according to claim 1, characterized in that In step 6), the fusion and feature learning is to input the fused features into the temporal convolutional network and output a mask matrix with a threshold range of 0 to 1.

8. The multi-scale multimodal time-domain speech separation method according to claim 1, characterized in that In step 7), steps 4) to 6) are regarded as a speech separation module, and steps 4) to 6) are repeated. The weight matrix output by the previous module is used as the input of the next module, and multiplied with the features of the original mixed audio as the audio feature representation in the current module.

9. The multi-scale multimodal time-domain speech separation method according to claim 1, characterized in that In step 8), the corresponding part is separated from the mixed audio according to the weight matrix, and the separated audio signal is restored by decoding. This is achieved by multiplying the mask matrix obtained in step 6) with the original mixed audio, thereby separating the individual audio data of a certain speaker, and reconstructing the audio signal through the decoder.

Images