A speech separation method, device and storage medium

Abstract

The embodiment of the present application discloses a speech separation method, device and storage medium. The method inputs mixed audio data and image data in the time domain into a first neural network for feature fusion, and outputs K first feature maps; inputs a spectrogram in the frequency domain into a second neural network for feature separation, and outputs K second feature maps; based on the K first feature maps and the K second feature maps, K spectrogram masks are obtained; finally, based on the K spectrogram masks and the spectrogram, K separated independent audio data are obtained. In this way, when performing speech separation, the first neural network is introduced for multi-perceptual feature extraction to enhance speech features, and K first feature maps are obtained. The second neural network is introduced to perform K component separation on the spectrogram of the mixed speech data, and K second feature maps are obtained. The use of the first feature map and the second feature map for spectrogram mask prediction can improve the prediction accuracy, thereby achieving effective separation of mixed audio data.

Description

Technical Field

[0001] The present application relates to audio and video signal processing technology, and in particular to a speech separation method, device and storage medium. Background Art

[0002] Currently, video streaming websites automatically generate subtitles for movies, television programs, short videos, and other videos based on automatic subtitle generation technology, and the generated subtitles are generally quite good. However, in reality, automatic subtitle generation technology often produces garbled characters in scenes with multiple speakers. To address this issue, it is necessary to separate the mixed sounds before generating subtitles. This involves multi-speaker speech separation, a classic problem in speech signal processing that requires continuous research and improvement. Summary of the Invention

[0003] To solve the above technical problems, the embodiments of the present application hope to provide a speech separation method, device and storage medium.

[0004] The technical solution of this application is achieved as follows:

[0005] In a first aspect, a speech separation method is provided, comprising:

[0006] Get the original video data;

[0007] The original video data is input into a speech separation model for speech separation processing to obtain K independent audio data; wherein K is a positive integer, and the speech separation processing includes:

[0008] extracting mixed audio data and image data from the original video data;

[0009] Inputting the mixed audio data and the image data into a first neural network for feature fusion, and outputting K first feature maps;

[0010] Performing short-time Fourier transform on the mixed audio data to obtain a spectrogram;

[0011] Inputting the spectrogram into a second neural network for feature separation, and outputting K second feature maps;

[0012] Obtaining K spectrogram masks based on the K first feature maps and the K second feature maps;

[0013] Based on the K spectrogram masks and the spectrogram, K separated independent audio data are obtained.

[0014] In the above solution, the feature fusion includes:

[0015] Converting and cropping images in the image data based on a preset image size to obtain multiple frames of first images;

[0016] Performing a three-dimensional convolution operation and a pooling operation on the multiple frames of first image to extract a third feature map corresponding to the image data;

[0017] Performing convolution and pooling operations on the mixed audio data to extract a fourth feature map corresponding to the mixed audio data;

[0018] Inputting the third feature map and the fourth feature map into the splicing subnet for channel-wise splicing to obtain a fifth feature map;

[0019] The fifth feature map is input into a three-dimensional residual subnetwork, and the K first feature maps are output.

[0020] In the above solution, the three-dimensional residual subnetwork includes: a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer and a time-space maximum pooling layer; wherein the sixth convolutional layer is a 3×3×3 convolutional layer whose output is a K channel feature map;

[0021] The fifth feature map passes through the first convolution layer, the second convolution layer, the third convolution layer, the fourth convolution layer, the fifth convolution layer, the sixth convolution layer and the time-space maximum pooling layer in sequence to obtain the K first feature maps.

[0022] In the above scheme, obtaining K spectrogram masks based on the K first feature maps and the K second feature maps includes: performing product operations on the K first feature maps and the K second feature maps according to channels to obtain K spectrogram masks.

[0023] In the above scheme, the separated K independent audio data are obtained based on the K spectrogram masks and the spectrogram, including: multiplying the K spectrogram masks with the spectrogram respectively to obtain K spectrograms; and performing short-time Fourier inverse transform on the K spectrograms to obtain the K independent audio data.

[0024] In the above solution, the speech separation process further includes: determining image position information of the K speakers in the image data based on the K spectrogram masks.

[0025] In the above scheme, the method also includes: obtaining N segments of video data from an audio-visual speech data set; wherein N is an integer greater than 1, and each segment of video data contains independent audio data and image data; according to a preset multi-person conversation scenario, at least two segments of the N segments of video data are mixed with audio and image to obtain video sample data; all video sample data are used to form a training sample set of the speech separation model; and the speech separation model is trained using the training sample set to obtain a trained speech separation model.

[0026] In the above solution, the loss function of the speech separation model is the pixel cross entropy loss of the predicted spectrogram mask and the true spectrogram mask obtained by the speech separation model.

[0027] In a second aspect, a speech separation device is provided, comprising:

[0028] an acquisition unit, configured to acquire original video data and input the original video data into a speech separation model;

[0029] A speech separation model is used to perform speech separation processing on the original video data to obtain K independent audio data; wherein K is a positive integer, and the speech separation processing includes:

[0030] extracting mixed audio data and image data from the original video data;

[0031] Inputting the mixed audio data and the image data into a first neural network for feature fusion, and outputting K first feature maps;

[0032] Performing short-time Fourier transform on the mixed audio data to obtain a spectrogram;

[0033] Inputting the spectrogram into a second neural network for feature separation, and outputting K second feature maps;

[0034] Obtaining K spectrogram masks based on the K first feature maps and the K second feature maps;

[0035] Based on the K spectrogram masks and the spectrogram, K separated independent audio data are obtained.

[0036] In a third aspect, a speech separation device is provided, the device comprising: a processor and a memory configured to store a computer program that can be run on the processor,

[0037] Wherein, the processor is configured to execute the steps of the aforementioned method when running the computer program.

[0038] Exemplarily, the above-mentioned device can be an electronic device with a speech separation function, or a chip applied to an electronic device. In the present application, the device can implement the functions of multiple units through software, hardware, or a combination of software and hardware, so that the device can perform the speech separation method provided in any of the first aspects above.

[0039] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program implements the steps of the aforementioned method when executed by a processor.

[0040] In an embodiment of the present application, a speech separation method, device and storage medium are provided. The method inputs the original video data into a speech separation model for speech separation processing. Specifically, the mixed audio data and image data in the time domain are input into a first neural network for feature fusion, and K first feature maps are output; the spectrogram in the frequency domain is input into a second neural network for feature separation, and K second feature maps are output; based on the K first feature maps and the K second feature maps, K spectrogram masks are obtained; finally, based on the K spectrogram masks and the spectrogram, K separated independent audio data are obtained. In this way, when performing speech separation, the first neural network is introduced for multi-perceptual feature extraction to enhance speech features, and K first feature maps are obtained. The second neural network is introduced to perform K component separation on the spectrogram of the mixed speech data to obtain K second feature maps. The use of the first feature map and the second feature map for spectrogram mask prediction can improve the prediction accuracy, thereby achieving effective separation of mixed audio data. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a schematic diagram of the first flow chart of the speech separation method in an embodiment of the present application;

[0042] Figure 2 Schematic diagram of the flow of the feature fusion method in the embodiment of the present application;

[0043] Figure 3 Schematic diagram of the structure of the UNET network in the embodiment of the present application;

[0044] Figure 4 A schematic diagram of the structure of a speech separation model in an embodiment of the present application;

[0045] Figure 5 This is a second flow chart of the speech separation method in an embodiment of the present application;

[0046] Figure 6 A spectrogram of mixed audio data in an embodiment of the present application;

[0047] Figure 7 This is the spectrogram of the first independent audio data in the embodiment of the present application;

[0048] Figure 8 This is a spectrogram of the second independent audio data in the embodiment of the present application;

[0049] Figure 9 This is a schematic diagram of the first speaker tracking and positioning in an embodiment of the present application;

[0050] Figure 10 This is a schematic diagram of the second speaker tracking and positioning in an embodiment of the present application;

[0051] Figure 11 Schematic diagram of the flow of the speech separation model construction and training method in the embodiment of the present application;

[0052] Figure 12 This is a schematic diagram of the first structural component of the speech separation device in an embodiment of the present application;

[0053] Figure 13 This is a schematic diagram of the second component structure of the speech separation device in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below with reference to the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0055] The mainstream approach to multi-speaker speech separation in the early days was computational auditory scene analysis. With the introduction of deep learning, deep learning-based speech separation methods have achieved promising results. This application embodiment provides a speech separation method to address the multi-speaker speech separation problem. The following examples illustrate the speech separation method.

[0056] Figure 1 This is a first flow chart of the speech separation method in the embodiment of the present application. Figure 1 As shown, the method may specifically include:

[0057] Step 101: Obtain original video data;

[0058] Here, the source of the original video data may be a video and audio acquisition device, a video database, or a video playback website.

[0059] Step 102: Input the original video data into the speech separation model for speech separation processing to obtain K independent audio data;

[0060] Here, the speech separation model has a mixed speech separation function. The speech separation model is an end-to-end model that can directly input original video data and output K separated independent audio data, where K is a positive integer.

[0061] Specifically, the speech separation process of the speech separation model includes:

[0062] Step 1021: extracting mixed audio data and image data from the original video data;

[0063] Step 1022: Input the mixed audio data and image data into a first neural network for feature fusion, and output K first feature maps;

[0064] Here, the first neural network is used for multi-sensory feature extraction. When extracting audio features, video image features are introduced to enhance speech features, so that the first feature map includes temporal features. The speech separation model that incorporates video image features can improve the performance of the speech separation algorithm.

[0065] For example, Figure 2 As shown, the feature fusion method may specifically include:

[0066] Step 201: converting and cropping images in the image data based on a preset image size to obtain multiple frames of first images;

[0067] Step 202: performing a three-dimensional convolution operation and a pooling operation on the multiple frames of the first image to extract a third feature map corresponding to the image data;

[0068] Step 203: performing convolution and pooling operations on the mixed audio data to extract a fourth feature map corresponding to the mixed audio data;

[0069] Step 204: Input the third feature map and the fourth feature map into the splicing subnet for channel-wise splicing to obtain a fifth feature map;

[0070] Step 205: Input the fifth feature map into the three-dimensional residual sub-network and output K first feature maps.

[0071] In practical applications, the three-dimensional residual subnetwork can be a ResNet network of different depths, for example, a ResNet network with a depth of 18, 34, 50, 101, or 152. The deeper the residual subnetwork, the stronger the network's feature extraction capability. Therefore, the residual subnetwork depth can be selected based on the speech separation performance requirements.

[0072] In some embodiments, the three-dimensional residual subnetwork includes: a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, and a time-space maximum pooling layer; wherein the sixth convolutional layer is a 3×3×3 convolutional layer whose output is a K channel feature map;

[0073] The fifth feature map passes through the first convolution layer, the second convolution layer, the third convolution layer, the fourth convolution layer, the fifth convolution layer, the sixth convolution layer and the time-space maximum pooling layer in sequence to obtain K first feature maps.

[0074] Table 1 shows the Resnet network with a depth of 18

[0075] Table 1

[0076]

[0077] It can be understood that the embodiment of the present application makes the following modifications to the existing Resnet network: the last average pooling layer and the fully connected layer are deleted, a 3×3×3 convolutional layer with K channel feature maps output is added, and a spatiotemporal max pooling operation layer is added.

[0078] Step 1023: Perform short-time Fourier transform on the mixed audio data to obtain a spectrogram;

[0079] A sonogram (or spectrogram) is created by applying a short-time Fourier transform to a speech signal. The spectrogram is plotted with time on the horizontal axis and frequency on the vertical axis, with color representing amplitude. Because it shows the changes in frequency and amplitude over time in a single graph, it is also called a "time-spectrogram."

[0080] In some embodiments, the mixed audio data is subjected to a short-time Fourier transform to obtain a spectrogram, including: downsampling the mixed audio data to obtain downsampled audio data; short-time Fourier transforming the downsampled audio data to obtain a spectrogram; and log-transforming the spectrogram to obtain a log-transformed spectrogram as input to the second neural network.

[0081] Exemplarily, before performing a short-time Fourier transform (SFT), the mixed speech data is downsampled, for example, to a sampling rate of 11 kHz and a mixed speech data length of approximately 6 seconds. Next, the mixed speech data is converted from the time domain to the frequency domain to obtain a spectrogram. For example, the time-to-frequency domain conversion is performed using a short-time Fourier transform (SIFT) using a Hamming window with a size of 1022 and a step size of 256. Furthermore, the spectrogram is log-transformed to achieve regularization. Finally, the log-transformed spectrogram is input into a second neural network for K-component separation.

[0082] Step 1024: Input the spectrogram into the second neural network for feature separation, and output K second feature maps;

[0083] Here, the second neural network is used to perform K-component separation on the spectrogram of the mixed speech data, the input is the spectrogram of the mixed speech data converted into the frequency domain, and the output is K second feature maps of the mixed speech data.

[0084] Exemplarily, the second neural network is a UNET network, which performs K-component separation on the spectrogram of mixed speech.

[0085] Figure 3 The schematic diagram of the structure of the UNET network in the embodiment of the present application is as follows: Figure 3 As shown, the UNET network 30 is a deep learning framework that generally includes an encoding process and a decoding process. The UNET network 30 includes: an input module, a downsampling module, an upsampling module, and an output module. The input module and the downsampling module implement the encoding process, and the upsampling module and the output module implement the decoding process. The spectrogram is input to the UNET network 30 for K component separation, and K second feature maps, namely S1, S2 to S K .

[0086] In practical applications, each layer of the UNET network 30 has a jump connection or a Squeeze and Excitation (SE) connection between the downsampling module and the upsampling module.

[0087] Step 1025: Obtain K spectrogram masks based on the K first feature maps and the K second feature maps;

[0088] Exemplarily, obtaining K spectrogram masks based on the K first feature maps and the K second feature maps includes: performing a product operation on the K first feature maps and the K second feature maps according to channels to obtain K spectrogram masks;

[0089] Specifically, the calculation formula for the sonogram / spectrogram mask is as follows:

[0090]

[0091] in, , is the first feature map, is the second feature map, refers to the sigmoid function, Filter invalid information to obtain valid information for threshold calculation, that is, obtain the spectrogram mask .

[0092] In practical applications, the spectrogram mask can be a binary mask or a ratio mask.

[0093] Step 1026: Based on the K spectrogram masks and the spectrogram, obtain K separated independent audio data.

[0094] In some embodiments, based on K spectrogram masks and spectrograms, K separated independent audio data are obtained, including: multiplying the K spectrogram masks with the spectrograms respectively to obtain K spectrograms; and performing short-time Fourier inverse transform on the K spectrograms to obtain K independent audio data.

[0095] For example, the conversion from frequency domain to time domain is done by Inverse Short-time Fourier Transform (ISIFT).

[0096] Any speech separation method provided in the embodiments of the present application can be applied to subtitle generation technology to extract corresponding text information from the separated K independent audio data; and add the text information to the image according to the speech time sequence.

[0097] Using the above-mentioned speech separation model, when performing speech separation, a first neural network is introduced for multi-perceptual feature extraction to enhance speech features, and K first feature maps are obtained. A second neural network is introduced to perform K-component separation on the spectrogram of the mixed speech data, and K second feature maps are obtained. Using the first feature map and the second feature map to perform spectrogram mask prediction can improve the prediction accuracy, thereby achieving effective separation of mixed audio data.

[0098] Based on the above embodiment, the speech separation model is illustrated. Figure 4 This is a structural diagram of a speech separation model in an embodiment of the present application. Figure 4 As shown, the speech separation model 40 can be divided into: a first neural network 401, a second neural network 402 and a product operation part 403 according to its function.

[0099] The first neural network 401 includes two input data channels, namely mixed audio data and image data. The image data is uniformly converted into 256*256 and randomly cropped to 224*224 size, and a series of 3D convolution operations and pooling operations are performed to reduce the temporal sampling rate. The mixed audio data is also subjected to convolution operations and pooling operations to keep the sampling rate of the voice data consistent with the sampling rate of the image data. The two can be spliced ​​by channel through the splicing layer.

[0100] For example, if the third feature map of image data is C1 × H × W, and the fourth feature map of mixed audio data is C2 × H × W, after channel-wise concatenation, the resulting fifth feature map is (C1 + C2) × H × W. Convolution and pooling are used to extract shallow features from both audio and image, and then feature fusion is performed.

[0101] The fifth feature map output by the splicing layer is input to the 3D residual sub-network for multi-feature perception. For example, the 3D residual sub-network obtains the first feature map of 256*256*K containing time information, which is recorded as , In this embodiment of the present application, the existing Resnet network is modified as follows: the last average pooling layer and the fully connected layer are deleted, a 3×3×3 convolutional layer with K channel feature maps as output is added, and a spatiotemporalmax pooling operation layer is added.

[0102] The second neural network 402 includes one input data channel, which downsamples the mixed speech data. For example, the sampling rate is 11kHz and the mixed speech length is about 6 seconds. Then, SIFT is performed on the mixed speech to obtain a spectrogram. Furthermore, the spectrogram is log-transformed to achieve regularization of the spectrogram. Finally, the log-transformed spectrogram is input into the second neural network 402 for K-component separation to obtain K second feature maps, which are recorded as , , the size is 256*256*K.

[0103] Exemplarily, the second neural network is a UNET network.

[0104] The product operation part 403 converts the first feature map output by the first neural network 401 into and the second feature map Perform product operations on each channel and predict K spectrogram masks. For example, the calculation formula of the spectrogram mask is as follows:

[0105]

[0106] Finally, a scalar product is performed on the K spectrogram masks and the spectrogram input by the second neural network 402, and then an ISFT is performed to obtain K separated independent audio data.

[0107] In some embodiments, the speech separation model is also capable of locating the pixel position of the speaker in the image. Figure 5 FIG. 1 is a second flow chart of the speech separation method according to an embodiment of the present application. Figure 5 As shown, the speech separation processing process of the speech separation module may specifically include:

[0108] Step 501: extracting mixed audio data and image data from original video data;

[0109] Step 502: Input the mixed audio data and image data into a first neural network for feature fusion, and output K first feature maps;

[0110] Here, the first neural network is used for multi-perceptual feature extraction to enhance speech features. When performing audio feature extraction, video image features are introduced. The speech separation model combined with video image features can improve the performance of the speech separation algorithm.

[0111] Step 503: Perform short-time Fourier transform on the mixed audio data to obtain a spectrogram;

[0112] In some embodiments, the mixed audio data is subjected to a short-time Fourier transform to obtain a spectrogram, including: downsampling the mixed audio data to obtain downsampled audio data; short-time Fourier transforming the downsampled audio data to obtain a spectrogram; and log-transforming the spectrogram to obtain a log-transformed spectrogram as input to the second neural network.

[0113] Step 504: Input the spectrogram into the second neural network for feature separation, and output K second feature maps;

[0114] Here, the second neural network is used to perform K-component separation on the spectrogram of the mixed speech data, the input is the spectrogram of the mixed speech data converted into the frequency domain, and the output is K second feature maps of the mixed speech data.

[0115] Exemplarily, the second neural network is a UNET network. The UNET network has a K-component separation function for the spectrogram of the mixed speech.

[0116] Step 505: Obtain K spectrogram masks based on the K first feature maps and the K second feature maps;

[0117] Exemplarily, obtaining K spectrogram masks based on the K first feature maps and the K second feature maps includes: performing channel-wise multiplication operations on the K first feature maps and the K second feature maps to obtain K spectrogram masks.

[0118] Step 506: obtaining K separated independent audio data based on the K spectrogram masks and the spectrogram;

[0119] In some embodiments, based on K spectrogram masks and spectrograms, K separated independent audio data are obtained, including: multiplying the K spectrogram masks with the spectrograms respectively to obtain K spectrograms; and performing short-time Fourier inverse transform on the K spectrograms to obtain K independent audio data.

[0120] Figure 6 This is a spectrogram of mixed audio data in an embodiment of the present application, where the mixed audio data is a mixture of two human voices. Figure 7 This is the spectrogram of the first independent audio data in the embodiment of the present application, Figure 8This is the spectrogram of the second independent audio data in the embodiment of this application.

[0121] Step 507: Determine image position information of the K speakers in the image data based on the K spectrogram masks.

[0122] For example, the spectrogram mask is multiplied by each frame of the image data to determine the image position information of the speaker in each frame of the image. The image position information is used to track and locate the position of the speaker in the image.

[0123] Figure 9 This is a schematic diagram of the first speaker tracking and positioning in the embodiment of this application. Figure 10 This is a schematic diagram of the second speaker tracking and positioning in an embodiment of this application. Figure 9 and Figure 10 The middle shaded area is the speaker’s image location information.

[0124] The above technical solution not only can effectively separate mixed audio data during speech separation, but also can locate and track the pixel position of the speaker in the video image, helping users to match the speech with the speaker.

[0125] Based on the above embodiment, the method further includes constructing and training a speech separation model. Figure 11 Schematic diagram of the process of constructing and training the speech separation model in the embodiment of the present application. Figure 11 As shown, the speech separation model construction and training method may specifically include:

[0126] Step 1101: constructing a speech separation model and a training sample set;

[0127] For example, Figure 4 Taking the first neural network, the second neural network, the product operation part, STFT and ISTFT as an example, an end-to-end speech separation model is constructed using the first neural network, the second neural network, the product operation part, the STFT and the ISTFT.

[0128] Exemplarily, in some embodiments, the method further includes: setting a K value of the speech separation model according to the number of speech separation requirements and the types of interfering audio; wherein the types of interfering audio include at least noise, silence and other non-human voice audio.

[0129] Here, the required number of speakers for speech separation refers to the maximum number of speakers that the speech separation model can separate. For example, if the required number of speakers for speech separation is 10 and the effects of noise, silence, and other non-human audio are taken into account, K can be set to 16.

[0130] Here, the training sample set may be one or more existing video datasets.

[0131] Exemplarily, in some embodiments, constructing a training sample set includes:

[0132] Obtain N video data segments from the audio-visual speech dataset; where N is an integer greater than 1, and each video data segment contains independent audio data and image data;

[0133] According to a preset multi-person conversation scenario, at least two video data segments from the N video data segments are mixed with audio and image to obtain video sample data;

[0134] All video sample data are used to form a training sample set.

[0135] The Audiovisual Speech Dataset (AVSpeech) contains approximately 4,700 hours of video segments, each 3-10 seconds long. These clips feature approximately 150,000 speakers, with clear, distraction-free speech. The speakers vary in age, gender, language, and video angle, and only one person appears in each video clip.

[0136] When constructing a training sample set, N videos can be selected from it, denoted as { }, where n={1,…,N}. Represents the image frame of the nth video segment; Represents the speech of the nth video; the mixed speech is generated according to the following formula, that is, linear mixing.

[0137]

[0138] For example, multi-person conversation scenarios are divided into several situations:

[0139] 1) The video image contains only one person, and the mixed voice contains two people, corresponding to the above formula N=2;

[0140] 2) There are two people in the video (synthesized), corresponding to their mixed speech, corresponding to the above formula N=2;

[0141] 3) There are three people in the video (synthesized), corresponding to their mixed speech, corresponding to the above formula N=3;

[0142] 4) There are five people (synthesized) in the video, corresponding to their mixed speech, corresponding to the above formula N=5;

[0143] Construct a training sample set containing the above four multi-person conversation scenarios, and calculate the true spectrogram mask of the spectrogram of the audio in the nth video segment , is to select whether the independent audio is the maximum value in each time-frequency unit of the mixed audio.

[0144] , for any m=(1,…,N). Here is the coordinate position in the spectrogram, and S represents the spectrogram.

[0145] Specifically, is the spectrogram of the independent audio in the nth video, indicating the , compare the same position of the matrices of the two spectrograms to achieve binary processing, If the value is the maximum, the position is 1, otherwise it is 0, so the true spectrogram mask of the independent audio in the nth video segment is , which can also be expressed as .

[0146] Step 1102: Train the speech separation model using the training sample set to obtain a trained speech separation model.

[0147] Exemplarily, the loss function of the speech separation model is the pixel intersection entropy loss of the predicted spectrogram mask and the true spectrogram mask obtained by the speech separation model.

[0148] That is, the loss function is

[0149] in, The spectrogram mask predicted by the speech separation model, is the true spectrogram mask.

[0150] This resulted in a new speech separation model that can effectively separate mixed audio data from video images containing multiple people speaking, generating independent audio data for each person. This model, when applied to subtitle generation, can improve subtitle accuracy and reduce the generation of garbled text.

[0151] In order to implement the method of the embodiment of the present application, based on the same inventive concept, the embodiment of the present application also provides a speech separation device, such as Figure 12 As shown, the device 120 includes:

[0152] An acquisition unit 1201 is configured to acquire original video data and input the original video data into a speech separation model;

[0153] The speech separation model 1202 is used to perform speech separation processing on the original video data to obtain K independent audio data. The speech separation processing includes:

[0154] extracting mixed audio data and image data from raw video data;

[0155] Inputting the mixed audio data and image data into the first neural network for feature fusion, and outputting K first feature maps;

[0156] Perform short-time Fourier transform on the mixed audio data to obtain a spectrogram;

[0157] Input the spectrogram into the second neural network for feature separation and output K second feature maps;

[0158] Obtaining K spectrogram masks based on the K first feature maps and the K second feature maps;

[0159] Based on the K spectrogram masks and the spectrogram, K separated independent audio data are obtained.

[0160] In some embodiments, feature fusion includes: converting and cropping images in the image data based on a preset image size to obtain multiple frames of first images; performing three-dimensional convolution operations and pooling operations on the multiple frames of first images to extract a third feature map corresponding to the image data; performing convolution operations and pooling operations on the mixed audio data to extract a fourth feature map corresponding to the mixed audio data; inputting the third feature map and the fourth feature map into the splicing subnetwork for channel-by-channel splicing to obtain a fifth feature map; inputting the fifth feature map into the three-dimensional residual subnetwork to output K first feature maps.

[0161] In some embodiments, the three-dimensional residual subnetwork includes: a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, and a time-space maximum pooling layer; wherein the sixth convolutional layer is a 3×3×3 convolutional layer whose output is a K channel feature map;

[0162] The fifth feature map passes through the first convolution layer, the second convolution layer, the third convolution layer, the fourth convolution layer, the fifth convolution layer, the sixth convolution layer and the time-space maximum pooling layer in sequence to obtain K first feature maps.

[0163] In some embodiments, obtaining K spectrogram masks based on the K first feature maps and the K second feature maps includes: performing channel-wise multiplication operations on the K first feature maps and the K second feature maps to obtain K spectrogram masks.

[0164] In some embodiments, performing a short-time Fourier transform on the mixed audio data to obtain a spectrogram includes:

[0165] Downsampling the mixed audio data to obtain downsampled audio data;

[0166] Perform short-time Fourier transform on the downsampled audio data to obtain a spectrogram;

[0167] Perform log transformation on the spectrogram to obtain a log transformed spectrogram as the input of the second neural network.

[0168] In some embodiments, the second neural network is a UNET network.

[0169] In some embodiments, based on the K spectrogram masks and the spectrogram, K separated independent audio data are obtained, including:

[0170] Multiply the K spectrogram masks with the spectrogram respectively to obtain K spectrograms;

[0171] Perform inverse short-time Fourier transform on the K spectrograms to obtain K independent audio data.

[0172] In some embodiments, the speech separation process further includes: determining image position information of K speakers in the image data based on the K spectrogram masks.

[0173] In some embodiments, the apparatus 120 further includes a construction unit ( Figure 12 ), used to construct a speech separation model and a training sample set; the speech separation model is trained using the training sample set until the loss function meets the convergence condition, thereby obtaining a trained speech separation model.

[0174] In some embodiments, a construction unit is used to obtain N segments of video data from an audio-visual speech dataset; wherein N is an integer greater than 1, and each segment of video data contains independent audio data and image data; according to a preset multi-person conversation scenario, at least two segments of the N segments of video data are mixed with audio and image to obtain video sample data; and all the video sample data are used to form a training sample set.

[0175] In some embodiments, the construction unit is also used to calculate the true spectrogram mask corresponding to the audio data in each video sample data; the loss function is the pixel cross entropy loss of the predicted spectrogram mask and the true spectrogram mask obtained by the speech separation model.

[0176] In some embodiments, the construction unit is further configured to set the K value of the speech separation model according to the number of speech separation requirements and the types of interfering audio; wherein the types of interfering audio include at least noise, silence, and other non-human voice audio.

[0177] The device 120 can be an electronic device with a speech separation function, or a chip applied to an electronic device. In the present application, the device can implement the functions of multiple units through software, hardware, or a combination of software and hardware, so that the device can perform the speech separation method provided in any of the above embodiments.

[0178] When using the above-mentioned speech separation device, a first neural network is introduced for multi-perceptual feature extraction to enhance speech features during speech separation, and K first feature maps are obtained. A second neural network is introduced to perform K component separation on the spectrogram of the mixed speech data, and K second feature maps are obtained. Using the first feature map and the second feature map to perform spectrogram mask prediction can improve the prediction accuracy, thereby achieving effective separation of mixed audio data.

[0179] Based on the hardware implementation of each unit in the above-mentioned speech separation device, the embodiment of the present application also provides another speech separation device, such as Figure 13 As shown, the apparatus 130 includes: a processor 1301 and a memory 1302 configured to store a computer program that can be run on the processor;

[0180] The processor 1301 is configured to execute the method steps in the aforementioned embodiment when running a computer program.

[0181] Of course, in actual application, Figure 13 As shown, the various components in the device 130 are coupled together via a bus system 1303. It is understood that the bus system 1303 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 1303 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 13 Various buses are labeled as bus system 1303.

[0182] In practical applications, the processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a controller, a microcontroller, and a microprocessor. It is understood that for different devices, the electronic components used to implement the functions of the processor may be other, and the embodiments of the present application do not specifically limit this.

[0183] The above-mentioned memory can be a volatile memory (volatile memory), such as random-access memory (RAM); or a non-volatile memory (non-volatile memory), such as read-only memory (ROM), flash memory, hard disk drive (HDD) or solid-state drive (SSD); or a combination of the above types of memory, and provides instructions and data to the processor.

[0184] In practical applications, the above-mentioned device can be an electronic device with a speech separation function, or a chip applied to an electronic device. In the present application, the device can implement the functions of multiple units through software, hardware, or a combination of software and hardware, so that the device can execute the speech separation method provided in any of the above embodiments. The technical effects of each technical solution of the device can refer to the technical effects of the corresponding technical solution in the speech separation method, and this application will not elaborate on them one by one.

[0185] In an exemplary embodiment, the present application also provides a computer-readable storage medium, such as a memory including a computer program, which can be executed by a processor of the speech separation device to complete the steps of the aforementioned method.

[0186] An embodiment of the present application also provides a computer program product, including computer program instructions.

[0187] Optionally, the computer program product can be applied to the speech separation device in the embodiments of the present application, and the computer program instructions enable the computer to execute the corresponding processes implemented by the speech separation device in the various methods of the embodiments of the present application. For the sake of brevity, they will not be repeated here.

[0188] The embodiment of the present application also provides a computer program.

[0189] Optionally, the computer program can be applied to the speech separation device in the embodiments of the present application. When the computer program runs on a computer, the computer executes the corresponding processes implemented by the speech separation device in the various methods of the embodiments of the present application. For the sake of brevity, they will not be repeated here.

[0190] It should be understood that the terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. The expressions "having", "can have", "including" and "comprising", or "can include" and "can include" in this application can be used to indicate the presence of corresponding features (e.g., elements such as numerical values, functions, operations or components), but do not exclude the presence of additional features.

[0191] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various types of information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another and are not necessarily used to describe a specific order or precedence. For example, first information could be referred to as second information, and similarly, second information could be referred to as first information without departing from the scope of the present invention.

[0192] The technical solutions described in the embodiments of this application can be combined arbitrarily unless there is any conflict.

[0193] In the several embodiments provided in this application, it should be understood that the disclosed methods, devices and equipment can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0194] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0195] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above-mentioned integrated units can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0196] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. A speech separation method, characterized in that: The method comprises: Get the original video data; The original video data is input into a speech separation model for speech separation processing to obtain K independent audio data; wherein K is a positive integer, and the speech separation processing includes: extracting mixed audio data and image data from the original video data; Inputting the mixed audio data and the image data into a first neural network for feature fusion, and outputting K first feature maps; Performing short-time Fourier transform on the mixed audio data to obtain a spectrogram; Inputting the spectrogram into a second neural network for feature separation, and outputting K second feature maps; Obtaining K spectrogram masks based on the K first feature maps and the K second feature maps; Obtaining K separated independent audio data based on the K spectrogram masks and the spectrogram; The feature fusion includes: converting and cropping images in the image data based on a preset image size to obtain multiple frames of first images; performing a three-dimensional convolution operation and a pooling operation on the multiple frames of first images to extract a third feature map corresponding to the image data; performing a convolution operation and a pooling operation on the mixed audio data to extract a fourth feature map corresponding to the mixed audio data; inputting the third feature map and the fourth feature map into a splicing subnet for channel-by-channel splicing to obtain a fifth feature map; inputting the fifth feature map into a three-dimensional residual subnet to output the K first feature maps; The obtaining K spectrogram masks based on the K first feature maps and the K second feature maps comprises: performing a product operation on the K first feature maps and the K second feature maps according to channels to obtain K spectrogram masks; The method of obtaining the separated K independent audio data based on the K spectrogram masks and the spectrogram comprises: multiplying the K spectrogram masks with the spectrogram respectively to obtain K spectrograms; and performing an inverse short-time Fourier transform on the K spectrograms to obtain the K independent audio data.

2. The method according to claim 1, characterized in that The three-dimensional residual subnetwork includes: a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer and a time-space maximum pooling layer; wherein the sixth convolutional layer is a 3×3×3 convolutional layer whose output is K channel feature maps; The fifth feature map passes through the first convolution layer, the second convolution layer, the third convolution layer, the fourth convolution layer, the fifth convolution layer, the sixth convolution layer and the time-space maximum pooling layer in sequence to obtain the K first feature maps.

3. The method according to claim 1, characterized in that The speech separation process further comprises: Based on the K spectrogram masks, image position information of K speakers in the image data is determined.

4. The method according to claim 1, wherein The method further comprises: Obtain N video data segments from the audio-visual speech dataset; where N is an integer greater than 1, and each video data segment contains independent audio data and image data; According to a preset multi-person conversation scenario, at least two video data segments of the N video data segments are mixed with audio and image to obtain video sample data; Utilizing all video sample data to form a training sample set for the speech separation model; The speech separation model is trained using the training sample set to obtain a trained speech separation model.

5. A speech separation device, characterized in that: The device comprises: an acquisition unit, configured to acquire original video data and input the original video data into a speech separation model; A speech separation model is used to perform speech separation processing on the original video data to obtain K independent audio data; wherein K is a positive integer, and the speech separation processing includes: extracting mixed audio data and image data from the original video data; Inputting the mixed audio data and the image data into a first neural network for feature fusion, and outputting K first feature maps; Performing short-time Fourier transform on the mixed audio data to obtain a spectrogram; Inputting the spectrogram into a second neural network for feature separation, and outputting K second feature maps; Obtaining K spectrogram masks based on the K first feature maps and the K second feature maps; Obtaining K separated independent audio data based on the K spectrogram masks and the spectrogram; The feature fusion includes: converting and cropping images in the image data based on a preset image size to obtain multiple frames of first images; performing a three-dimensional convolution operation and a pooling operation on the multiple frames of first images to extract a third feature map corresponding to the image data; performing a convolution operation and a pooling operation on the mixed audio data to extract a fourth feature map corresponding to the mixed audio data; inputting the third feature map and the fourth feature map into a splicing subnet for channel-by-channel splicing to obtain a fifth feature map; inputting the fifth feature map into a three-dimensional residual subnet to output the K first feature maps; The obtaining K spectrogram masks based on the K first feature maps and the K second feature maps comprises: performing a product operation on the K first feature maps and the K second feature maps according to channels to obtain K spectrogram masks; The method of obtaining the separated K independent audio data based on the K spectrogram masks and the spectrogram comprises: multiplying the K spectrogram masks with the spectrogram respectively to obtain K spectrograms; and performing an inverse short-time Fourier transform on the K spectrograms to obtain the K independent audio data.

6. A speech separation device, characterized in that: The apparatus comprises: a processor and a memory configured to store a computer program capable of being executed on the processor, Wherein, the processor is configured to execute the steps of the method according to any one of claims 1 to 4 when running the computer program.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are performed.

Images