A method, device and medium for separating multiple voices based on voiceprint features

Abstract

The present invention relates to a method, device and medium for multi-person speech separation based on voiceprint features, the method comprising: S1: obtaining the voiceprint feature X of the target speaker ref And use short-time Fourier transform to extract the spectrum feature X of the mixed audio mix ; S2: Spectral features X of mixed audio by splicing mix The target speaker's voiceprint feature X ref Get the spectral feature X′ of the reference voiceprint feature mix , and the spectrum feature X′ mix Input to the dilated convolution layer for capturing low-level audio features, and obtain the input feature X of the speech separation model input ; S3: Obtain the spectrum mask based on the speech separation model and combine it with the spectrum feature X of the mixed audio mix Multiplying the two, we obtain the predicted frequency spectrum of the target speaker's pure audio. By referring to the phase spectrum of the mixed audio and combining it with an inverse short-time Fourier transform, we obtain the predicted pure audio of the target speaker in the time domain. Compared with existing technologies, this invention has the advantage of high speech separation accuracy.

Description

Technical Field

[0001] The present invention relates to the field of intelligent speech separation, and in particular to a method, device and medium for multi-person speech separation based on voiceprint features. Background Art

[0002] Humans can selectively listen to multiple sound sources, but computers lack this ability. In everyday life, when our attention is focused on a conversation with a specific target or object, we often ignore conversations between other people or ambient noise. This is known as the cocktail party effect. The cocktail party effect typically occurs in two situations: 1. When a person's attention is focused on a single sound, such as the sound source of a movie; 2. When a person's auditory senses are stimulated by something, such as the roar of an explosion, which causes them to ignore other surrounding sounds.

[0003] In 1953, Cherry and E. Colin proposed the famous cocktail party problem: how to accurately track and identify the speech of a specific speaker in a space with multiple speakers speaking simultaneously and in the presence of other background noise. The cocktail party problem can also be understood as an auditory version of the figure-ground problem in computer vision, where the sound of interest is the figure and the other sounds are the background. Currently, the cocktail party problem faces two challenging issues:

[0004] (1) How to separate the target speech signal from the mixed speech signal?

[0005] (2) How to track and maintain attention on the target sound source and be able to switch attention between different sound sources?

[0006] In most cases, these two challenges interact with each other: tracking the target sound source can benefit from good speech separation, and speech separation can also benefit from tracking the target sound source. In fact, current research on solving the cocktail party problem mainly focuses on the first challenging problem, speech separation.

[0007] In real-world applications, voice interaction is typically one-to-one. This means that smart devices typically focus solely on the speech signal emitted by the target speaker and ignore other sources. Therefore, the fundamental goal of speech separation is to isolate the target speaker's speech signal from a mixed signal composed of multiple speakers. However, most existing deep neural network-based speech separation methods typically use only the spectral features of the mixed audio as model input, without considering the target speaker's other speech characteristics.

[0008] With the rise of multimodal machine learning methods, some scholars have proposed a speaker-independent joint audio-visual model to separate the target speech signal from the mixed speech signal. The visual features are used to track the target speaker in the scene. Although multimodal machine learning speech separation methods have achieved good results, multimodal methods require the simultaneous use of audio and visual information. In many real-world speech interaction applications, visual information is difficult to obtain. Therefore, multimodal machine learning speech separation methods have high requirements for the types of information available in the application scenario, and their application scope is not wide.

[0009] The technical problem to be solved by the present invention is to accurately separate a target speech signal from a mixed speech signal without requiring other information other than speech, such as vision. Summary of the Invention

[0010] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a multi-person speech separation method, device and medium based on voiceprint features. The method can accurately separate the target speech signal from the mixed speech signal based on the voiceprint features.

[0011] The purpose of the present invention can be achieved by the following technical solutions:

[0012] According to a first aspect of the present invention, a method for separating multi-person speech based on voiceprint features is provided, the method comprising the following steps:

[0013] Step S1: Voiceprint feature extraction, including obtaining the target speaker's voiceprint feature X ref And use short-time Fourier transform to extract the spectrum feature X of the mixed audio mix ;

[0014] Step S2: Voiceprint feature fusion, by splicing the spectral features X of the mixed audio mix The target speaker's voiceprint feature X ref Get the spectral feature X′ of the reference voiceprint feature mix , and the spectrum feature X′ mixInput to the dilated convolution layer for capturing low-level audio features, and obtain the input feature X of the speech separation model input ;

[0015] Step S3: Speech separation, by obtaining the spectrum mask based on the speech separation model and combining it with the spectrum feature X of the mixed audio mix By multiplying, the spectrum of the predicted pure audio of the target speaker is obtained; by referring to the phase spectrum of the mixed audio and combining it with the inverse short-time Fourier transform, the pure audio of the predicted target speaker in the time domain is obtained.

[0016] Preferably, the voiceprint feature X of the target speaker is obtained in step S1. ref Specifically, the reference audio of the target speaker is input into the voiceprint feature extractor to obtain the Mel-frequency cepstral coefficients MFCCs of the target speaker and use them as the voiceprint feature X of the target speaker. ref , specifically including the following steps:

[0017] Step S11: trimming silence segments from the reference audio and mixed audio of the target speaker simultaneously;

[0018] Step S12: Processing the reference audio with silence segments removed and the mixed audio with silence segments removed to ensure that the length of the reference audio is consistent with the length of the mixed audio;

[0019] Step S13: Extract Mel-frequency cepstral coefficients (MFCCs) from the reference audio with silence removed, and take the first P dimensions as the target speaker’s voiceprint feature X ref .

[0020] Preferably, step S12 is: if the length of the reference audio after removing the silence segment is less than the length of the mixed audio after removing the silence segment, the reference audio is cyclically spliced; if the length of the reference audio after removing the silence segment is greater than the length of the mixed audio after removing the silence segment, the reference audio is trimmed to ensure that the length of the reference audio is consistent with the length of the mixed audio; wherein the silence segment is a speech segment below 20db.

[0021] Preferably, in step S1, the frequency spectrum feature X of the mixed audio is extracted by short-time Fourier transform. mix The specific steps include:

[0022] Step S14: performing a short-time Fourier transform on the mixed audio with the silence segment removed using a window size of 256 and a frame shift of 64, and obtaining an amplitude spectrum and a phase spectrum of the mixed audio;

[0023] Step S15: Use the amplitude spectrum as the spectrum feature X of the mixed audio mix; The phase spectrum is used to restore the phase spectrum used by the separation model to predict the pure audio of the target speaker.

[0024] Preferably, the dilated convolutional layer in step S2 includes a convolutional neural network (CNN).

[0025] Preferably, the process of obtaining the spectrum mask of the speech model is specifically as follows: using the deep clustering model DPCL based on the input feature X input The embedded vector is obtained, and the K-Means algorithm is used to cluster the obtained embedded vector to obtain the spectrum mask.

[0026] Preferably, the spectrum mask is a binary spectrum mask, ie each time-frequency bin in each spectrogram belongs to only one speaker.

[0027] Preferably, the deep clustering model DPCL includes a bidirectional long short-term memory network BiLSTM.

[0028] According to a second aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the method when executing the program.

[0029] According to a third aspect of the present invention, a computer-readable storage medium is provided on which a computer program is stored, and when the program is executed by a processor, the method described is implemented.

[0030] Compared with the prior art, the present invention has the following advantages:

[0031] 1) The multi-speech separation method based on voiceprint features proposed in this invention uses the target speaker's voiceprint features extracted from the target speaker's reference audio as a supplementary feature and splices it with the spectral features of the mixed audio. A deep neural network model is used to separate the target speaker's pure audio from the mixed audio, which to some extent improves the accuracy of the speech separation model's prediction of the target speaker's pure audio.

[0032] 2) Compared with the use of d-vector features, the feature extraction speed of the present invention using Mel-frequency cepstral coefficients (MFCC) is faster and more generalizable, because the features extracted by d-vectors largely depend on the selection of training data sets;

[0033] 3) Compared with the audio-visual joint model that uses visual information, this type of model has higher requirements for visual information, and obtaining high-quality visual information in daily life is often more cumbersome than obtaining audio information. The present invention only extracts voiceprint features from the speech segment of the target speaker, which is more convenient and faster. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1This is a workflow diagram of the multi-person speech separation method based on voiceprint features of the present invention. DETAILED DESCRIPTION

[0035] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0036] like Figure 1 FIG. 1 is a flowchart of a method for separating multiple voices based on voiceprint features according to the present invention. The method includes the following steps:

[0037] Step S1: Input the reference audio of the target speaker into the voiceprint feature extractor to obtain the Mel frequency cepstral coefficient of the target speaker as the voiceprint feature X of the target speaker ref ; At the same time, short-time Fourier transform is applied to the mixed audio to extract the spectral features of the mixed audio X mix ;

[0038] Step S11: trimming the silent segments (less than 20dB) of the reference audio and the mixed audio of the target speaker simultaneously;

[0039] Step S12: If the length of the reference audio without the silence segment is less than the length of the mixed audio without the silence segment, the reference audio is cyclically spliced; if the length of the reference audio without the silence segment is greater than the length of the mixed audio without the silence segment, the reference audio is trimmed to ensure that the length of the reference audio is consistent with the length of the mixed audio;

[0040] Step S13: Use a window size of 256, a frame shift of 64, and a Mel triangle filter number of 40 to extract Mel frequency cepstral coefficients (MFCCs) from the reference audio after removing the silent segments, and take the first 13 dimensions as the voiceprint feature X of the target speaker. ref The Mel-frequency cepstral coefficient is a linear transformation of the logarithmic energy spectrum of the nonlinear Mel-scale based on the sound frequency. Under the Mel-scale, the human ear's perception of frequency presents a linear relationship. That is, when the Mel-frequency of the signal is a multiple relationship, then the difference in pitch perceived by the human ear is probably also a multiple relationship. The mapping relationship between the normal frequency scale f and the Mel-frequency scale Mel(f) is:

[0041]

[0042] Step S14: performing a short-time Fourier transform on the mixed audio with the silence segment removed using a window size of 256 and a frame shift of 64, and obtaining an amplitude spectrum and a phase spectrum of the mixed audio;

[0043] Step S15: Use the amplitude spectrum as the spectrum feature X of the mixed audio mix ; The phase spectrum is used to restore the phase spectrum used by the separation model to predict the pure audio of the target speaker;

[0044] Step S2: First, the spectrum feature X of the mixed audio mix Combined with the target speaker's voiceprint feature X ref Get the spectral feature X′ of the reference voiceprint feature mix ; Then X′ mix Input to the dilated convolutional layer used to capture low-level audio features to obtain the input feature X of the speech separation model input The dilated convolution layer is composed of an 8-layer convolutional neural network (CNN), and its parameter settings are shown in Table 1. The parameter settings are referenced from Google's Voice Filter: Target Voice Separation by Speaker-Conditioned Spectrogram Mask paper.

[0045] Table 1 Parameter settings of dilated convolutional layer

[0046]

[0047] Step S3: Multiplying the binary spectrum mask obtained by the DPCL model with the spectrum of the mixed audio to obtain the predicted spectrum of the pure audio of the target speaker;

[0048] Step S4: The predicted pure audio of the target speaker in the time domain can be restored by referring to the phase spectrum of the mixed audio and combining it with the inverse short-time Fourier transform.

[0049] The following further illustrates the multi-person speech separation method based on the voiceprint features of the reference target speaker shown in the present invention with reference to specific experimental data.

[0050] Experimental conditions and scoring criteria:

[0051] This paper uses the LibriSpeech dataset provided by Daniel Povey, which contains approximately 1000 hours of English speech read aloud at a sampling rate of 16000Hz. The proposed model is trained using the LibriSpeech dataset (train-clean-100), which contains approximately 100 hours of audio from 251 speakers (126 male and 125 female). The proposed model is tested using the LibriSpeech dataset (test-clean), which contains approximately 5 hours of audio from 40 speakers (20 male and 20 female). The mixed audio required for training is generated by fusing the clean audio. To ensure training reliability, the LibriSpeech audio is normalized using ffmpeg-normalize.

[0052] The present invention uses the signal-distortion ratio (SDR) to evaluate the performance of the model. The SDR reflects the similarity between the separated audio signal and the original audio signal. The calculation formula is as follows:

[0053]

[0054] Where s represents the target speech signal, Represents the output of the speech separation model. The signals are all time domain speech signals. The higher the similarity between the separated audio signal and the original audio signal, the higher the difference. The smaller it is, the larger the log(·) value, the larger the SDR, and the better the performance of the model.

[0055] Experimental results:

[0056] This paper uses three speech separation models, DPCL, PITNet, and the bare Voice Filter without the speaker recognition network, as the basic models, uses MFCCs and d-vector as the voiceprint features of the target speaker, and conducts comparative experiments using three methods of applying voiceprint features: direct concatenation (DC), dilated convolution followed by concatenation (EC), and concatenation followed by dilated convolution (CE).

[0057] Table 2 SDR comparison results before and after reference voiceprint features

[0058]

[0059] (1) Horizontal comparison: ① Regardless of which voiceprint feature is applied, and regardless of which proposed method of applying voiceprint features is adopted, the performance of the speech separation model can be significantly improved. ② The performance of the speech separation model using MFCCs as the voiceprint feature of the target speaker is better than that of the speech separation model using d-vector as the voiceprint feature of the target speaker. MFCCs extracts the voiceprint features of the target speaker from the perspective of traditional signal processing. The information obtained is relatively primitive and the amount of information is more complete. However, the voiceprint features extracted by d-vector are largely dependent on the training data set. Therefore, the MFCCs voiceprint features are more universal and the separation results are better.

[0060] (2) Among the speech models using voiceprint features: ① The DPCL model using MFCCs voiceprint features with concatenated and dilated convolution (CE) performed best, with an SDR of 12.430, which was approximately 50% better than the DPCL model without voiceprint features. ② The Voice Filter model using d-vector voiceprint features with dilated convolution and concatenation (EC) performed worst, with an SDR of only 8.159.

[0061] (3) Among the speech separation models using MFCCs as the target speaker's voiceprint features: ① The DPCL model using concatenation followed by dilated convolution (CE) performed best, with an SDR of 12.430. ② The Voice Filter model using direct concatenation performed the worst, with an SDR of only 10.420, which was 2.01 lower than the SDR of the optimal DPCL. ③ Except for the DPCL model, there was no significant difference in the performance of the speech separation models using dilated convolution followed by concatenation (EC) and the speech separation models using concatenation followed by dilated convolution (CE). ④ The presence of the dilated convolution layer can slightly improve the performance of the speech separation model.

[0062] In summary, the DPCL model proposed in the present invention that uses the concatenated dilated convolution method (CE) and applies MFCCs voiceprint features performs best.

[0063] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0064] The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0065] Many components in a device are connected to the I / O interface, including: input units, such as a keyboard and mouse; output units, such as various types of displays and speakers; storage units, such as magnetic disks and optical disks; and communication units, such as network cards, modems, and wireless communication transceivers. The communication unit allows the device to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunication networks.

[0066] The processing unit performs the various methods and processes described above, such as methods S1 to S3. For example, in some embodiments, methods S1 to S3 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of methods S1 to S3 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S3 by any other appropriate means (e.g., by means of firmware).

[0067] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), and the like.

[0068] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0069] In the context of the present invention, machine-readable medium can be a tangible medium that can contain or store a program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0070] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A multi-speech separation method based on voiceprint features, characterized in that: The method comprises the following steps: Step S1: Voiceprint feature extraction, including obtaining the target speaker's voiceprint feature X ref And use short-time Fourier transform to extract the spectrum feature X of the mixed audio mix ; Step S2: Voiceprint feature fusion, by splicing the spectral features X of the mixed audio mix The target speaker's voiceprint feature X ref Get the spectral feature X′ of the reference voiceprint feature mix , and the spectrum feature X′ mix Input to the dilated convolution layer for capturing low-level audio features, and obtain the input feature X of the speech separation model input ; Step S3: Speech separation, by obtaining the spectrum mask based on the speech separation model and combining it with the spectrum feature X of the mixed audio mix Multiply them to obtain the spectrum of the predicted pure audio of the target speaker; by referring to the phase spectrum of the mixed audio and combining it with the inverse short-time Fourier transform, the predicted pure audio of the target speaker in the time domain is obtained; In step S1, the target speaker's voiceprint feature X is obtained. ref Specifically: Input the reference audio of the target speaker into the voiceprint feature extractor, obtain the Mel frequency cepstral coefficients MFCCs of the target speaker, and use it as the voiceprint feature X of the target speaker ref , specifically including the following steps: Step S11: trimming silence segments from the reference audio and mixed audio of the target speaker simultaneously; Step S12: Processing the reference audio with silence segments removed and the mixed audio with silence segments removed to ensure that the length of the reference audio is consistent with the length of the mixed audio; Step S13: Extract Mel-frequency cepstral coefficients (MFCCs) from the reference audio with silence removed, and take the first P dimensions as the target speaker’s voiceprint feature X ref ; The process of obtaining the spectrum mask of the speech model is specifically as follows: using the deep clustering model DPCL based on the input feature X input The embedded vector is obtained, and the K-Means algorithm is used to cluster the obtained embedded vector to obtain the spectrum mask.

2. The method for separating multiple voices based on voiceprint features according to claim 1, characterized in that: The step S12 is as follows: if the length of the reference audio after removing the silence segment is less than the length of the mixed audio after removing the silence segment, the reference audio is cyclically spliced; if the length of the reference audio after removing the silence segment is greater than the length of the mixed audio after removing the silence segment, the reference audio is trimmed to ensure that the length of the reference audio is consistent with the length of the mixed audio; wherein the silence segment is a speech segment with a sound level lower than 20dB.

3. The method for separating multiple voices based on voiceprint features according to claim 1, characterized in that: In step S1, the mixed audio is subjected to short-time Fourier transform to extract the spectrum feature X of the mixed audio. mix The specific steps include: Step S14: performing a short-time Fourier transform on the mixed audio with the silence segment removed using a window size of 256 and a frame shift of 64, and obtaining an amplitude spectrum and a phase spectrum of the mixed audio; Step S15: Use the amplitude spectrum as the spectrum feature X of the mixed audio mix ; The phase spectrum is used to restore the phase spectrum used by the separation model to predict the pure audio of the target speaker.

4. The method for separating multiple voices based on voiceprint features according to claim 1, characterized in that: The dilated convolutional layer in step S2 includes a convolutional neural network (CNN).

5. The method for separating multiple voices based on voiceprint features according to claim 1, characterized in that: The spectrum mask is a binary spectrum mask, that is, each time-frequency bin in each spectrogram belongs to only one speaker.

6. The method for separating multiple voices based on voiceprint features according to claim 1, characterized in that: The deep clustering model DPCL includes a bidirectional long short-term memory network BiLSTM.

7. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to claims 1 to 6 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images