Multi-speaker voice separation method and related device

Abstract

The invention discloses a multi-speaker voice separation method and a related device, and relates to the technical field of voice processing, and the method comprises the steps: obtaining multi-speaker voice signals collected by a two-microphone array for a first speaker and a second speaker located in different directions; the phase difference of the two microphones is determined according to the multi-speaker voice signals, fixed beam forming processing aiming at a first area and fixed beam forming processing aiming at a second area are carried out on the multi-speaker voice signals, the first area is the area where a first speaker is located, and the second area is the area where a second speaker is located; determining voice time-frequency masks of speakers in two different directions according to the phase difference and the beam forming signals corresponding to the two areas respectively; and separating the voice signal of the first speaker and the voice signal of the second speaker from the signal of any channel of the voice signals of the multiple speakers according to the determined voice time-frequency mask. The multi-speaker voice separation method disclosed by the invention has a relatively good separation effect.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a multi-speaker speech separation method and related devices. Background Art

[0002] In many scenarios, multi-speaker speech is generated. In order to facilitate subsequent data processing and analysis, multi-speaker speech separation is usually required, that is, separating the speech of different speakers from the multi-speaker speech.

[0003] Taking the enterprise management scenario as an example, enterprise employees usually need to wear ID cards. With the rapid development of science and technology, smart electronic ID cards, as a new type of management tool, have entered the management fields of more and more enterprises and are being popularized by more and more sales and service companies. Smart electronic ID cards are most commonly required to have a multi-speaker voice separation function, that is, the function of separating the voice of the person wearing the ID card from the voice of the customer from the recorded multi-speaker voice.

[0004] At present, most of the multi-speaker speech separation methods are based on voiceprint information, that is, voiceprint information is used to separate the speech of different speakers from the multi-speaker speech. However, voiceprint information has a weak ability to distinguish speakers of the same sex, which leads to certain application limitations of multi-speaker speech separation methods based on voiceprint information and cannot meet application requirements. Summary of the invention

[0005] In view of this, the present application provides a multi-speaker speech separation method and related devices to solve the problem that the current multi-speaker speech separation method based on voiceprint information has certain application limitations and cannot meet application requirements. The technical solution is as follows:

[0006] The first aspect of the present application provides a multi-speaker speech separation method, comprising:

[0007] Acquire multi-speaker speech signals collected by a microphone array consisting of two microphones arranged at different positions from a first speaker and a second speaker located at different positions;

[0008] Determining the phase difference between the two microphones according to the multi-speaker speech signals, and performing fixed beamforming processing for a first area and fixed beamforming processing for a second area on the multi-speaker speech signals, respectively, to obtain beamforming signals corresponding to the first area and the second area, respectively, wherein the first area is an area where the first speaker is located, and the second area is an area where the second speaker is located;

[0009] Determine a first speech time-frequency mask of the first speaker and a first speech time-frequency mask of the second speaker according to the phase difference and the beamforming signals corresponding to the first area and the second area respectively;

[0010] According to the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker, the speech signal of the first speaker and the speech signal of the second speaker are separated from the signal of any channel of the multi-speaker speech signal.

[0011] In a possible implementation, determining the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker according to the phase difference and the beamforming signals corresponding to the first area and the second area respectively includes:

[0012] Obtaining logarithmic power spectra of the beamforming signals corresponding to the first area and the second area respectively, to obtain two logarithmic power spectra;

[0013] Using the pre-trained spatial separation model, based on the phase difference and the two logarithmic power spectra, determining the speech time-frequency masks corresponding to the first area and the second area respectively, the speech time-frequency mask corresponding to the first area is used as the first speech time-frequency mask of the first speaker, and the speech time-frequency mask corresponding to the second area is used as the first speech time-frequency mask of the second speaker;

[0014] The spatial separation model is trained using training samples with real speech time-frequency masks marked with two different regions respectively corresponding to the training samples, wherein the training samples include microphone phase differences and two logarithmic power spectra determined based on training noisy mixed speech signals, and the training noisy mixed speech signals are noisy speech signals of multiple speakers in the two different regions.

[0015] In a possible implementation, the multi-speaker speech separation method further includes:

[0016] Clustering the multi-speaker speech signals by using a Gaussian mixture model to determine a second speech time-frequency mask of the first speaker and a second speech time-frequency mask of the second speaker;

[0017] The step of separating the speech signal of the first speaker and the speech signal of the second speaker from the signal of any channel of the multi-speaker speech signal according to the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker comprises:

[0018] According to the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker, and in combination with the second speech time-frequency mask of the first speaker and the second speech time-frequency mask of the second speaker, the speech signal of the first speaker and the speech signal of the second speaker are separated from the signal of any channel of the multi-speaker speech signal.

[0019] In a possible implementation, separating the speech signal of the first speaker and the speech signal of the second speaker from signals of any channel of the multi-speaker speech signal according to the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker, and combining the second speech time-frequency mask of the first speaker and the second speech time-frequency mask of the second speaker, includes:

[0020] fusing the first speech time-frequency mask of the first speaker with the second speech time-frequency mask of the first speaker to obtain the target speech time-frequency mask of the first speaker, and fusing the first speech time-frequency mask of the second speaker with the second speech time-frequency mask of the second speaker to obtain the target speech time-frequency mask of the second speaker;

[0021] According to the target speech time-frequency mask of the first speaker and the target speech time-frequency mask of the second speaker, the speech signal of the first speaker and the speech signal of the second speaker are separated from the signal of any channel of the multi-speaker speech signal.

[0022] In a possible implementation, the first speech time-frequency mask of the first speaker, the first speech time-frequency mask of the second speaker, the second speech time-frequency mask of the first speaker, and the second speech time-frequency mask of the second speaker are all frame-level speech time-frequency masks;

[0023] The step of fusing the first speech time-frequency mask of the first speaker with the second speech time-frequency mask of the first speaker to obtain the target speech time-frequency mask of the first speaker, and fusing the first speech time-frequency mask of the second speaker with the second speech time-frequency mask of the second speaker to obtain the target speech time-frequency mask of the second speaker comprises:

[0024] Taking the maximum value of the first speech time-frequency mask of the first speaker and the second speech time-frequency mask of the first speaker at the frame level to obtain the target speech time-frequency mask of the first speaker;

[0025] The maximum value of the first speech time-frequency mask of the second speaker and the second speech time-frequency mask of the second speaker is taken at the frame level to obtain the target speech time-frequency mask of the second speaker.

[0026] In a possible implementation, the target speech time-frequency mask of the first speaker and the target speech time-frequency mask of the second speaker are both frame-level speech time-frequency masks;

[0027] The step of separating the speech signal of the first speaker and the speech signal of the second speaker from the signal of any channel of the multi-speaker speech signal according to the target speech time-frequency mask of the first speaker and the target speech time-frequency mask of the second speaker comprises:

[0028] According to the target speech time-frequency mask of the first speaker, each frame signal of the multi-speaker speech signal is distinguished as a speech segment and a non-speech segment to obtain a first distinction result, and according to the target speech time-frequency mask of the second speaker, each frame signal of the multi-speaker speech signal is distinguished as a speech segment and a non-speech segment to obtain a second distinction result;

[0029] According to the first discrimination result and the second discrimination result, the speech signal of the first speaker and the speech signal of the second speaker are separated from the signal of any channel of the multi-speaker speech.

[0030] In a possible implementation, the distinguishing between speech segments and non-speech segments of each frame signal of the multi-speaker speech signal according to the target speech time-frequency mask of the first speaker includes:

[0031] For each frame signal of the multi-speaker speech signal, if the speech time-frequency mask corresponding to the frame signal in the target speech time-frequency mask of the first speaker is greater than a set threshold, the frame signal is determined to be a speech segment; otherwise, the frame signal is determined to be a non-speech segment;

[0032] The step of distinguishing speech segments and non-speech segments of each frame signal of the multi-speaker speech signal according to the target speech time-frequency mask of the second speaker includes:

[0033] For each frame signal of the multi-speaker speech signal, if the speech time-frequency mask corresponding to the frame signal in the target speech time-frequency mask of the second speaker is greater than a set threshold, the frame signal is determined to be a speech segment; otherwise, the frame signal is determined to be a non-speech segment.

[0034] In a possible implementation, the first discrimination result and the second discrimination result are represented by a 0 / 1 sequence, and for any frame signal of the multi-speaker speech signal, if the frame signal is a speech segment, the discrimination result of the frame signal is represented by 0, and if the frame signal is a non-speech segment, the discrimination result of the frame signal is represented by 1;

[0035] The step of separating the speech signal of the first speaker and the speech signal of the second speaker from the signal of any channel of the multi-speaker speech according to the first discrimination result and the second discrimination result comprises:

[0036] For a signal of any channel of the multi-speaker speech signal, the signal of the channel is multiplied by the first discrimination result to obtain the speech signal of the first speaker, and the signal of the channel is multiplied by the second discrimination result to obtain the speech signal of the second speaker.

[0037] In a possible implementation, the multi-speaker speech separation method further includes:

[0038] The speech signal of the first speaker and the speech signal of the second speaker are denoised respectively by using a single-channel post-filtering model to obtain a denoised speech signal of the first speaker and a denoised speech signal of the second speaker.

[0039] The second aspect of the present application provides a multi-speaker speech separation device, comprising: a speech signal acquisition module, a phase difference determination module, a beamforming processing module, a first speech time-frequency mask determination module and a speech separation module;

[0040] The speech signal acquisition module is used to acquire multi-speaker speech signals collected by a microphone array consisting of two microphones arranged at different positions for a first speaker and a second speaker located at different positions;

[0041] The phase difference determination module is used to determine the phase difference between the two microphones according to the multi-speaker speech signal;

[0042] The beamforming processing module is used to perform fixed beamforming processing for a first area and fixed beamforming processing for a second area on the multi-speaker speech signals, respectively, to obtain beamforming signals corresponding to the first area and the second area, respectively, wherein the first area is the area where the first speaker is located, and the second area is the area where the second speaker is located;

[0043] The first speech time-frequency mask determination module is used to determine the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker according to the phase difference and the beamforming signals corresponding to the first area and the second area respectively;

[0044] The multi-speaker speech separation module is used to separate the speech signal of the first speaker and the speech signal of the second speaker from the signal of any channel of the multi-speaker speech signal according to the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker.

[0045] A third aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0046] The memory is used to store computer programs;

[0047] The processor is used to execute the computer program so that the electronic device can implement the steps of any one of the above-mentioned multi-speaker speech separation methods.

[0048] In a fourth aspect, the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of any one of the multi-speaker speech separation methods described above.

[0049] A fifth aspect of the present application provides a computer program product, comprising computer-readable instructions, which, when executed on an electronic device, enables the electronic device to implement any of the steps of the multi-speaker speech separation method described above.

[0050] By means of the above technical scheme, the multi-speaker speech separation method provided by the present application first obtains the multi-speaker speech signals collected by the two-microphone array for the first speaker and the second speaker located at different positions, then determines the phase difference of the two microphones according to the multi-speaker speech signals, and performs fixed beamforming processing for the first area (the area where the first speaker is located) and the fixed beamforming processing for the second area (the area where the second speaker is located) on the multi-speaker speech signals, respectively, to obtain beamforming signals corresponding to the first area and the second area, respectively, and then determines the speech time-frequency mask of the first speaker and the speech time-frequency mask of the second speaker according to the phase difference and the beamforming signals corresponding to the first area and the second area, respectively, and finally separates the speech signal of the first speaker and the speech signal of the second speaker from the signal of any channel of the multi-speaker speech according to the speech time-frequency mask of the first speaker and the speech time-frequency mask of the second speaker. The multi-speaker speech separation method provided by the present application uses the spatial information of the signals collected by the two-microphone array to realize speaker differentiation. The method is not affected by the gender of the speaker, has good separation effect and stability, and can meet application requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0052] Figure 1 A schematic diagram of a system architecture involved in this application;

[0053] Figure 2 A schematic diagram of a hardware structure of a terminal provided in an embodiment of the present application;

[0054] Figure 3 A schematic diagram of a hardware structure of a server provided in an embodiment of the present application;

[0055] Figure 4 A flowchart of a multi-speaker speech separation method provided in an embodiment of the present application;

[0056] Figure 5 A schematic diagram of a first area and a second area provided in an embodiment of the present application;

[0057] Figure 6 A schematic diagram of a flow chart of determining a first speech time-frequency mask of a first speaker and a first speech time-frequency mask of a second speaker according to a phase difference and beamforming signals corresponding to a first area and a second area respectively provided in an embodiment of the present application;

[0058] Figure 7 A flow chart of separating the speech signal of the first speaker and the speech signal of the second speaker from the signal of any channel of the multi-speaker speech signal according to the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker, and combining the second speech time-frequency mask of the first speaker and the second speech time-frequency mask of the second speaker provided in an embodiment of the present application;

[0059] Figure 8 A schematic diagram of the structure of a multi-speaker speech separation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0061] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0062] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0063] In one possible implementation, Figure 1 As shown, the system architecture involved in the present application may include a terminal 101 and a server 102, and the terminal 101 may interact with the server 102 via a network (wired network or wireless network). The server 102 may include one or more servers ( Figure 1 The example includes one server. The terminal 101 can obtain multi-speaker speech signals, transmit the obtained multi-speaker speech signals to the server 102 through the network, and the server 102 uses the multi-speaker speech separation method provided by the present application to perform speech separation on the multi-speaker speech signals, and feeds back the separation results to the terminal.

[0064] In another possible implementation, the system architecture involved in the present application may include a terminal. The terminal has strong data processing capabilities. The terminal can obtain multi-speaker speech signals and use the multi-speaker speech separation method provided in the present application to perform speech separation on the multi-speaker speech signals.

[0065] Next, the product form of the above-mentioned terminal is described.

[0066] The above-mentioned terminal can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, a robot, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiments of the present application do not impose any restrictions on this.

[0067] Figure 2 An optional hardware structure diagram of a terminal is shown.

[0068] refer to Figure 2As shown, the terminal may include a radio frequency unit 210, a memory 220, an input unit 230, a display unit 240, a camera 250 (optional), an audio circuit 260 (optional), a speaker 261 (optional), a microphone 262 (optional), an earphone jack 263 (optional), a processor 270, an external interface 280, a power supply 290 and other components. Those skilled in the art will appreciate that Figure 2 This is merely an example of a terminal and does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown in the figure, or a combination of certain components, or different components.

[0069] The input unit 230 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the terminal. Specifically, the input unit 230 may include a touch screen 231 (optional) and / or other input devices 232. The touch screen 231 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 270, and can receive and execute the command sent by the processor 270; the touch signal at least includes the contact point coordinate information. The touch screen 231 can provide an input interface and an output interface between the terminal and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch screen 231, the input unit 230 can also include other input devices. Specifically, other input devices 232 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0070] The display unit 240 may be used to display information input by a user or information provided to a user, various menus of the terminal, an interactive interface, file display, and / or playback of any multimedia file.

[0071] The memory 220 can be used to store instructions and data. The memory 220 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc. The instruction storage area can store software units such as operating systems, applications, and instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; it provides the processor 270 with hardware, software, and data resources including management of computing and processing equipment, and supports control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0072] The processor 270 is the control center of the terminal. It uses various interfaces and lines to connect various parts of the entire terminal. By running or executing instructions stored in the memory 220 and calling data stored in the memory 220, it executes various functions of the terminal and processes data, thereby controlling the terminal as a whole. Optionally, the processor 270 may include one or more processing units; preferably, the processor 270 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 270. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 270 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 220, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0073] The memory 220 may be used to store software codes related to the multi-speaker speech separation method, and the processor 270 may execute the software codes in the memory 220 or schedule other units (such as the input unit 230 and the display unit 240) to implement corresponding functions.

[0074] The radio frequency unit 210 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 270 for processing; in addition, the designed uplink data is sent to the base station. Generally, the radio frequency unit 210 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LowNoiseAmplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 210 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General PacketRadio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0075] In the embodiment of the present application, the radio frequency unit 210 can send data to other devices, and can also receive data sent by other devices. It should be understood that the radio frequency unit 210 is optional, and can be replaced by other communication interfaces, such as a network port.

[0076] The terminal also includes a power supply 290 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 270 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions.

[0077] The terminal also includes an external interface 280, which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal to communicate with other devices, or to connect a charger to charge the terminal.

[0078] Although not shown, the terminal may further include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described in detail here.

[0079] Next, the product form of the above server is described.

[0080] Figure 3 A structural diagram of the above server is provided, such as Figure 3As shown, the server may include a bus 301, a processor 302, a communication interface 303, and a memory 304. The processor 302, the memory 304, and the communication interface 303 communicate with each other via the bus 301.

[0081] The bus 301 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0082] The processor 302 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0083] The memory 304 may include a volatile memory, such as a random access memory (RAM). The memory 304 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0084] The memory 304 may be used to store software codes related to the multi-speaker speech separation method. The processor 302 may call the software codes stored in the memory 304 or schedule other units to implement corresponding functions.

[0085] The processors in the above-mentioned terminals and servers (for example, processor 270 and processor 302) may be hardware circuits (such as application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), general-purpose processors, digital signal processors (DSP), microprocessors or microcontrollers, etc.), or combinations of these hardware circuits. For example, the processor may be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0086] Next, the multi-speaker speech separation method provided by the present application is introduced through the following embodiments.

[0087] See also Figure 4 , shows a flow chart of a multi-speaker speech separation method provided in an embodiment of the present application, and the multi-speaker speech separation method may include:

[0088] Step S401: acquiring multi-speaker speech signals collected by a microphone array consisting of two microphones arranged at different positions from a first speaker and a second speaker located at different positions.

[0089] This embodiment uses a dual-microphone array (a microphone array composed of two microphones arranged at different positions) to collect speech signals from a first speaker and a second speaker located at different positions to obtain a dual-channel multi-speaker speech signal.

[0090] Exemplarily, the application scenario is an enterprise management scenario, and the microphone array is a microphone array set on a smart electronic work badge. The two microphones that make up the microphone array can be set vertically on the smart electronic work badge. The microphone array on the smart electronic work badge can collect voice signals of the wearer of the smart electronic work badge (first speaker) and others (second speaker), thereby obtaining multi-speaker voice signals.

[0091] Step S402: determining the phase difference between the two microphones according to the multi-talker speech signals, and performing fixed beamforming processing for the first region and fixed beamforming processing for the second region on the multi-talker speech signals, respectively, to obtain beamforming signals corresponding to the first region and the second region, respectively.

[0092] The first area is the area where the first speaker (such as the wearer of the smart electronic badge) is located, and the second area is the area where the second speaker (such as the non-wearer of the smart electronic badge) is located. Figure 5 , showing a schematic diagram of the first region and the second region.

[0093] There will be a slight time difference between the speech signals collected by the two microphones, and this time difference will cause a phase difference between the signals. In this embodiment, the "phase difference between the two microphones" refers to the phase difference between the signals collected by the two microphones.

[0094] Performing fixed beamforming processing on multi-speaker speech signals for the first area can enhance the speech signals of the speakers in the first area and suppress the speech signals of the speakers in the second area. Performing fixed beamforming processing on multi-speaker speech signals for the second area can enhance the speech signals of the speakers in the second area and suppress the speech signals of the speakers in the first area.

[0095] Step S403: determining a first speech time-frequency mask of the first speaker and a first speech time-frequency mask of the second speaker according to the phase difference and the beamforming signals corresponding to the first area and the second area respectively.

[0096] The first speech time-frequency mask of the first speaker can represent the proportion of the first speaker's speech in the multi-speaker speech signal, and the first speech time-frequency mask of the second speaker can represent the proportion of the second speaker's speech in the multi-speaker speech signal.

[0097] This embodiment predicts speech time-frequency masks corresponding to the first area and the second area respectively based on the phase difference and the beamforming signals corresponding to the first area and the second area respectively, and uses the speech time-frequency mask corresponding to the first area as the first speech time-frequency mask of the first speaker, and uses the speech time-frequency mask corresponding to the second area as the first speech time-frequency mask of the second speaker.

[0098] Step S404: Separate the speech signal of the first speaker and the speech signal of the second speaker from the signal of any channel of the multi-speaker speech signal according to the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker.

[0099] The multi-speaker speech signal is a dual-channel signal. The speech signal of the first speaker and the speech signal of the second speaker can be separated from the signal of any channel according to the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker.

[0100] The multi-speaker speech separation method provided in the embodiment of the present application first obtains the multi-speaker speech signal collected by the two-microphone array for the first speaker and the second speaker located in different directions, then determines the phase difference of the two microphones according to the multi-speaker speech signal, and performs fixed beamforming processing for the first area and fixed beamforming processing for the second area on the multi-speaker speech signal, respectively, to obtain the beamforming signals corresponding to the first area and the second area respectively, then determines the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker according to the phase difference and the beamforming signals corresponding to the first area and the second area respectively, and finally separates the speech signal of the first speaker and the speech signal of the second speaker from the signal of any channel of the multi-speaker speech according to the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker. The multi-speaker speech separation method provided in the embodiment of the present application utilizes the spatial information of the signal collected by the two-microphone array to realize speaker differentiation. The method is not affected by the gender of the speaker, has good separation effect and stability, and can meet application requirements.

[0101] In another embodiment of the present application, the specific implementation process of "step S403: determining the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker according to the phase difference and the beamforming signals corresponding to the first area and the second area respectively" in the above embodiment is introduced.

[0102] In one possible implementation, Figure 6 As shown, the process of determining the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker according to the phase difference and the beamforming signals corresponding to the first area and the second area respectively may include:

[0103] Step S601: Obtain logarithmic power spectra of beamforming signals corresponding to the first region and the second region respectively, to obtain two logarithmic power spectra.

[0104] A logarithmic power spectrum is obtained for the beamforming signal corresponding to the first region, and a logarithmic power spectrum is obtained for the beamforming signal corresponding to the second region, thereby obtaining two logarithmic power spectra.

[0105] Step S602: Using the pre-trained spatial separation model, based on the phase difference and two logarithmic power spectra, determine the speech time-frequency masks corresponding to the first area and the second area respectively, the speech time-frequency mask corresponding to the first area is used as the first speech time-frequency mask of the first speaker, and the speech time-frequency mask corresponding to the second area is used as the first speech time-frequency mask of the second speaker.

[0106] The phase difference and the two logarithmic power spectra are input into a pre-trained spatial separation model. The spatial separation model predicts and outputs speech time-frequency masks corresponding to the first region and the second region respectively based on the input phase difference and the two logarithmic power spectra.

[0107] The spatial separation model in this embodiment is trained using training samples with real speech time-frequency masks corresponding to two different areas (such as the area where the smart electronic work badge wearer is located and the area where the smart electronic work badge non-wearer is located). The training samples include microphone phase differences and two logarithmic power spectra determined based on training noisy mixed speech signals. The training noisy mixed speech signals are noisy speech signals of multiple speakers in two different areas (such as the smart electronic work badge wearer and the smart electronic work badge non-wearer).

[0108] The method for obtaining the training noisy mixed speech signal is as follows: obtain the clean speech signals s of multiple speakers in two different areas, and obtain the noise signal n (which can be obtained by simulation or collected by a microphone array); simulate and generate the room impulse response I of the sound source and the microphone array at different distances, different room environments, and different azimuth and elevation angles by the image source method, convolve it with the clean speech signal s, and superimpose it with the noise signal n, as shown in the following formula:

[0109] (1)

[0110] After obtaining the noisy mixed speech signal y, the speech energy of the noisy mixed speech signal y can be adjusted so that its signal-to-noise ratio is in the range of 0dB to 20dB, that is, the noisy mixed speech signal y is adjusted to a noisy mixed speech signal y′ with a signal-to-noise ratio in the range of 0dB to 20dB, and the noisy mixed speech signal y′ with a signal-to-noise ratio in the range of 0dB to 20dB is used as a training noisy mixed speech signal.

[0111] After obtaining the training noisy mixed speech signal y′, the real speech time-frequency masks corresponding to the two different regions can be determined according to the clean speech signal s and the training noisy mixed speech signal y′. Specifically, the clean speech signal s and the training noisy mixed speech signal y′ are respectively subjected to short-time Fourier transform to obtain the frequency domain amplitude spectrum S of the clean speech signal s and the frequency domain amplitude spectrum Y of the training noisy mixed speech signal y′. The real speech time-frequency masks corresponding to the two different regions are obtained according to the following formula:

[0112] (2)

[0113] It should be noted that the real speech time-frequency mask is calculated frame by frame. When calculating the real speech time-frequency mask corresponding to one of the regions, for each frame signal s of the clean speech signal s i , if the frame signal s iis the speech signal of the speaker in the area, then the frequency domain amplitude spectrum S of the frame signal is calculated i The frequency domain amplitude spectrum Y of the corresponding frame in the training noisy mixed speech signal y′ i The ratio S i / Y i , that is, mask i ′=S i / Y i , if the frame signal s i If it is not the speech signal of the speaker in the area, then the frame signal s i The frequency domain amplitude spectrum S i 0, that is, mask i ′ is 0. The real speech time-frequency mask corresponding to one region can be obtained through the above process. The real speech time-frequency mask corresponding to another region can be obtained in a similar way.

[0114] After obtaining the training noisy mixed speech signal y′, the phase difference IPD12 of the two microphones is determined according to the training noisy mixed speech signal y′, and the training noisy mixed speech signal y′ is subjected to fixed beamforming processing for two different regions to obtain beamforming signals corresponding to the two different regions, and the logarithmic power spectra of the beamforming signals corresponding to the two different regions are respectively obtained to obtain two logarithmic power spectra LPS_fix1 and LPS_fix2.

[0115] After obtaining the phase difference IPD12 and two logarithmic power spectra LPS_fix1 and LPS_fix2, the phase difference IPD12 and the two logarithmic power spectra LPS_fix1 and LPS_fix2 are used as training samples, and the real speech time-frequency masks mask1′ and mask2′ corresponding to two different areas are used as sample annotation information to obtain training data (training samples: LPS_fix1, LPS_fix2, IPD12; sample annotation information: mask1′, mask2′).

[0116] A large amount of training data can be obtained in the above manner, and then the spatial separation model can be trained using the large amount of training data.

[0117] Specifically, the training process of the spatial separation model may include: obtaining training data; inputting training samples (i.e., LPS_fix1, LPS_fix2, and IPD12) in the training data into the spatial separation model; the spatial separation model predicts speech time-frequency masks mask1 and mask2 corresponding to two different regions according to the input IPD12, LPS_fix1, and LPS_fix2; after predicting speech time-frequency masks mask1 and mask2 corresponding to two different regions, the prediction loss of the spatial separation model is determined according to mask1 and mask2 and sample annotation information in the obtained training data (i.e., real speech time-frequency masks mask1′ and mask2′ corresponding to two different regions), and the parameters of the spatial separation model are updated according to the prediction loss of the spatial separation model. Optionally, the prediction loss may adopt a mean square error loss, i.e., a mean square error loss L1 is calculated for mask1 and mask1′, and a mean square error loss L2 is calculated for mask2 and mask2′, L1 is fused with L2, and then the parameters of the spatial separation model are updated according to the fused loss. The spatial separation model is iteratively trained multiple times in the above manner until the training end condition is met (for example, a preset number of training times is reached or the model converges, etc.).

[0118] The spatial separation model in this embodiment may be, but is not limited to, a feedforward sequential memory neural network (FSMN).

[0119] In another embodiment of the present application, the specific implementation process of "step S404: separating the speech signal of the first speaker and the speech signal of the second speaker from the signal of any channel of the multi-speaker speech signal according to the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker" in the above embodiment is introduced.

[0120] There are multiple implementation methods for separating the speech signal of the first speaker and the speech signal of the second speaker from the signal of any channel of the multi-speaker speech signal according to the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker. In one possible implementation method, the first speech time-frequency mask of the first speaker can be used as the target speech time-frequency mask of the first speaker, and the first speech video mask of the second speaker can be used as the target speech time-frequency mask of the second speaker. Then, according to the target speech time-frequency mask of the first speaker and the target speech time-frequency mask of the second speaker, the speech signal of the first speaker and the speech signal of the second speaker are separated from the signal of any channel of the multi-speaker speech signal.

[0121] The above implementation method is to directly separate the speech signal of the first speaker and the speech signal of the second speaker from the signal of any channel of the multi-speaker speech signal according to the first speech time-frequency mask of the first speaker and the first speech video mask of the second speaker.

[0122] In order to improve the speech signal separation effect, in another possible implementation, the multi-speaker speech signals can be clustered by using a Gaussian mixture model to determine the second speech time-frequency mask of the first speaker and the second speech time-frequency mask of the second speaker, and then, the speech signal of the first speaker and the speech signal of the second speaker can be separated from the signal of any channel of the multi-speaker speech signal according to the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker, and combined with the second speech time-frequency mask of the first speaker and the second speech time-frequency mask of the second speaker. The above-mentioned training noisy mixed speech signal and the corresponding real speech time-frequency mask can be used to train the Gaussian mixture model, and then the trained Gaussian mixture model can be used to cluster the multi-speaker speech signals to determine the second speech time-frequency mask of the first speaker and the second speech time-frequency mask of the second speaker.

[0123] Specifically, Figure 7 As shown, the process of separating the speech signal of the first speaker and the speech signal of the second speaker from the signal of any channel of the multi-speaker speech signal according to the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker, and combining the second speech time-frequency mask of the first speaker and the second speech time-frequency mask of the second speaker may include:

[0124] Step S701: The first speech time-frequency mask of the first speaker is fused with the second speech time-frequency mask of the first speaker to obtain the target speech time-frequency mask of the first speaker, and the first speech time-frequency mask of the second speaker is fused with the second speech time-frequency mask of the second speaker to obtain the target speech time-frequency mask of the second speaker.

[0125] Specifically, the process of fusing the first speech time-frequency mask of the first speaker with the second speech time-frequency mask of the first speaker to obtain the target speech time-frequency mask of the first speaker may include: taking the maximum value of the first speech time-frequency mask of the first speaker and the second speech time-frequency mask of the first speaker at the frame level to obtain the target speech time-frequency mask of the first speaker.

[0126] Specifically, the process of fusing the first speech time-frequency mask of the second speaker with the second speech time-frequency mask of the second speaker to obtain the target speech time-frequency mask of the second speaker may include: taking the maximum value of the first speech time-frequency mask of the second speaker and the second speech time-frequency mask of the second speaker at the frame level to obtain the target speech time-frequency mask of the second speaker.

[0127] If the first speech time-frequency mask of the first speaker is represented as nn_mask1, the second speech time-frequency mask of the first speaker is represented as CGMMmask1, the first speech time-frequency mask of the second speaker is represented as nn_mask2, and the second speech time-frequency mask of the second speaker is represented as CGMMmask2, then the target speech time-frequency mask MASK1 of the first speaker and the target speech time-frequency mask MASK2 of the second speaker can be expressed as:

[0128] MASK1=max(nn_mask1,CGMMmask1) (3)

[0129] MASK2=max(nn_mask2,CGMMmask2) (4)

[0130] Exemplarily, the first speech time-frequency mask nn_mask1 of the first speaker = {nn_mask11, nn_mask12, nn_mask13, nn_mask14, nn_mask15 ...}, the second speech time-frequency mask CGMMmask1 of the first speaker = {CGMMmask11, CGMMmask12, CGMMmask13, CGMMmask14, CGMMmask15, ...}, assuming that nn_mask11>CGMMmask11, nn_mask12<CGMMmask12,nn_mask13> CGMMmask13, nn_mask14<CGMMmask14,nn_mask15> CGMMmask14,…then the target speech time-frequency mask MASK1 of the first speaker = max(nn_mask1, CGMMmask1) = {nn_mask11, CGMMmask12, nn_mask13, CGMMmask14, nn_mask15, …}.

[0131] Step S702: Separate the speech signal of the first speaker and the speech signal of the second speaker from the signal of any channel of the multi-speaker speech signal according to the target speech time-frequency mask of the first speaker and the target speech time-frequency mask of the second speaker.

[0132] Next, the specific implementation process of "separating the speech signal of the first speaker and the speech signal of the second speaker from the signal of any channel of the multi-speaker speech signal according to the target speech time-frequency mask of the first speaker and the target speech time-frequency mask of the second speaker" is introduced.

[0133] like Figure 8As shown, the specific implementation process of separating the speech signal of the first speaker and the speech signal of the second speaker from the signal of any channel of the multi-speaker speech signal according to the target speech time-frequency mask of the first speaker and the target speech time-frequency mask of the second speaker may include:

[0134] Step S801: based on the target speech time-frequency mask of the first speaker, distinguish the speech segment and the non-speech segment of each frame signal of the multi-speaker speech signal to obtain a first distinction result, and based on the target speech time-frequency mask of the second speaker, distinguish the speech segment and the non-speech segment of each frame signal of the multi-speaker speech signal to obtain a second distinction result.

[0135] The target speech time-frequency mask of the first speaker and the target speech time-frequency mask of the second speaker are both frame-level speech time-frequency masks.

[0136] Specifically, according to the target speech time-frequency mask of the first speaker, each frame signal of the multi-speaker speech signal is distinguished as a speech segment or a non-speech segment, and the process of obtaining the first distinction result may include: for each frame signal of the multi-speaker speech signal, if the speech time-frequency mask corresponding to the frame signal in the target speech time-frequency mask of the first speaker is greater than a set threshold (for example, 0.2), then the frame signal is judged to be a speech segment; otherwise, the frame signal is judged to be a non-speech segment, so that the first distinction result composed of the distinction results of each frame signal of the multi-speaker speech signal can be obtained.

[0137] Specifically, according to the target speech time-frequency mask of the second speaker, each frame signal of the multi-speaker speech signal is distinguished as a speech segment or a non-speech segment, and the process of obtaining the second distinction result may include: for each frame signal of the multi-speaker speech signal, if the speech time-frequency mask corresponding to the frame signal in the target speech time-frequency mask of the second speaker is greater than a set threshold (for example, 0.2), then the frame signal is judged to be a speech segment; otherwise, the frame signal is judged to be a non-speech segment, so that the second distinction result composed of the distinction results of each frame signal of the multi-speaker speech signal can be obtained.

[0138] Step S802: Separate the speech signal of the first speaker and the speech signal of the second speaker from the signal of any channel of the multi-speaker speech according to the first discrimination result and the second discrimination result.

[0139] For the signal of any channel of the multi-speaker speech signal, the speech signal of the first speaker can be separated from the signal of the channel according to the first discrimination result, and the speech signal of the second speaker can be separated from the signal of the channel according to the second discrimination result.

[0140] In a possible implementation, for any frame signal of a multi-speaker speech signal, if the frame signal is a speech segment, the discrimination result of the frame signal can be represented by 1, and if the frame signal is a non-speech segment, the discrimination result of the frame signal can be represented by 0, that is, the first discrimination result and the second discrimination result can be represented by a 0 / 1 sequence. For the signal of any channel of the multi-speaker speech signal, the signal of the channel can be multiplied by the first discrimination result (0 / 1 sequence) to obtain the speech signal of the first speaker, and the signal of the channel can be multiplied by the second discrimination result (0 / 1 sequence) to obtain the speech signal of the second speaker.

[0141] If the first discrimination result is represented as VAD1, the second discrimination result is represented as VAD2, and the signal of one channel of the multi-speaker speech signal is represented as mic-sig1 (the signal collected by a microphone in the microphone array), then the speech signal SIG1 of the first speaker and the speech signal SIG2 of the second speaker can be represented as:

[0142] (5)

[0143] (6)

[0144] Taking into account that the speech signal SIG1 of the first speaker and the speech signal SIG2 of the second speaker may need to be transcribed later, in order to improve the transcription rate, in another embodiment of the present application, after obtaining the speech signal SIG1 of the first speaker and the speech signal SIG2 of the second speaker, a single-channel post-filtering model can be used to perform noise reduction on the speech signal of the first speaker and the speech signal of the second speaker, respectively, to obtain the noise-reduced speech signal of the first speaker and the noise-reduced speech signal of the second speaker.

[0145] In one possible implementation, the single-channel post-filtering model can be a neural network denoising model, which is trained using a training single-channel noisy speech signal and a corresponding single-channel clean speech signal. When training the neural network denoising model, the training goal is to make the signal obtained by the neural network denoising model for denoising the training single-channel noisy speech signal approach the single-channel clean speech signal corresponding to the training single-channel noisy speech signal.

[0146] The multi-speaker speech separation method provided in the embodiment of the present application utilizes the spatial information of the signals received by the two-microphone array to distinguish the speakers. A more accurate separation result can be obtained by combining the separation strategy based on the spatial separation model with the separation strategy based on Gaussian mixture model clustering. In addition, further post-filtering of the separation result can enhance the speech, thereby improving the transcription rate of subsequent speech transcription.

[0147] The above describes the multi-speaker speech separation method provided by the embodiment of the present application. The following describes a device corresponding to the multi-speaker speech separation method.

[0148] See also Figure 8 , Figure 8 This is a schematic diagram of the structure of a multi-speaker speech separation device provided in an embodiment of the present application. The multi-speaker speech separation device may include: a speech signal acquisition module 801, a phase difference determination module 802a, a beamforming processing module 802b, a first time-frequency mask determination module 803a and a speaker speech separation module 804.

[0149] The speech signal acquisition module 801 is used to acquire multi-speaker speech signals collected by a microphone array consisting of two microphones arranged at different positions from a first speaker and a second speaker located at different positions.

[0150] The phase difference determination module 802a is used to determine the phase difference between two microphones according to the multi-speaker speech signals.

[0151] The beamforming processing module 802b is used to perform fixed beamforming processing for the first area and fixed beamforming processing for the second area on the multi-speaker speech signals, respectively, to obtain beamforming signals corresponding to the first area and the second area, respectively. The first area is the area where the first speaker is located, and the second area is the area where the second speaker is located.

[0152] The first speech time-frequency mask determining module 803a is used to determine the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker according to the phase difference and the beamforming signals corresponding to the first area and the second area respectively.

[0153] The multi-speaker speech separation module 804 is used to separate the speech signal of the first speaker and the speech signal of the second speaker from the signal of any channel of the multi-speaker speech signal according to the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker.

[0154] In a possible implementation, the first speech time-frequency mask determining module 803a determines the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker according to the phase difference and the beamforming signals corresponding to the first area and the second area respectively, and is specifically used to:

[0155] Obtaining logarithmic power spectra of the beamforming signals corresponding to the first region and the second region respectively, to obtain two logarithmic power spectra;

[0156] Using the pre-trained spatial separation model, the speech time-frequency masks corresponding to the first area and the second area are determined based on the phase difference and two logarithmic power spectra. The speech time-frequency mask corresponding to the first area is used as the first speech time-frequency mask of the first speaker, and the speech time-frequency mask corresponding to the second area is used as the first speech time-frequency mask of the second speaker.

[0157] Among them, the spatial separation model is trained using training samples with real speech time-frequency masks marked with two different areas respectively corresponding to each other. The training samples include microphone phase differences and two logarithmic power spectra determined based on training noisy mixed speech signals. The training noisy mixed speech signals are noisy speech signals of multiple speakers in the two different areas.

[0158] In a possible implementation, the multi-speaker speech separation apparatus may further include: a second speech time-frequency mask determination module 803b.

[0159] The second speech time-frequency mask determination module 803b is used to determine the second speech time-frequency mask of the first speaker and the second speech time-frequency mask of the second speaker by clustering the multi-speaker speech signals using a Gaussian mixture model.

[0160] When the multi-speaker speech separation module 804 separates the speech signal of the first speaker and the speech signal of the second speaker from the signal of any channel of the multi-speaker speech signal according to the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker, it is specifically used to:

[0161] According to the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker, and in combination with the second speech time-frequency mask of the first speaker and the second speech time-frequency mask of the second speaker, the speech signal of the first speaker and the speech signal of the second speaker are separated from the signal of any channel of the multi-speaker speech signal.

[0162] In a possible implementation, the multi-speaker speech separation module 804 may include: a time-frequency mask fusion module and a separation module.

[0163] The time-frequency mask fusion module is used to fuse the first speech time-frequency mask of the first speaker with the second speech time-frequency mask of the first speaker to obtain the target speech time-frequency mask of the first speaker, and to fuse the first speech time-frequency mask of the second speaker with the second speech time-frequency mask of the second speaker to obtain the target speech time-frequency mask of the second speaker.

[0164] The separation module is used to separate the speech signal of the first speaker and the speech signal of the second speaker from the signal of any channel of the multi-speaker speech signal according to the target speech time-frequency mask of the first speaker and the target speech time-frequency mask of the second speaker.

[0165] In a possible implementation, the first speech time-frequency mask of the first speaker, the first speech time-frequency mask of the second speaker, the second speech time-frequency mask of the first speaker, and the second speech time-frequency mask of the second speaker are all frame-level speech time-frequency masks.

[0166] The time-frequency mask fusion module fuses the first speech time-frequency mask of the first speaker with the second speech time-frequency mask of the first speaker to obtain the target speech time-frequency mask of the first speaker, and fuses the first speech time-frequency mask of the second speaker with the second speech time-frequency mask of the second speaker to obtain the target speech time-frequency mask of the second speaker, specifically for:

[0167] Taking the maximum value of the first speech time-frequency mask of the first speaker and the second speech time-frequency mask of the first speaker at the frame level to obtain the target speech time-frequency mask of the first speaker;

[0168] The maximum value of the first speech time-frequency mask of the second speaker and the second speech time-frequency mask of the second speaker is taken at the frame level to obtain the target speech time-frequency mask of the second speaker.

[0169] In a possible implementation, the target speech time-frequency mask of the first speaker and the target speech time-frequency mask of the second speaker are both frame-level speech time-frequency masks;

[0170] When the separation module separates the speech signal of the first speaker and the speech signal of the second speaker from the signal of any channel of the multi-speaker speech signal according to the target speech time-frequency mask of the first speaker and the target speech time-frequency mask of the second speaker, it is specifically used to:

[0171] According to the target speech time-frequency mask of the first speaker, each frame signal of the multi-speaker speech signal is distinguished as a speech segment and a non-speech segment to obtain a first distinction result, and according to the target speech time-frequency mask of the second speaker, each frame signal of the multi-speaker speech signal is distinguished as a speech segment and a non-speech segment to obtain a second distinction result;

[0172] According to the first discrimination result and the second discrimination result, the speech signal of the first speaker and the speech signal of the second speaker are separated from the signal of any channel of the multi-speaker speech.

[0173] In a possible implementation, when the separation module discriminates the speech segment and the non-speech segment of each frame signal of the multi-speaker speech signal according to the target speech time-frequency mask of the first speaker, it is specifically used to:

[0174] For each frame signal of the multi-speaker speech signal, if the speech time-frequency mask corresponding to the frame signal in the target speech time-frequency mask of the first speaker is greater than the set threshold, the frame signal is determined to be a speech segment, otherwise, the frame signal is determined to be a non-speech segment.

[0175] In a possible implementation, when the separation module discriminates the speech segment and the non-speech segment of each frame signal of the multi-speaker speech signal according to the target speech time-frequency mask of the second speaker, it is specifically used to:

[0176] For each frame signal of the multi-speaker speech signal, if the speech time-frequency mask corresponding to the frame signal in the target speech time-frequency mask of the second speaker is greater than the set threshold, the frame signal is determined to be a speech segment, otherwise, the frame signal is determined to be a non-speech segment.

[0177] In a possible implementation, the first discrimination result and the second discrimination result are represented by a 0 / 1 sequence. For any frame signal of the multi-speaker speech signal, if the frame signal is a speech segment, the discrimination result of the frame signal is represented by 0; if the frame signal is a non-speech segment, the discrimination result of the frame signal is represented by 1.

[0178] When the separation module separates the speech signal of the first speaker and the speech signal of the second speaker from the signal of any channel of the multi-speaker speech according to the first discrimination result and the second discrimination result, it is specifically used to:

[0179] For a signal of any channel of the multi-speaker speech signal, the signal of the channel is multiplied by the first discrimination result to obtain the speech signal of the first speaker, and the signal of the channel is multiplied by the second discrimination result to obtain the speech signal of the second speaker.

[0180] In a possible implementation, the multi-speaker speech separation apparatus may further include: a post-filtering module 805 .

[0181] The post-filtering module 805 is used to perform noise reduction on the speech signal of the first speaker and the speech signal of the second speaker respectively using a single-channel post-filtering model to obtain the noise-reduced speech signal of the first speaker and the noise-reduced speech signal of the second speaker.

[0182] The multi-speaker speech separation device provided in the embodiment of the present application uses the spatial information of the signal collected by the two-microphone array to realize speaker differentiation. The method is not affected by the gender of the speaker, has good separation effect and stability, and can meet application requirements.

[0183] An embodiment of the present application also provides an electronic device, which may include: at least one processor, at least one communication interface, at least one memory and at least one communication bus.

[0184] In the embodiment of the present application, the number of the processor, the communication interface, the memory, and the communication bus is at least one, and the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0185] The processor may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application;

[0186] The memory may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0187] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to implement the steps of the multi-speaker speech separation method provided in the above embodiment.

[0188] The embodiment of the present application also provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the multi-speaker speech separation method provided in the above embodiment.

[0189] The embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the steps of the multi-speaker speech separation method provided in the above embodiment.

[0190] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0191] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0192] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0193] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A multi-speaker speech separation method, characterized in that: include: Acquire multi-speaker speech signals collected by a microphone array consisting of two microphones arranged at different positions from a first speaker and a second speaker located at different positions; Determining the phase difference between the two microphones according to the multi-speaker speech signals, and performing fixed beamforming processing for a first area and fixed beamforming processing for a second area on the multi-speaker speech signals, respectively, to obtain beamforming signals corresponding to the first area and the second area, respectively, wherein the first area is an area where the first speaker is located, and the second area is an area where the second speaker is located; Determine a first speech time-frequency mask of the first speaker and a first speech time-frequency mask of the second speaker according to the phase difference and the beamforming signals corresponding to the first area and the second area respectively; According to the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker, the speech signal of the first speaker and the speech signal of the second speaker are separated from the signal of any channel of the multi-speaker speech signal.

2. The multi-speaker speech separation method according to claim 1, characterized in that: The determining, according to the phase difference and the beamforming signals corresponding to the first area and the second area respectively, a first speech time-frequency mask of the first speaker and a first speech time-frequency mask of the second speaker comprises: Obtaining logarithmic power spectra of the beamforming signals corresponding to the first area and the second area respectively, to obtain two logarithmic power spectra; Using the pre-trained spatial separation model, based on the phase difference and the two logarithmic power spectra, determining the speech time-frequency masks corresponding to the first area and the second area respectively, the speech time-frequency mask corresponding to the first area is used as the first speech time-frequency mask of the first speaker, and the speech time-frequency mask corresponding to the second area is used as the first speech time-frequency mask of the second speaker; The spatial separation model is trained using training samples with real speech time-frequency masks marked with two different regions respectively corresponding to the training samples, wherein the training samples include microphone phase differences and two logarithmic power spectra determined based on training noisy mixed speech signals, and the training noisy mixed speech signals are noisy speech signals of multiple speakers in the two different regions.

3. The multi-speaker speech separation method according to claim 1, characterized in that: Also includes: Clustering the multi-speaker speech signals by using a Gaussian mixture model to determine a second speech time-frequency mask of the first speaker and a second speech time-frequency mask of the second speaker; The step of separating the speech signal of the first speaker and the speech signal of the second speaker from the signal of any channel of the multi-speaker speech signal according to the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker comprises: According to the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker, and in combination with the second speech time-frequency mask of the first speaker and the second speech time-frequency mask of the second speaker, the speech signal of the first speaker and the speech signal of the second speaker are separated from the signal of any channel of the multi-speaker speech signal.

4. The multi-speaker speech separation method according to claim 3, characterized in that: The method of separating the speech signal of the first speaker and the speech signal of the second speaker from the signal of any channel of the multi-speaker speech signal according to the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker, and combining the second speech time-frequency mask of the first speaker and the second speech time-frequency mask of the second speaker, comprises: fusing the first speech time-frequency mask of the first speaker with the second speech time-frequency mask of the first speaker to obtain the target speech time-frequency mask of the first speaker, and fusing the first speech time-frequency mask of the second speaker with the second speech time-frequency mask of the second speaker to obtain the target speech time-frequency mask of the second speaker; According to the target speech time-frequency mask of the first speaker and the target speech time-frequency mask of the second speaker, the speech signal of the first speaker and the speech signal of the second speaker are separated from the signal of any channel of the multi-speaker speech signal.

5. The multi-speaker speech separation method according to claim 4, characterized in that: The first speech time-frequency mask of the first speaker, the first speech time-frequency mask of the second speaker, the second speech time-frequency mask of the first speaker, and the second speech time-frequency mask of the second speaker are all frame-level speech time-frequency masks; The step of fusing the first speech time-frequency mask of the first speaker with the second speech time-frequency mask of the first speaker to obtain the target speech time-frequency mask of the first speaker, and fusing the first speech time-frequency mask of the second speaker with the second speech time-frequency mask of the second speaker to obtain the target speech time-frequency mask of the second speaker comprises: Taking the maximum value of the first speech time-frequency mask of the first speaker and the second speech time-frequency mask of the first speaker at the frame level to obtain the target speech time-frequency mask of the first speaker; The maximum value of the first speech time-frequency mask of the second speaker and the second speech time-frequency mask of the second speaker is taken at the frame level to obtain the target speech time-frequency mask of the second speaker.

6. The multi-speaker speech separation method according to claim 4, characterized in that: The target speech time-frequency mask of the first speaker and the target speech time-frequency mask of the second speaker are both frame-level speech time-frequency masks; The step of separating the speech signal of the first speaker and the speech signal of the second speaker from the signal of any channel of the multi-speaker speech signal according to the target speech time-frequency mask of the first speaker and the target speech time-frequency mask of the second speaker comprises: According to the target speech time-frequency mask of the first speaker, each frame signal of the multi-speaker speech signal is distinguished as a speech segment and a non-speech segment to obtain a first distinction result, and according to the target speech time-frequency mask of the second speaker, each frame signal of the multi-speaker speech signal is distinguished as a speech segment and a non-speech segment to obtain a second distinction result; According to the first discrimination result and the second discrimination result, the speech signal of the first speaker and the speech signal of the second speaker are separated from the signal of any channel of the multi-speaker speech.

7. The multi-speaker speech separation method according to claim 6, characterized in that: The step of distinguishing speech segments and non-speech segments of each frame signal of the multi-speaker speech signal according to the target speech time-frequency mask of the first speaker includes: For each frame signal of the multi-speaker speech signal, if the speech time-frequency mask corresponding to the frame signal in the target speech time-frequency mask of the first speaker is greater than a set threshold, the frame signal is determined to be a speech segment; otherwise, the frame signal is determined to be a non-speech segment; The step of distinguishing speech segments and non-speech segments of each frame signal of the multi-speaker speech signal according to the target speech time-frequency mask of the second speaker includes: For each frame signal of the multi-speaker speech signal, if the speech time-frequency mask corresponding to the frame signal in the target speech time-frequency mask of the second speaker is greater than a set threshold, the frame signal is determined to be a speech segment; otherwise, the frame signal is determined to be a non-speech segment.

8. The multi-speaker speech separation method according to claim 6, characterized in that: The first discrimination result and the second discrimination result are represented by a 0 / 1 sequence. For any frame signal of the multi-speaker speech signal, if the frame signal is a speech segment, the discrimination result of the frame signal is represented by 0, and if the frame signal is a non-speech segment, the discrimination result of the frame signal is represented by 1; The step of separating the speech signal of the first speaker and the speech signal of the second speaker from the signal of any channel of the multi-speaker speech according to the first discrimination result and the second discrimination result comprises: For a signal of any channel of the multi-speaker speech signal, the signal of the channel is multiplied by the first discrimination result to obtain the speech signal of the first speaker, and the signal of the channel is multiplied by the second discrimination result to obtain the speech signal of the second speaker.

9. The multi-speaker speech separation method according to any one of claims 1 to 8, characterized in that: Also includes: The speech signal of the first speaker and the speech signal of the second speaker are denoised respectively by using a single-channel post-filtering model to obtain a denoised speech signal of the first speaker and a denoised speech signal of the second speaker.

10. A multi-speaker speech separation device, characterized in that: include: A speech signal acquisition module, a phase difference determination module, a beamforming processing module, a first speech time-frequency mask determination module and a speech separation module; The speech signal acquisition module is used to acquire multi-speaker speech signals collected by a microphone array consisting of two microphones arranged at different positions for a first speaker and a second speaker located at different positions; The phase difference determination module is used to determine the phase difference between the two microphones according to the multi-speaker speech signal; The beamforming processing module is used to perform fixed beamforming processing for a first area and fixed beamforming processing for a second area on the multi-speaker speech signals, respectively, to obtain beamforming signals corresponding to the first area and the second area, respectively, wherein the first area is the area where the first speaker is located, and the second area is the area where the second speaker is located; The first speech time-frequency mask determination module is used to determine the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker according to the phase difference and the beamforming signals corresponding to the first area and the second area respectively; The multi-speaker speech separation module is used to separate the speech signal of the first speaker and the speech signal of the second speaker from the signal of any channel of the multi-speaker speech signal according to the first speech time-frequency mask of the first speaker and the first speech time-frequency mask of the second speaker.

11. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the steps of the multi-speaker speech separation method as described in any one of claims 1 to 9.

12. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the multi-speaker speech separation method as described in any one of claims 1 to 9.

13. A computer program product, characterized in that The method comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the steps of the multi-speaker speech separation method as claimed in any one of claims 1 to 9.

Images