Low latency speaker separation

Abstract

A system for speech separation includes data processing hardware and memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including (i) generating a two-dimensional representation of a speech mix, (ii) separating the speech mix into an initial separation, and (ii) generating a two-dimensional representation of the speech mix. The method comprises the steps of (iii) providing an initial separation and a speaker representation to a refinement module, (iv) refining the initial separation based on the initial separation and the speaker representation, (v) estimating a mask for each speaker, and (vi) applying the mask to the two-dimensional representation to create a two-dimensional per-speaker representation.

Description

[0001] introduction

[0002] The information provided in this section is for the purpose of generally presenting the context of the present disclosure. The work of the presently named inventors, to the extent it is described in this section and in aspects of the description that may not qualify as prior art at the time of filing, is neither explicitly nor implicitly admitted to be prior art against the present disclosure. Technical Field

[0003] The present disclosure relates generally to systems and methods for speech separation, and more particularly to systems and methods for low-latency, single-microphone speech separation utilizing speaker representations. Background Art

[0004] Speech separation aims to separate target speech from background interference and can be used in a wide range of applications. For example, speech separation can be used by a processor associated with a hearing prosthesis or by a communication device such as a mobile phone or tablet. In each of the aforementioned applications, speech separation is employed to separate the desired speech from background interference such as other speakers, noise and / or music.

[0005] In the context of a hearing prosthesis such as a hearing aid, a processor associated with the hearing aid separates desired signals associated with a speaker, music, and / or a communication device such as a mobile phone or tablet from background noise such as signals associated with construction, road noise, wind, etc. In the case of a communication device such as a mobile phone, a processor associated with the mobile phone separates desired signals associated with a speaker from undesired signals associated with interference such as signals associated with background music, construction, road noise, etc.

[0006] When a phone call is made in a vehicle on a mobile device or by a system associated with the vehicle, a processor associated with the mobile device or the vehicle attempts to separate desired signals associated with the speech of one or more persons speaking within the vehicle's cabin from undesired signals associated with background noise in and around the vehicle. As can be appreciated, background noise is typically abundant in the context of a vehicle and originates from a variety of sources. That is, background noise may originate from road noise, construction, operation of vehicle systems (e.g., radio, windows and door locks, etc.), operation of other vehicles, and other vehicle occupants.

[0007] With respect to background noise originating from other vehicle occupants, speech separation can be utilized to separate the speech of the various occupants located in the cabin to attribute the speech to the appropriate occupant. In this case, speech separation can be used to solve the so-called cocktail party problem, where multiple people are speaking simultaneously in a single location. In this example, multiple vehicle occupants are all located in the cabin and can speak simultaneously, requiring the speech to be separated and attributed to each occupant.

[0008] In short, while speech separation is a difficult task for any device, speech separation within a vehicle is often exacerbated by the ever-changing environment within the vehicle, the many conditions in and around the vehicle that cause interference, and the general unpredictability of interference experienced during vehicle operation. Summary of the invention

[0009] A system for speech separation is provided, the system comprising data processing hardware and memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations comprising (i) generating a two-dimensional representation of a speech mixture, (ii) separating the speech mixture into initial separations, (iii) providing the initial separations and speaker representations to a refinement module, (iv) refining the initial separations based on the initial separations and the speaker representations, (v) estimating a mask for each speaker, and (vi) applying the mask to the two-dimensional representation to create a two-dimensional per-speaker representation.

[0010] The system may include one or more of the following optional features. For example, generating a two-dimensional representation of the speech mixture may include passing the speech mixture through an encoder. Additionally or alternatively, at least one feature projection may be provided to a refinement module for use by the refinement module in refining the initial separation.

[0011] Speaker embedding tables may be used to refine the speaker representations. Additionally or alternatively, the two-dimensional per-speaker representations may be passed through a decoder to produce per-speaker waveforms.

[0012] The microphone may be in communication with the data processing hardware and may be configured to detect a speech mixture. The vehicle may include a microphone.

[0013] In another configuration, a system for speech separation is provided, and the system includes data processing hardware and memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including (i) generating a two-dimensional representation of a speech mixture, (ii) separating the speech mixture into an initial separation, (iii) providing the initial separation and at least one feature projection to a refinement module, (iv) refining the initial separation based on the initial separation and the at least one feature projection, (v) estimating a mask for each speaker, and (vi) applying the mask to the two-dimensional representation to create a two-dimensional per-speaker representation.

[0014] The system may include one or more of the following optional features. For example, generating a two-dimensional representation of the speech mixture may include passing the speech mixture through an encoder. Additionally or alternatively, the speaker representation may be provided to a refinement module for use by the refinement module in refining the initial separation.

[0015] Speaker embedding tables may be used to refine the speaker representations. Additionally or alternatively, the two-dimensional per-speaker representations may be passed through a decoder to produce per-speaker waveforms.

[0016] The microphone may be in communication with the data processing hardware and may be configured to detect a speech mixture. The vehicle may include a microphone.

[0017] In yet another configuration, a system for speech separation is provided and includes data processing hardware and memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including (i) generating a two-dimensional representation of a speech mixture, (ii) separating the speech mixture into initial separations, (iii) providing the initial separations to a speaker module, (iv) generating a per-frame representation of the initial separations consistent with a stored speaker representation, (v) refining the initial separations based on the initial separations and the speaker representations, (vi) estimating a mask for each speaker, and (vii) applying the mask to the two-dimensional representation to create a two-dimensional per-speaker representation.

[0018] The system may include one or more of the following optional features. For example, generating a two-dimensional representation of the speech mixture may include passing the speech mixture through an encoder. Additionally or alternatively, a speaker embedding table may be used to refine the speaker representation.

[0019] In one configuration, the two-dimensional per-speaker representation may be passed through a decoder to produce a per-speaker waveform.

[0020] The microphone may be in communication with the data processing hardware and may be configured to detect a speech mixture. The vehicle may include a microphone. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are for illustrative purposes only of selected configurations and are not intended to limit the scope of the present disclosure.

[0022] Figure 1 is a perspective view of a vehicle according to the principles of the present disclosure;

[0023] Figure 2 is with Figure 1 a schematic diagram of a body control module for use with a vehicle; and

[0024] Figure 3 is a flow chart detailing a system and method for speech separation according to the principles of the present disclosure.

[0025] Corresponding reference characters indicate corresponding parts throughout the several views of the drawings. DETAILED DESCRIPTION

[0026] Example configurations will now be described more fully with reference to the accompanying drawings. Example configurations are provided so that the present disclosure will be thorough and will fully convey the scope of the present disclosure to those of ordinary skill in the art. Specific details, such as examples of specific components, devices, and methods, are set forth to provide a thorough understanding of the configurations of the present disclosure. It will be apparent to those of ordinary skill in the art that specific details need not be employed, that example configurations may be embodied in many different forms, and that the specific details and example configurations should not be construed as limiting the scope of the present disclosure.

[0027] The terms used herein are only used for the purpose of describing specific exemplary configurations and are not intended to be limiting. As used herein, the singular articles "a", "an" and "the" may also be intended to include plural forms unless the context clearly states otherwise. The terms "comprises", "comprising", "including" and "having" are inclusive and therefore specify the presence of features, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, steps, operations, elements, parts and / or groups thereof. The method steps, processes and operations described herein should not be interpreted as necessarily requiring them to be performed in the specific order discussed or shown, unless specifically identified as an execution order. Additional or alternative steps may be adopted.

[0028] When an element or layer is referred to as being "on another element or layer," "engaged to," "connected to," "attached to," or "coupled to" another element or layer, it may be directly on, directly engaged with, connected to, attached to, or coupled to another element or layer, or there may be intermediate elements or layers. Conversely, when an element is referred to as being "directly on another element or layer," "directly engaged to," "directly connected to," "directly attached to," or "directly coupled to" another element or layer, there may be no intermediate elements or layers. Other words used to describe the relationship between elements should be interpreted in a similar manner (e.g., "between" versus "directly between," "adjacent" versus "directly adjacent," etc.). As used herein, the term "and / or" includes any and all combinations of one or more associated listed items.

[0029] The terms first, second, third, etc. can be used in this article to describe various elements, components, regions, layers and / or parts. These elements, components, regions, layers and / or parts should not be limited by these terms. These terms can only be used to distinguish an element, component, region, layer or part from another region, layer or part. Unless the context clearly indicates, terms such as "first", "second" and other numerical terms do not imply order or sequence. Therefore, without departing from the teaching of the example configuration, the first element, component, region, layer or part discussed below can be referred to as the second element, component, region, layer or part.

[0030] In this application, including the definitions below, the term module may be replaced by the term circuit. The term module may refer to, be part of, or include: an application specific integrated circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor (shared, dedicated, or group) that executes code; a memory (shared, dedicated, or group) that stores code executed by the processor; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system on a chip.

[0031] The term code as used above may include software, firmware and / or microcode, and may refer to a program, a routine, a function, a class and / or an object. The term shared processor includes a single processor that executes some or all codes from multiple modules. The term group processor includes a processor that executes some or all codes from one or more modules in combination with an additional processor. The term shared memory includes a single memory that stores some or all codes from multiple modules. The term group memory includes a memory that stores some or all codes from one or more modules in combination with an additional memory. The term memory may be a subset of the term computer-readable medium. The term computer-readable medium does not include transient electrical signals and electromagnetic signals propagated through the medium, and therefore can be considered to be tangible and non-transient memory. Non-limiting examples of non-temporary memory include tangible computer-readable media, which include non-volatile memory, magnetic memory, and optical memory.

[0032] The apparatus and methods described in this application may be implemented in part or in whole by one or more computer programs executed by one or more processors. The computer program includes processor executable instructions stored on at least one non-transitory tangible computer readable medium. The computer program may also include and / or rely on stored data.

[0033] Software applications (i.e., software resources) may refer to computer software that enables a computing device to perform tasks. In some examples, software applications may be referred to as "applications," "applications," or "programs." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0034] Non-transitory memory can be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0035] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0036] Various implementations of the systems and techniques described herein can be realized in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system that includes at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to the storage system, at least one input device, and at least one output device.

[0037] The processes and logic flows described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware), which execute one or more computer programs to perform functions by operating on input data and generating outputs. The processes and logic flows can also be performed by special logic circuits (e.g., FPGA (field programmable gate array) or ASIC (application-specific integrated circuit)). As an example, processors suitable for executing computer programs include both general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or be operably coupled to receive data from it or transmit data to it or both. However, a computer does not need to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0038] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen) for displaying information to the user and optionally a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0039] Special References Figure 1 , a vehicle 10 is provided. The vehicle 10 includes a vehicle compartment 12, one or more seats 14 disposed in the vehicle compartment 12, and a microphone 16. Figure 1As shown, the microphone 16 is located within the vehicle cabin 12 proximate one or both of the seats 14. For example, the microphone 16 may be located between a pair of front seats 14 such that the microphone 16 is substantially equidistant from each seat 14. In one configuration, the microphone 16 is located in a headliner 20 of the vehicle 10 to place the microphone 16 in close proximity to an occupant of the vehicle 10 when the occupant of the vehicle 10 is seated in the seats 14. Although the vehicle 10 is described and shown as including a single microphone 16, the vehicle 10 may include multiple microphones 16 located at various locations within the vehicle cabin 12.

[0040] Special References Figure 2 , schematically illustrates a body control module (BCM) 20 of the vehicle 10. The BCM 20 includes data processing hardware 22 and memory hardware 24, which cooperate to perform Figure 3 The illustrated speech separation system and method. The data processing hardware 22 and memory hardware 24 include an encoder 26 that communicates with the microphone 16 and a decoder 28 that communicates with an external device 30 such as, for example, a mobile phone.

[0041] The encoder 26 may utilize a fast Fourier transform (FFT) method to process the speech or waveform mixture signal 32 received by the microphone 16. The FFT may be a short-time fast Fourier transform (STFT), and further may be used in conjunction with a learning filter bank. For example, the encoder 26 may employ a learning filter bank having a frame size of 16 samples and an overlap of eight (8) samples with a depth of 512 implemented using a one-dimensional convolutional layer, followed by a rectified linear unit (ReLU) activation function. The encoder 26 may utilize an FFT to convert the speech mixture signal 32 from its original time domain into a two-dimensional representation of the speech mixture signal 32. For example, the encoder 26 may transform the original time domain of the speech mixture signal 32 into the frequency domain to generate a two-dimensional representation of the signal 32.

[0042] The decoder 28 may similarly utilize an FFT method to convert the two-dimensional representation of the speech mixture signal 32 into a waveform signal (i.e., to return the speech mixture signal 32 from the frequency domain to the time domain) and then transmit the signal to the external device 30. The decoder 28 may employ an inverse short-time FFT (iSTFT) and may also be used in conjunction with a learning filter bank. For example, the encoder 26 may employ a learning filter bank having a frame size of 16 samples and an overlap of eight (8) samples with a depth of 512 implemented using a one-dimensional convolutional layer, followed by a rectified linear unit (ReLU) activation function. It should be noted that the decoder 28 does not convert the same speech mixture signal 32 received by the encoder 26 into the time domain, but rather converts the speech mixture signal 32 based on the frequency domain into the time domain. Figure 3 The illustrated flow chart determines a two-dimensional representation of the separated speech signal, as will be described in more detail below.

[0043] Continue to refer Figure 2 , the memory hardware 24 stores and the data processing hardware 22 executes an early separation (ES) module 34, a separation refinement (SR) module 36, a mask module 38, a speaker module 40, a feature projection (FP) module 42, and an embedding table 44. As will be described in more detail below, the ES module 34, the SR module 36, and the speaker module 40 cooperate to separate the two-dimensional representation of the speech mixture 32 received from the encoder 26 into individual speakers, which are then sent to the decoder 28 for use by the decoder 28 in generating per-speaker waveforms. In this way, the ES module 34, the SR module 36, and the speaker module 40 collectively form a speech separator 47.

[0044] Specific reference Figure 3 , a flow chart of the operation of the speech module 40 and the associated encoder 26 and decoder 28 will be described in detail. The speech module 40 is an encoder-decoder model based on masking. As will be described, the input speech or waveform mixture 32 detected by the microphone 16 is processed by the encoder 26, which generates a two-dimensional representation 46 of the mixture 32. The two-dimensional representation 46 is then passed to the mask module 38, which estimates the mask for each speaker. The mask is applied to the two-dimensional signal representation 46 to create a two-dimensional per-speaker representation estimate 48 of the two-dimensional signal representation 46. The estimate 48 is then passed through the decoder 28 to achieve a per-speaker waveform 50.

[0045] In operation, when one or more occupants are seated in the vehicle cabin 12, an input speech mixture 32 may be detected by the microphone 16. For example, the microphone 16 may detect speech from an occupant seated in a respective front seat 14 of the vehicle cabin 12. The detected speech will include a mixture of the two seated occupants and additional interfering noise. The input speech mixture 32 is sent to the encoder 26, which converts the speech mixture signal 32 from its original time domain to a two-dimensional representation 46 of the speech mixture signal 32, as described above.

[0046] The two-dimensional representation 46 is sent to the ES module 34, which generates an estimate of the speaker mask. The estimated speaker mask 52 is applied to the two-dimensional representation 46 to achieve an initial separation 54 or encoded speaker estimate of the two-dimensional representation 46. The encoded speaker estimate 54 is then provided to the SR module 36.

[0047] The ES module 34 is initially trained as a separate separator within the encoder-decoder model. The ES module 34 can be trained using utterance-level permutation invariant training (uPIT) with the same reconstruction loss. For example, the ES module 34 can be trained as a separator using uPIT with a negative scale-invariant signal-distortion ratio (SI-SDR) used as the loss term. The same encoder 26 and decoder 28 can also be used for Figure 3The system shown in FIG. 1 is a system in which the speech separator 47 is used as an early separation module. Figure 3 When the entire system is shown, the ES module 34 and the encoder 26 are frozen while continuing the training of the decoder 28. Then the negative SI-SDR can be used as the loss to train Figure 3 It should be noted that although the ES module 34 and the encoder 26 are described as being frozen after training, the ES module 34 can be used with Figure 3 The complete system shown in FIG. 1 continues to be trained together in an effort to continuously improve the ability of the ES module 34 to separate the two-dimensional representation 46.

[0048] A temporal convolutional network (TCN) may be used to implement the ES module 34. The TCN may have 512 convolutional block channels, a skip connection path, and a residual (bottleneck). Each dilated convolution has a kernel size of three (3) with 128 channels each. The ES module 34 consists of three (3) repetitions, each with eight (8) convolutional blocks. In one configuration, all convolutions are implemented as causal, creating a fully causal model. Alternatively, the bottom convolutional block of the ES model 34 may be non-causal to allow for small look-ahead filters.

[0049] The encoded speaker estimates 54 are sent to the speaker module 40. The speaker module 40 receives the speaker estimates or initial separations 54 and generates per-frame representations 56 that are consistent with the stored speaker representations. These per-frame representations can be used to separate the speakers in the two-dimensional representation 46. The embedding table 44 is used to train the consistency with the speaker representations, as will be described in more detail below.

[0050] The average of one hundred (100) frames where the speaker representations have 50% overlap is compared to the speaker embedding table value contained in embedding table 44. The penalty term increases as speaker representation 56 deviates from the table value contained in embedding table 44 for the speaker. In other words, the more the speaker representation 56 deviates from the value contained in embedding table 44 for the particular speaker, the higher the penalty term.

[0051] For example, the classification loss is used to measure speaker bias, whereby the loss increases when the representation estimate for a particular speaker (A) moves away from the representation for speaker (A) stored in the embedding table 44. The loss may also increase if the representation estimate for speaker (A) is close to any other stored representation in the embedding table 44 (i.e., if the representation estimate for speaker (A) is close to the stored representation for speaker (B)). The loss may be used to train the speaker module 40 to move the representation estimates for each speaker closer to their table values ​​and further away from the table values ​​for any other speaker. In this way, the speaker module 40 is trained to move the speaker representations 56 closer to the speaker representations contained in the embedding table 44, thereby causing the speaker module 40 to generate more accurate speaker representations 56 in the future.

[0052] A causal TCN may be used in an embedding table 44 with ten (10) convolutional blocks and no repetition. The causal TCN may have a kernel of three (3), 512 channels, and 128 bottleneck channels. The first layer of the speaker module 40 may receive all of the initially separated signals together such that for each frame, the module 40 receives a full filter bank representation size vector multiplied by the number of speakers. As an example, for two (2) speakers and a representation size of 512, the speaker module 40 will receive 1024 size vectors per frame. The speaker module 40 will output a vector of the number of speakers estimated by the speaker representation size. In one configuration, the speaker representation size is 512, and therefore, for two (2) speakers, the module 40 will output two (2) 512 size vectors per frame, concatenated into one vector of length 1024.

[0053] Continue to refer Figure 3 , embedding table 44 receives speaker representations 56 and helps train speaker module 40. Embedding table 44 is determined by the number of speakers in the dataset and the size of the speaker representations. For example, for a dataset of one hundred (100) speakers with a speaker representation (embedding) size of 512, table 44 would have the following dimensions: 100×512. Embedding table 44 is only used during the training phase and is adjusted so that each speaker representation is pulled towards the representation generated by speaker module 40 to obtain better separation refinement results. Although embedding table 44 is described as being used only during the training phase, embedding table 44 can be utilized in an ongoing manner to continually improve the ability of speaker module 40 to determine speaker representations 56.

[0054] The speaker representations 56 generated by the speaker module 40 are sent to the feature projection module 42. The concatenated speaker representations are projected to the SR module 36 at the bottleneck layer of the SR module 36 as element-wise addition and multiplication features by placing the features from the concatenated speaker representation vectors through a fully connected (FC) layer into a vector of the size of the bottleneck layer of the SR module 36. In one configuration, for 24 bottleneck representations in the SR module 36, 48 FC layers may be used, which receive an input of size 1024 and output a vector of length 128. For each bottleneck representation, one of the vectors is used to multiply the representation element-wise, while the other is added to the result.

[0055] As shown, the SR module 36 receives three (3) groups of signals. That is, the SR module 36 receives the original two-dimensional representation 46 of the speech mixture 32, the initial separation 54 provided by the ES module 34, and the speaker representation 56 via the feature projection 42. The first layer of the SR module 36 receives all of the initially separated signals concatenated. That is, for each frame, the SR module 36 receives the full filter bank representation size vector multiplied by the number of speakers. The SR module 36 also receives the encoded mixture representation 54. In the case where the system and method are fully causal, the initially separated signals are concatenated to the encoded mixture representation 54 and provided together to the SR module 36. If a look-ahead buffer is implemented, causal alignment may also be used.

[0056] The SR module 36 provides a mask for each speaker estimated. The mask provided by the SR module 36 is applied to the original coded mixture 54 and decoded back into the waveform domain into separate channels by the decoder 28. In addition to the processing of the additional inputs specified above and when causal alignment is utilized, the SR module 36 may contain a TCN with the same dimensionality properties as the ES module 34.

[0057] Although the speaker modules 40 are aligned in time with the initially separated signals 54, the initially separated signals 54 are not aligned when the look-ahead (ie, the mixed representation 54) is employed. In this case, causal alignment may be used to align the initially separated signals 54.

[0058] Lookahead in other causal models can be achieved by converting several layers to non-causal. In a lookahead implementation, the same selection of non-causal layers is used in both separators to process the mixture representation 54. Only the causal layers are used to process the early separated signal, in parallel with the processing of the original mixture 46 by the lookahead layers. When the non-causal portion of the SR module 36 is completed, this representation is added to the representation from the early separated signal 54.

[0059] In one configuration, the SR module 36 begins by processing the early separation signal 54 using a separate convolutional block of the same number of non-causal layers used for the mixing process. When passing through the SR module 36 non-causal blocks, the products of both the mixed signal and the early separation signal are summed and time alignment between the mixing process and the early separation is achieved.

[0060] As described, Figure 3 The system and method utilizes the initial separation results provided by the ES module 34 of the speech mixture 32, the speaker representation 56 together with the feature projections 42 and the initial speech mixture 32 to separate the initial speech mixture 32. The SR module 36 utilizes the aforementioned information to generate a mask at the mask module 38. The generated mask is applied to the original two-dimensional representation 46 of the speech mixture 32 to create a separated representation 48. The separated representation 48 is sent to the decoder 28 to be transformed into a separate speaker waveform or speech estimate 50. The aforementioned process provides improved speech separation performance and low latency.

[0061] Many embodiments have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of the present disclosure. Therefore, other embodiments are within the scope of the appended claims.

[0062] The foregoing description is provided for the purpose of illustration and description. It is not intended to be exhaustive or to limit the present disclosure. The individual elements or features of a particular configuration are generally not limited to that particular configuration, but are interchangeable where applicable and can be used in a selected configuration, even if not specifically shown or described. It can also be varied in many ways. Such variations should not be considered as departing from the present disclosure, and all such modifications are intended to be included within the scope of the present disclosure.

Claims

1. A system for speech separation, comprising: Data processing hardware; memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations comprising: generating a two-dimensional representation of the speech mixture; separating the speech mixture into initial separations; providing the initial separation and speaker representation to a refinement module; refining the initial separation based on the initial separation and the speaker representation; estimating a mask for each speaker; and The mask is applied to the two-dimensional representation to create a two-dimensional per-speaker representation.

2. The system according to claim 1, wherein: Generating a two-dimensional representation of the speech mixture includes passing the speech mixture through an encoder.

3. The system according to claim 2, wherein: The encoder processes the speech mixture using a Fast Fourier Transform (FFT) method.

4. The system according to claim 3, wherein: The FFT is a Short-Time Fast Fourier Transform (STFT) used in conjunction with a learned filter bank.

5. The system of claim 1, further comprising providing at least one feature projection to the refinement module for use by the refinement module in refining the initial separation.

6. The system of claim 1, further comprising refining the speaker representation using a speaker embedding table.

7. The system of claim 1, further comprising passing the two-dimensional per-speaker representation through a decoder to produce a per-speaker waveform.

8. The system according to claim 7, wherein: The decoder converts the two-dimensional per-speaker representation into the per-speaker waveform using an FFT method.

9. The system of claim 1, further comprising a microphone in communication with the data processing hardware and configured to detect the speech mixture.

10. A vehicle comprising a microphone according to claim 9.

Images