Speech separation method and system based on degradation separation estimation technology

Abstract

A speech separation method and system based on the DUET algorithm are disclosed. The method comprises: receiving a signal from a microphone; performing Fourier transform on the received signal (S110); calculating a relative attenuation parameter and a relative delay parameter for each data point (S120); selecting a clustering range of the relative delay parameter based on the distance between the microphones and the sampling frequency of the microphones; clustering the data points within the clustering range of the relative delay parameter into subsets (S130); and performing an inverse Fourier transform on each subset (S140). It provides an efficient and intelligent solution to deploy DUET on software and / or hardware.

Description

Technical Field

[0001] The present disclosure relates to speech processing, and more particularly to a speech separation method and system based on a Degraded Separation Estimation Technique (DUET) algorithm. Background Art

[0002] As the demand for smart lifestyle and connected cars continues to increase, speech separation has become ubiquitous in the industry as a key part of human-computer interaction systems. There are two main methods for speech separation, one of which is to use microphone arrays to achieve speech enhancement, and the other is to use blind source separation algorithms such as frequency domain independent component analysis (FDICA), degraded separation estimation technique (DUET) algorithm or their extended algorithms.

[0003] The DUET algorithm can separate any number of sources using only two mixes, which is very suitable for speech separation in a relatively small space. The technique is effective even when the number of sources is greater than the number of mixes. The DUET algorithm separates speech based on relative delay and attenuation pairs extracted from the mix. However, the appropriate range for clustering relative delays and attenuations in the DUET algorithm is important but vague, because the range is usually selected based on experience, and if there are many invalid data points inside the selected range, the phase wrapping effect may not be negligible. Therefore, methods and systems for selecting an appropriate clustering range to improve speech separation are needed.

[0004] In addition, the DUET algorithm usually requires time synchronization of the source, and traditional time synchronization methods may not meet the requirements because the sampling frequency of the microphone may be as high as tens of kilohertz or higher, while the system time is usually in milliseconds. Therefore, a new method and system are proposed below to achieve more accurate time synchronization. Summary of the invention

[0005] According to one aspect of the present disclosure, a DUET-based speech separation method is provided, the method comprising: receiving a signal from a microphone; performing Fourier transform on the received signal; calculating a relative attenuation parameter and a relative delay parameter for each data point; selecting a clustering range of the relative delay parameter based on the distance between the microphones and the sampling frequency of the microphones; clustering the data points within the clustering range of the relative delay parameter into subsets; and performing an inverse Fourier transform on each subset.

[0006] Typically, the range of the relative attenuation parameter can be set to a constant.

[0007] Usually, the method can be implemented in the head unit of the vehicle. In addition, the method can be implemented in other environments such as indoor environments (e.g., offices, homes, shopping malls), outdoor environments (e.g., automatic service terminals, stations).

[0008] Typically, the step of selecting the range of clusters for the relative delay parameters is also based on the maximum frequency in the speech.

[0009] In general, the range of clustering of the relative delay parameters is related to the relationship between the distance between the microphones and the ratio between the speed of sound and the maximum frequency in speech.

[0010] In general, the clustering range of the relative delay parameters in terms of sampling points can be given by:

[0011]

[0012] where f s is the sampling frequency of the microphone, d is the distance between the microphones, f max is the maximum frequency in speech, c is the speed of sound, and n 0 is the maximum synchronization error of the microphone with respect to a data point.

[0013] Generally, the method may generate a synchronization sound by a speaker to synchronize the signal received by the microphone. The synchronization sound may be generated once or periodically and may be ultrasonic so that it is inaudible to humans. After synchronization, the maximum synchronization error (n 0 ) may be equal to 0.

[0014] According to another aspect of the present disclosure, a DUET-based speech separation system is provided, which includes: a recording module, which is configured to store a signal received from a microphone; a processor, which is configured to perform Fourier transform on the received signal, calculate a relative attenuation parameter and a relative delay parameter for each data point, select a clustering range of the relative delay parameter based on the distance between the microphones and the sampling frequency of the microphones, cluster the data points within the clustering range of the relative delay parameter into subsets, and perform an inverse Fourier transform on each subset.

[0015] The system can be included in the head unit of the vehicle. In addition, the system can be implemented in other environments such as indoor environments (e.g., offices, homes, shopping malls), outdoor environments (e.g., automatic service terminals, stations).

[0016] The system may further include a speaker configured to generate a synchronization signal to synchronize the signal received from the microphone, and the system may further include a synchronization and filtering module configured to synchronize the signal received from the microphone with the synchronization signal and filter out the synchronization signal from the received signal.

[0017] According to the present disclosure, an efficient and intelligent solution can be provided to deploy DUET in software and / or hardware. A solution can also be provided to achieve more accurate time synchronization of signals to be processed by DUET.

[0018] The significance and benefits of the present disclosure will be clear from the following description of the embodiments. However, it should be understood that those embodiments are merely examples of how the invention can be implemented, and the meaning of the terms used to describe the invention are not limited to the specific meanings in which they are used to describe the embodiments.

[0019] After reviewing the drawings and detailed description, other systems, methods, features and advantages of the present disclosure will be or will become apparent to those skilled in the art. It is intended that all such additional systems, methods, features and advantages be included in this description, be within the scope of the present disclosure and be protected by the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The present disclosure may be better understood with reference to the accompanying drawings and descriptions. The components in the accompanying drawings are not necessarily to scale, but rather emphasize the principles of the present disclosure. In addition, in the accompanying drawings, the same reference numerals designate corresponding parts in all different views.

[0021] Figure 1 is a flow chart of a DUET-based speech separation method according to an embodiment of the present disclosure;

[0022] Figure 2A is a schematic diagram illustrating an example of a clustered subset of relative attenuation and relative delay pairs of data points according to an embodiment of the present disclosure, and Figure 2B is a schematic diagram showing an example of a subset of relative attenuation and relative delay pairs of data points where phase wrapping effects occur;

[0023] Figure 3 is a block diagram of a DUET-based speech separation system according to an embodiment of the present disclosure;

[0024] Figure 4A and Figure 4B is a diagram showing the clustering results of the speech of four passengers in a vehicle by using an example of the speech separation system of the present disclosure, wherein Figure 4B yes Figure 4A A top view of

[0025] Figure 5 is a block diagram of a speech separation system according to an embodiment of the present disclosure; and

[0026] Figure 6 is a flow chart of speech separation according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] Hereinafter, preferred embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings. In the following description of the present disclosure, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present disclosure unclear.

[0028] The present disclosure provides a DUET-based speech separation method and system. Figure 1 This is a flow chart of a speech separation method based on DUET. This method can be used in various environments, such as train compartments, offices, homes, shopping malls, automatic service terminals, stations, etc.

[0029] like Figure 1 As shown, microphones (two microphones are shown as an example) receive and sample the sound, which may include multiple sources. The sampling frequency of the microphones can be in the order of kilohertz, tens of kilohertz, or even higher. A higher sampling frequency will facilitate the separation process because less information is lost during the discretization process. If the sound includes multiple sources, the signal sampled by microphone 1 and the signal sampled by microphone 2 will be a mixture of signals each including signals from multiple sources.

[0030] The signals received from microphone 1 and microphone 2 are input to the DUET module ( Figure 1 ), the DUET module performs signal separation (such as Figure 1 (shown in the dashed box).

[0031] First, a Fourier transform (eg, short-time Fourier transform, windowed Fourier transform) is performed on the received signal to output a large number of time-frequency data points (step S110).

[0032] In order to segment the time-frequency data points, relative delay and relative attenuation parameters are calculated for each data point, wherein the relative delay parameter is related to the time difference between the arrival times from the source to the two microphones, and the relative attenuation parameter corresponds to the ratio of the path attenuation between the source and the two microphones (step S120). The relative delay and relative attenuation pairs corresponding to one of the sources should be different from the relative delay and relative attenuation pairs corresponding to the other of the sources, respectively, and thus the time-frequency points can be segmented according to different relative delay-attenuation pairs. That is, the data points within the clustering range of the relative attenuation and relative delay parameters can be clustered into several subsets (step S130). Finally, each subset can be subjected to an inverse Fourier transform (e.g., an inverse short-time Fourier transform) to output separated signals corresponding to different sources (step S140).

[0033] In step S120, the clustering range of the relative attenuation and relative delay parameters is intelligently selected.

[0034] Since the relative attenuation is typically small given the small relative delay required for DUET, the range of relative attenuation can be simply set to a constant, such as [-0.7, 0.7], [-1.0, 1.0]. If the two microphones are set close enough (e.g., about 15 cm), the relative attenuation can be essentially determined by the distance between them.

[0035] For the relative delay, a range needs to be found. When the real relative delay of the signal is within this range, the relative delay can be uniquely determined. Such a range is referred to as an effective range in the present disclosure.

[0036] To clarify the process of determining a valid range for relative delays, the following parameters are defined as follows:

[0037] f s (Unit: Hz): Sampling frequency of microphone;

[0038] f (unit: Hz): frequency of continuous speech signal;

[0039] f MAX (Unit: Hz): Maximum frequency in speech;

[0040] ω (unit: rad / s): frequency of continuous speech signal (ω = 2pf);

[0041] δ (unit: seconds): relative delay between the signals received by the two microphones;

[0042] n (unit: sampling point): relative delay between the signals received by the two microphones with respect to the sampling point;

[0043] d (unit: meter): microphone spacing distance;

[0044] c (unit: m / s): speed of sound.

[0045] If the speech is human speech, f is the frequency of the continuous speech signal; f MAX is the maximum frequency in speech; and ω is the frequency of the continuous speech signal in rad / s.

[0046] The relative delay is set to e -iωδ , which has e -iωδ =e -i(ωδ+2π) Therefore, ωδ can only be uniquely determined when |ωδ|≤π, and if |ωδ|>π, the wrong delay will be returned, and this phenomenon is called the phase wrapping effect.

[0047] Assume that the microphones are synchronized. Then, the valid range of relative delay for a signal of frequency f is given by

[0048]

[0049] and the intersection of the valid ranges of all frequencies in speech is

[0050]

[0051] When the continuous signal is sampled at frequency f s When discretized, the effective range in terms of sampling points becomes

[0052]

[0053] Therefore, if the maximum frequency from any direction is f MAX If the relative delay of speech is within the effective range, the critical point of d is determined as follows:

[0054]

[0055] Maximum frequency f max It may be determined by measurement or may be preset based on the frequency range of the sound of interest.

[0056] when When the effective range is greater than the maximum relative delay between the two microphones, this means

[0057]

[0058] when hour,

[0059]

[0060] Therefore, when When , the selected range is Within this range, there is no phase wrapping effect and the signal of interest of the synchronous microphone will not be located outside this range. That is, if d is small enough, the selected range of the relative delay of the synchronous microphone is

[0061] when hour,

[0062]

[0063] In this case, the selected range of relative delays is When the true relative delay is within this range, there is no phase wrapping effect. Since the effective range is smaller than the maximum relative delay between the two microphones, there may be relative delays within the effective range. If so, phase wrapping effects will occur and their relative delays may be scattered on the axis (see Figure 2B). Some of the shifted data points may fall within the selected range. Nevertheless, those shifted points within the selected range are negligible and do not affect the clustering results of the signals within that range. Therefore, data points outside the valid range will be discarded.

[0064] Therefore, the clustering range of the relative delay parameters of the synchronized microphones with respect to the sampling points is given by:

[0065]

[0066] For non-synchronous microphones, the range selected is,

[0067]

[0068] Where n 0 is the maximum synchronization error measured by the system with respect to the sampling points.

[0069] Figure 2A is a schematic diagram showing an example of a cluster subset of relative attenuation and relative delay pairs of data points within a cluster range calculated by a method according to an embodiment of the present disclosure, and Figure 2B is a schematic diagram showing an example of a subset of relative attenuation and relative delay pairs of data points where phase wrapping effects occur.

[0070] like Figure 2A As shown, there are four subsets of relative attenuation-delay pairs (shown by vertical dashed lines) within the cluster range of relative delays, which means that four sources can be recovered. There is no phase wrapping effect because the relative delays are all within the cluster range.

[0071] If the relative delays of the cross-marked speech move beyond the cluster (e.g., the person corresponding to the subset of cross-marks moves away), a phase wrapping effect will occur, such as Figure 2B The corresponding data points may be scattered on the relative delay axis, but those shifted points will not affect the clustering of signals within the range. Signals outside the range can be discarded.

[0072] The method in the above-mentioned embodiment of the present disclosure can realize speech separation. The method automatically selects the clustering range based on the system setting. In the speech separation process, either there is no phase wrapping effect or the phase wrapping effect can be ignored, and any data points outside the range can be ignored, which ensures the recovery and accuracy of speech separation and makes the calculation more efficient.

[0073] Figure 3 is a block diagram of a DUET-based speech separation system according to an embodiment of the present disclosure.

[0074] One or more of microphones 318 may be considered part of system 300 , or may be considered separate from system 300 . Figure 1 and Figure 3 The number of microphones shown should not be construed as limiting, but is selected for illustration purposes only, and the number of microphones may be greater than two. Microphone 318 senses sounds in the surrounding environment and sends a sampled signal for further processing.

[0075] The system includes a DUET module 312 for performing speech separation and a memory 314 for recording signals received from the microphone. The DUET module 312 may be implemented by hardware, software, or any combination thereof (such as a software program executed by a processor). If the system 300 is included in a vehicle, the DUET module 312 or even the system 300 may be implemented by the vehicle's head unit or a portion thereof.

[0076] DUET module 312 may perform the following operations: Figure 1 The process is shown in the dashed box.

[0077] The system does not require manual adjustment of the clustering range and can be implemented at a relatively low cost and relatively low complexity. In addition, the system can also be adapted to various scenarios, such as carriages, offices, homes, shopping malls, automatic service terminals, stations, etc.

[0078] For illustrative purposes, the embodiments will be described hereinafter by taking a vehicle as an example. Figure 4A and Figure 4B is a diagram showing the clustering results of the speech of four passengers in a vehicle according to an example of the speech separation system of the present disclosure, wherein Figure 4B The chart in Figure 4A Top view of the chart.

[0079] like Figure 4A As shown, the coordinate system includes three axes, namely, a relative delay axis, a relative attenuation axis and a weight axis. The circle at the center of the plane defined by the relative delay axis and the relative attenuation axis is the origin (0, 0). Figure 4B Shows the corresponding Figure 4A , which omits the weight axis.

[0080] In this embodiment, since the human voice frequency is usually within 85-1100 Hz, the maximum frequency f in speech is MAX Set to 1100 Hz. The speed of sound c can be determined based on the ambient temperature and humidity. The sampling frequency f of the microphone s is known, such as 32KHz, 44.1Khz, etc. The maximum synchronization error n of the microphone in terms of sampling points can be automatically measured 0After the microphones are time synchronized, the maximum synchronization error n 0 can be very small or even equal to zero (see reference Figure 5 The DUET module calculates the relative delay range based on equation (9). The relative attenuation range is set to a constant, as shown in the reference Figure 1 described.

[0081] like Figure 4A and Figure 4B As shown, the clustered subsets of relative delay and attenuation pairs correspond to the speech of four passengers. It is possible to determine which subset belongs to which passenger based on the relative phase difference and relative attenuation, and thus the driver's request can be determined. In addition, after setting the range of relative delays according to the method of the present disclosure, the phase wrapping effect does not occur. In addition, since data points outside the range are discarded, the computational cost is reduced.

[0082] In order to reduce or even eliminate the synchronization error of the microphones, the two microphones are controlled to start recording at the same time. However, the software instructions for turning on the microphones may not be executed at the same time, and the system time is accurate to milliseconds, which is much longer than the sampling interval of the microphones. The present disclosure provides a new system for achieving time synchronization of microphones, which is as follows: Figure 5 Illustratively shown.

[0083] Figure 5 is a block diagram of a speech separation system according to an embodiment of the present disclosure. Figure 5 As shown, the system 500 includes a synchronous sounding module 507 for controlling a speaker to generate synchronous sound, a recording module 509 for storing signals received from microphone 1 and microphone 2, a sound synchronization and filtering module 511 for synchronizing the signals from microphone 1 and microphone 2, and a DUET module 513 for speech separation. In various embodiments, the synchronous sounding module 507, the recording module 509, and the filtering module 511 can be implemented by software, hardware, or a combination thereof. For example, they can be implemented by one or more processors.

[0084] The system 500 also includes a speaker 505 for generating a synchronization sound under the control of the synchronization sound module 507. The synchronization sound may be a trigger synchronization sound, which is emitted once after the microphone starts recording. Alternatively, the synchronization sound may be a periodic synchronization sound. In addition, the synchronization sound may be inaudible to humans, such as ultrasound. The synchronization sound may be a pulse signal for easy identification. The speaker 505 may be arranged at a point on a line perpendicular to the line between microphone 1 and microphone 2 and passing through the midpoints of the two microphones, so that the speaker is equidistant from the two microphones.

[0085] The sound mix received from the microphone may include the synchronous sound, speech 1 and speech 2, and is stored in the recording module 509. The sound synchronization and filtering module 511 detects the synchronous signal in the sound mix to synchronize the two sound mixes. Then, the sound synchronization and filtering module 511 removes the synchronous sound from the two sound mixes. The synchronous sound can be removed by a filter or an appropriate algorithm.

[0086] According to this embodiment, time synchronization can achieve microsecond accuracy. For example, if the recording frequency is 44.1KHz, the accuracy of time synchronization may be less than ten microseconds.

[0087] The synchronization signal is input to the DUET module 513 for speech separation. Figure 3 The DUET module 312 is the same as shown. However, it may not be necessary to measure the maximum synchronization error of the microphones in terms of sampling points, and the clustering range of relative delays is calculated by equation (8). In addition, if the distance between the two microphones is small enough, the clustering range of relative delays can be

[0088] Figure 6 is a flow chart of speech separation according to an embodiment of the present disclosure.

[0089] like Figure 6 As shown, the method starts at step S610, where the microphone starts sampling the sound. At step S620, the synchronous sound module 507 controls the speaker to generate a trigger or periodic synchronous sound. At step S630, the received mixed sound, that is, the signal received from the microphone is stored in the memory. The mixed sound is synchronized by using the synchronous sound and then filtering the synchronous sound from the mixed sound (S640), which has been described with reference to the sound synchronization and filtering module 511. The synchronous mixed sound is input to the DUET module 513, and the DUET module 513 performs speech separation (S650) and outputs the separated speech signal (S660). The process of the DUET module 513 has been described with reference to Figure 1 Described.

[0090] The method and system of the above-mentioned embodiments of the present disclosure can achieve synchronization of microphones and thereby improve the accuracy and efficiency of the DUET algorithm at a relatively low cost.

[0091] Those skilled in the art will understand that Figure 1-Figure 6One or more units, processes or subprocesses described may be executed by hardware and / or software. If the process is executed by software or the unit is implemented by software, the software may reside in a software memory (not shown) in a suitable electronic processing component or system and may be executed by a processor. The software in the memory may include executable instructions for implementing logical functions (i.e., "logic" that can be implemented in digital form such as digital circuits or source code, or "logic" that can be implemented in analog form such as analog circuits or analog sources, such as analog electrical signals), and may be selectively embodied in any computer-readable medium for use by or in conjunction with an instruction execution system, device or apparatus. The computer-readable medium may selectively be, for example, (but not limited to) electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or apparatuses, such as RAM, ROM, EPROM, etc.

[0092] With respect to the processes, systems, methods, heuristics, etc. described herein, it should be understood that although the steps of the processes, etc. have been described as occurring according to a particular ordered sequence, the processes may be practiced using the described steps performed in an order other than that described herein. It should be further understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. In other words, the description of the processes herein is provided for the purpose of illustrating certain embodiments and should in no way be construed to limit the claims.

[0093] To clarify use in the pending claims and to provide notice to the public herein, the phrase “ 、 ,......,and <n> At least one of "or"< / n> 、 、......、 <n>or combinations thereof” is defined in the broadest sense by the applicant to mean one or more elements selected from the group consisting of A, B, ..., and N, that is, any combination of one or more of the elements A, B, ..., or N including any one element alone or in combination with one or more of the other elements (which may also include combinations of additional elements not listed).

[0094] Although various embodiments of the present disclosure have been described, it will be apparent to those skilled in the art that many more embodiments and implementations are possible within the scope of the present disclosure.Accordingly, the present disclosure is not to be limited except in accordance with the appended claims and their equivalents.< / n>

Claims

1. A speech separation method based on Degraded Separation Estimation Technology (DUET), comprising: receiving a signal from a microphone; Performing Fourier transform on the received signal; Calculate the relative attenuation parameter and relative delay parameter for each data point; selecting a clustering range of the relative delay parameters based on the distance between the microphones and the sampling frequency of the microphones, clustering the data points within the clustering range of the relative delay parameter into subsets, and Perform an inverse Fourier transform on each subset.

2. The method of claim 1, wherein selecting the cluster range for the relative delay parameter is also based on a maximum frequency in speech. The method of claim 1 , further comprising setting the range of relative attenuation to a constant.

4. The method of any one of claims 1 to 3, wherein the clustering range of the relative delay parameter is given by: where f s is the sampling frequency of the microphone, d is the distance between the microphones, f max is the maximum frequency in speech, c is the speed of sound, and n0 is the maximum synchronization error of the microphone with respect to a data point. The method of claim 1 , further comprising generating a synchronization sound by a speaker to synchronize the received signals.

6. The method of claim 5, further comprising filtering out the synchronization sound from the received signal. The method of claim 5 , wherein the synchronous sound is generated once or periodically.

8. The method of claim 5, wherein the synchronous sound is ultrasonic sound.

9. The method of claim 1 or 5, wherein and when the signals received from the microphones are synchronized, the clustering range of the relative delay parameters is determined by Given, where f s is the sampling frequency of the microphone, d is the distance between the microphones, f max is the maximum frequency in speech and c is the speed of sound.

10. A speech separation system based on Degraded Separation Estimation Technology (DUET), comprising a recording module configured to store a signal received from the microphone; processor, which is configured to Performing Fourier transform on the received signal; Calculate the relative attenuation parameter and relative delay parameter for each data point; selecting a clustering range of the relative delay parameters based on the distance between the microphones and the sampling frequency of the microphones, clustering the data points within the clustering range of the relative delay parameter into subsets, and Perform an inverse Fourier transform on each subset.

11. The system of claim 10, wherein the processor is further configured to select the clustering range of the relative delay parameter based on a maximum frequency in speech.

12. The system of claim 10, wherein the processor is further configured to set the range of relative attenuation to a constant.

13. The system of any one of claims 10-12, wherein the clustering range of the relative delay parameter is given by: where f s is the sampling frequency of the microphone, d is the distance between the microphones, f max is the maximum frequency in speech, c is the speed of sound, and n0 is the maximum synchronization error of the microphone with respect to a data point.

14. The system of claim 10, further comprising a speaker configured to generate a synchronization signal to synchronize the signal received from the microphone.

15. The system of claim 14, further comprising a synchronization and filtering module configured to synchronize the signal received from the microphone with the synchronization signal and to filter the synchronization signal from the received signal.

16. The system of claim 14, wherein the synchronous sound is generated once or periodically.

17. The system of claim 10, wherein the system is implemented in a head unit of a vehicle.

18. The system of claim 10 or 15, wherein and when the signals received from the microphones are synchronized, the clustering range of the relative delay parameters is determined by Given, where f s is the sampling frequency of the microphone, d is the distance between the microphones, f max is the maximum frequency in speech and c is the speed of sound.

19. A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors, perform the following steps: performing a Fourier transform on the signal received from the microphone; Calculate the relative attenuation parameter and relative delay parameter for each data point; selecting a clustering range of the relative delay parameters based on the distance between the microphones and the sampling frequency of the microphones, clustering the data points within the clustering range of the relative delay parameter into subsets, and Perform an inverse Fourier transform on each subset.

Images