Method and system for improving speech separation by eliminating overlap

Abstract

The present invention relates to a method and system for improving speech separation by eliminating overlaps or overlapping points, wherein the time-frequency points of a mixture from two records are separated by using a DUET (203) algorithm. The method or system further eliminates overlapping (204) time-frequency points that do not belong to any of the original sound sources.

Description

Technical Field

[0001] The present invention generally relates to speech separation. More specifically, the present invention relates to a method for improving speech separation by eliminating overlap. The present invention also relates to a system for improving speech separation by eliminating overlap. Background Art

[0002] Nowadays, speech separation is widely used by ordinary users in many occasions, one of the scenarios is, for example, in a car with speech recognition. When more than one person is talking or when there is noise in the car, the car's host cannot recognize the voice from the driver. Therefore, in this case, speech separation is needed to improve speech recognition. There are mainly two well-known speech separation methods. One is to create a microphone array to achieve speech enhancement. The other is to use a speech separation algorithm such as frequency domain independent component analysis (FDICA), degraded separation estimation technique (DUET), or other extended algorithms. Since the FDICA algorithm for separating speech is more complex, the DUET algorithm is usually selected to achieve speech separation.

[0003] However, in the conventional DUET algorithm, some overlapping time-frequency points can be separated into any speech. In this case, one of the separated speech may contain another person's speech, which may cause the separated speech to be not pure enough.

[0004] Therefore, it may be necessary to divide these overlapping time-frequency points into a single cluster to avoid them appearing in the separated speech, so that the quality of the separated speech can be improved. Summary of the invention

[0005] The present invention overcomes some of these disadvantages by providing a method and system for improving speech separation performance by eliminating overlap.

[0006] On the one hand, the present invention provides a method for improving speech separation performance by eliminating overlap. The method comprises the following steps: picking up at least two mixed sounds including a mixed first sound and a second sound by at least two microphones respectively; recording and storing at least two mixed sounds from at least two microphones in a sound recording module; analyzing the two mixed sounds in an algorithm module to separate time-frequency points. Specifically, the algorithm module is configured to apply a degenerate separation estimation technique (DUET) algorithm, and the algorithm module further performs a step of eliminating overlapping points from time-frequency points. Therefore, the first sound and the second sound are respectively restored to the time domain from the time-frequency points from which the overlapping points are eliminated. The overlapping points include the time-frequency points that are neither the first sound nor the second sound. In this way, by using the method provided in this article, the first sound is respectively restored from the time-frequency points belonging only to the first sound, and the second sound is restored from the time-frequency points belonging only to the second sound.

[0007] Specifically, in the method provided herein, eliminating overlapping points includes determining overlapping points according to the rule of |d1 - d2| < d0 / 4, where d1 is the distance between an overlapping point and the center of the first peak, d2 is the distance between the overlapping point and the center of the second peak, and d0 is the distance between the center of the first peak and the center of the second peak.

[0008] On the other hand, the present invention also provides a system for implementing the method to improve speech separation performance by eliminating overlapping. The system includes: at least two microphones for picking up at least two mixed sounds including a mixed first sound and a second sound; a sound recording module for recording and storing at least two mixed sounds from at least two microphones; and an algorithm module configured to analyze the two mixed sounds to separate time-frequency points. Specifically, the algorithm module is configured to apply the Degraded Unmixing Estimation Technique (DUET) algorithm, and the algorithm module further performs the step of eliminating overlapping points from the time-frequency points. Thus, the first sound and the second sound are respectively restored to the time domain from the time-frequency points that respectively belong only to the first sound or the second sound.

[0009] Specifically, in the system provided herein, eliminating overlapping points includes determining overlapping points according to the rule of |d1 - d2| < d0 / 4, where d1 is the distance between an overlapping point and the center of the first peak, d2 is the distance between the overlapping point and the center of the second peak, and d0 is the distance between the center of the first peak and the center of the second peak. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The present invention can be better understood by reading the following description of non-limiting embodiments with reference to the accompanying drawings. In the drawings, the same reference numerals refer to corresponding parts, where:

[0011] Figure 1 is a schematic diagram showing a system for improving speech separation according to an embodiment of the present invention.

[0012] Figure 2 is a flowchart showing a method for improving speech separation according to the embodiment of the present invention.

[0013] Figure 3 is a schematic diagram showing a smoothed weighted histogram of the DUET algorithm according to the embodiment of the present invention. DETAILED DESCRIPTION

[0014] The following discloses a detailed description of embodiments of the present invention; however, it should be understood that the disclosed embodiments are merely exemplary of the present invention that may be embodied in various and alternative forms. The drawings are not necessarily drawn to scale; some features may be exaggerated or minimized to show details of particular components. Therefore, the specific structural and functional details disclosed herein should not be interpreted as limiting, but merely as a representative basis for teaching those skilled in the art to employ the present invention in various ways.

[0015] An object of the present invention is to provide a method for improving speech separation performance by eliminating overlap.

[0016] In one embodiment, Figure 1 The system design diagram of speech separation is shown. For example, two microphones (microphone 1, microphone 2) are turned on at the same time, and the two microphones (microphone 1, microphone 2) are recording, and then two people (person 1, person 2) start talking. Figure 1 As shown, sound 1 belongs to person 1, and sound 2 belongs to person 2. However, in this case, each of the two microphones (microphone 1, microphone 2) picks up a mixed sound including both sound 1 and sound 2. Figure 1 The sound recording module shown is responsible for recording and storing the mixed voice coming in from two microphones (microphone 1, microphone 2). The algorithm module analyzes the mixed voice recorded and stored in the sound recording module and eliminates the overlap from them, and finally, we can get the separated voice 1 and the separated voice 2 respectively from the mixed voice.

[0017] Figure 2 A flow chart is shown, which illustrates a method provided herein for improving speech separation according to an embodiment of the present invention. The method begins at step 201. In step 201, as shown in reference Figure 1 For example, two microphones (microphone 1, microphone 2) are picking up two sounds (sound 1, sound 2) mixed from two people (person 1, person 2).

[0018] In step 202, the mixed sounds picked up by the two microphones (microphone 1 and microphone 2) are recorded and stored in a sound recording module.

[0019] Next, in step 203, the algorithm module analyzes the recorded and stored audio mixture. In the algorithm module, DUET is proposed as the speech separation algorithm of the present embodiment. The DUET algorithm is one of the blind signal separation (BSS) methods, which retrieves source signals from their audio mixtures without prior information about the source signals and the mixing process.

[0020] The DUET blind source separation method works when the sources are W-separated orthogonal, i.e. when the support of the windowed Fourier transform of the signals in the mixture is separable. This DUET algorithm can roughly separate any number of sources using only two mixtures. For anechoic mixtures of decaying and delayed sources, the DUET algorithm allows estimating the mixture parameters by clustering the relative decay-delay pairs extracted from the ratio of the time-frequency representation of the mixture. The estimated values ​​of the mixture parameters are then used to partition the time-frequency representation of one mixture to recover the original source.

[0021] The DUET speech separation algorithm is divided into the following steps:

[0022] ·From Remixx 1 (t) and x 2 (t) Construct time-frequency representation and where x 1 (t) and x 2 (t) is the mixed speech signal.

[0023] Calculate the relative decay delay pair:

[0024]

[0025] Construct a 2D smoothed weighted histogram H(α, δ). Histograms of both direction of arrival (DOA) and distance are formed from the sound mixture observed using two microphones. And then, histogram-based time-frequency masking can be used to achieve signal separation. An example of a histogram is shown in Figure 3 Shown in.

[0026] The histogram is constructed as follows:

[0027]

[0028] The X axis is It refers to relative delay;

[0029] The Y axis is It indicates symmetrical decay, and

[0030] The Z-axis is H(α, δ), which represents weight.

[0031] Locate the peaks and peak centers (Pc_1, Pc_2) in the histogram, which determine the mixture parameter estimates. For example, we use a k-means clustering algorithm to approximate the points in the histogram.

[0032] For each peak center Construct a time-frequency binary mask as follows:

[0033]

[0034] And apply each mask separately to the appropriately aligned mixins like this:

[0035]

[0036] As from Figure 3 As can be seen from the histogram shown, in this embodiment, the application process is performed twice for each of the two peak centers (Pc_1, Pc_2).

[0037] So far, each estimated source time-frequency representation has been divided into each of two peak centers (Pc_1, Pc_2), which can be converted back to the time domain to obtain separated sound 1 and sound 2.

[0038] However, recorded source mixes are usually not orthogonal to separation. In this embodiment, it is assumed that, for example, only two people are talking at the same time. According to the rule, the time-frequency point is divided into two parts by non-zero or one. If some time-frequency points between two peaks are not orthogonal to W separation, and these time-frequency points are mixed from the voices of two people (person 1, person 2). In the present invention, these time-frequency points are defined as overlapping points. In this case, due to the existence of these overlapping time-frequency points, one of the separated voices may contain the voice of another person, which means that the separated sound 1 may also contain sound 2, and cause the separated voice to be not pure enough. In fact, the overlapping time-frequency points of the mixed two people's voices do not belong to any of the people. Overlapping points should be classified into the third category to be eliminated.

[0039] In order to solve the above technical problems, the present invention provides a method for improving speech separation performance by eliminating overlap, wherein overlapping time-frequency points are found and divided into a single cluster, and they will not appear in the separated speech. Therefore, the quality of the separated speech can be improved.

[0040] Specifically, Figure 2 As shown in step 204, the method of finding these overlapping time-frequency points is taken as an example. Figure 3 , we calculate the first distance d1 between the time-frequency point Pt_r and the first peak center Pc_1, then calculate the second distance d2 between the time-frequency point Pt_r and the second peak center Pc_2, and finally calculate the distance d0 between the first peak center Pc_1 and the second peak center Pc_2, that is, calculate |d1-d2|. When |d1-d2| is less than the threshold, the time-frequency point Pt_r can be determined as an overlapping point. That is, when the difference between the first distance d1 and the second distance d2 is less than the threshold, the overlapping point can be determined. In this embodiment, the threshold can be set to one-fourth of the distance d0 between the two peak centers (Pc_1, Pc_2). In other words, when the time-frequency point meets this requirement:

[0041]

[0042] It can be determined that the time-frequency point (Pt_r) does not belong to Figure 3 Any of the two peaks in , and can be identified as overlapping time-frequency points. These overlapping time-frequency representations are not converted back to the time domain. Overlapping points can be found by traversing all the time-frequency points, such as Figure 3 shown.

[0043] Finally, in Figure 2 In step 205, the overlapping points selected from the time-frequency points are eliminated, and the remaining time-frequency points separated into each of the two persons are converted into the time domain to restore the original source with the separated sound 1 and sound 2. The method is completed at step 206.

[0044] Another object of the present invention is to provide a system for improving speech separation performance by eliminating overlap.

[0045] In such Figure 1 In the illustrated embodiment, the system for improving speech separation includes two microphones (microphone 1, microphone 2) which are turned on simultaneously and record the mixed speech signals of two persons (person 1, person 2). Figure 1 , sound 1 belongs to person 1, and sound 2 belongs to person 2. However, in Figure 1 In this case, each of the two microphones (microphone 1, microphone 2) picks up a mixed sound including both sound 1 and sound 2. Figure 1 The sound recording module shown is responsible for recording and storing the mixed voices coming in from two microphones (microphone 1, microphone 2). In order to obtain separated voices 1 and 2 respectively from the mixed voices, the system also includes an algorithm module, which uses the DUET algorithm to analyze the mixed voices recorded and stored in the sound recording module and eliminate the overlaps therefrom, and finally, we can obtain separated voices 1 and 2 respectively from the mixed voices.

[0046] As described above, the method and system provided herein eliminate the overlap existing in the separated speech signals, thereby improving the quality of speech separation. It will be appreciated by those skilled in the art that the signals picked up by the microphone of the present invention are not limited to two, but can be extended to any number of mixed signals. The algorithm processed in the method and system herein can be executed iteratively.

[0047] As used in this application, an element or step described in the singular and followed by the word "a or an" should be understood not to exclude a plurality of said elements or steps, unless such exclusion is specified. In addition, reference to "one embodiment" or "an example" of the present disclosure is not intended to be interpreted as excluding the existence of additional embodiments that also include the features. The terms "first", "second", and "third" etc. are used only as labels and are not intended to impose numerical requirements or specific positional order on their objects.

[0048] Although exemplary embodiments are described above, these embodiments are not intended to describe all possible forms of the present invention. In fact, the wording used in the specification is for description rather than limitation, and it should be understood that various changes can be made without departing from the spirit and scope of the present invention. In addition, the features of the various embodiments realized can be combined to form other embodiments of the present invention.

Claims

1. A method for improving speech separation performance by eliminating overlap, comprising the following steps: Picking up at least two mixed sounds including a mixed first sound and a mixed second sound by at least two microphones respectively; recording and storing the at least two mixed sounds from the at least two microphones in a sound recording module; analyzing in an algorithm module the at least two sound mixtures for respectively restoring the first sound and the second sound, The algorithm module further comprises the following steps: Eliminating overlapping points from the time-frequency points, wherein the overlapping points are found among the time-frequency points, the overlapping points include the time-frequency points that are both the first sound and the second sound, and wherein each of the overlapping points is determined when a difference between a first distance and a second distance is less than a threshold, wherein the first distance is a distance from one of the time-frequency points to be determined to a first peak center, and the second distance is a distance from the same time-frequency point to be determined to a second peak center; and The time-frequency points respectively associated with the first sound and the second sound are separated to eliminate the overlapping points. 2 . The method of claim 1 , wherein the threshold is set to one quarter of the distance between the first peak center and the second peak center. 3 . The method of claim 1 , wherein the overlapping point is determined by traversing all the time-frequency points respectively associated with the first sound and the second sound.

4. The method of claim 1, wherein analyzing the at least two mixes comprises executing a Degraded Separation Estimation Technique (DUET) algorithm. 5 . The method of claim 1 , wherein restoring the first sound and the second sound comprises converting the time-frequency points with the overlapping points eliminated back to the time domain.

6. The method of claim 1, wherein the method can be implemented in any situation where more than one person is speaking at the same time.

7. A system for improving speech separation performance by eliminating overlap, comprising: at least two microphones, the at least two microphones being adapted to pick up at least two mixed sounds including a mixed first sound and a second sound, respectively; a sound recording module adapted to record and store the at least two mixed sounds from the at least two microphones; an algorithm module adapted to analyze the at least two sound mixtures for respectively restoring the first sound and the second sound, The algorithm module is further configured as follows: Eliminating overlapping points from the time-frequency points, wherein the overlapping points are found among the time-frequency points, the overlapping points include the time-frequency points that are both the first sound and the second sound, and wherein each of the overlapping points is determined when a difference between a first distance and a second distance is less than a threshold, wherein the first distance is a distance from one of the time-frequency points to be determined to a first peak center, and the second distance is a distance from the same time-frequency point to be determined to a second peak center; and The time-frequency points respectively associated with the first sound and the second sound are separated to eliminate the overlapping points.

8. The system of claim 7, wherein the threshold is set to one quarter of the distance between the first peak center and the second peak center.

9. The system of claim 7, wherein the overlapping point is found by traversing all the time-frequency points respectively associated with the first sound and the second sound.

10. The system of claim 7, wherein the algorithm module for analyzing the at least two mixes implements a Degraded Separation Estimation Technique (DUET) algorithm.

11. The system of claim 7, wherein the first sound and the second sound are restored by converting the time-frequency points with the overlapping points eliminated back to the time domain.

12. The system of claim 7, wherein the system can be used in any situation where more than one person is speaking at the same time.

13. A non-transitory computer-readable storage medium comprising instructions, When the instructions are executed by a processor, the processor is configured to perform the steps of the method according to any one of claims 1 to 6.

Images