Method and system for separating the voice component and the noise component in an audio flow

Abstract

The invention relates to a method and a system for separating in real time in an audio stream the voice component and the noise component.

Description

[0001] The invention relates to a method and a system for separating, in real time in an audio stream, the part of the stream associated with a voice or speech, from another part of the stream containing noises.

[0002] The invention finds its application in a context where one or more people are speaking in a noisy environment (background noise, engine noise, ventilation, etc.). The speech signal superimposed on the noisy signals is digitized into an audio stream by a sound sensor.

[0003] The invention also relates to a method and a system for enhancing a voice signal in real time in an audio stream from a delayed-time audio source separation method.

[0004] The applicant's known state of the art is divided into two categories: so-called classical approaches and possible approaches using artificial intelligence known by the Anglo-Saxon term "deep learning".

[0005] In the "deep learning" approach, some approaches deal directly with the problem of voice / background noise separation, others concern signal / signal, voice / voice separation.

[0006] US patent application 20190066713 discloses a method for obtaining, by means of a device, a combined sound signal from combined signals originating from multiple sound sources in an area where a person is located. The processing employed uses deep neural networks.

[0007] An example of a prior art method for separating multiple voices in an audio signal includes the steps described below, which are not shown for the sake of simplicity. The incoming audio signal is denoted X, its length L. The signal is transmitted to an encoder M 1 which transforms X in a tensor X (1)< of dimensions F × T Or Tis a divisor of L And F a number of filters specified by the designer. The encoder M 1 consists of a 1D Convolution to F Filters. The coefficients of the convolution kernels are tuned during a learning phase. The tensor is passed on one side to a multiplier for future use and on the other side to a separation module. The separation module is divided into two sub-modules. M 2 and M 4. The first sub-module M 2 transforms the tensor X (1)< in a tensor X (2)< of dimensions F × T The first sub-module M 2 consists of a normalization layer, a 1x1 convolution and a stacking of 1D-Conv modules known from the prior art and whose parameters are set during a learning phase.

[0008] The second sub-module M 4 transforms X (2)< in X(4)< dimension tensor 2F × T. For this purpose, the second sub-module M 4 combines a non-linearity, a 1x1 convolution, and a sigmoid function. The coefficients of the 1x1 convolution are set during a learning phase.

[0009] X (1)< is concatenated to itself to form a tensor of dimensions 2F x T which is multiplied by X (4)< to form X (5)< .

[0010] The module M 5 takes as its entry X (5)< and outputs two signals of length L by means of a 1D deconvolution whose parameters are set during a learning phase.

[0011] The numerical parameters defining the processing of the different modules are obtained in a preliminary learning phase on a database.

[0012] By replacing one of the voices with noise, it is immediately possible to use the methods described in the state of the art to separate the voice from the background noise in an audio signal and, by keeping only the output containing the voice signal, to enhance the voice of a noisy signal.

[0013] There figure 1 illustrates an application to the separation of signals of different types, by separating the voice channel and the noise channel.

[0014] As described, the state of the art does not directly allow for real-time processing of an audio stream.

[0015] The document by Mimilakis Stylianos Ioannis et al, entitled "A recurrent encoder-decoder approach with skip-filtering connections for monaural singing voice separation", dated September 25, 2017, pages 1-6, XP 033263882, discloses a method for separating the voice from a musical background.

[0016] In the technical field of "deep learning," data is represented as tensors. The data is modified by a succession of modules. At the output of each module, the data is projected into an abstract space generally defined by its dimensions.

[0017] To achieve this, the present invention implements the following processing: The signal (input stream) X ) is divided into N frames of length L, with X N the nth frame. The process performs the following operations: The frame X N is encoded by a 1D convolutional network. The result is a tensor X N 1 of dimensions F x T, where F is the number of filters given by the designer, T is a divisor of L depending on the size of the filters F, and 100 is the result. X N 1 is then transformed by a module M 2 , 101. The result X N 2 is a tensor of dimensions F x T. The modulus M 4 estimates, 103, from X N 2 a tensor X N 4 of dimensions 2F x T. X N 1 is concatenated to itself, 104, to form a tensor of dimensions 2F x T which is multiplied by X N 4 to form X N 5 The M5 module, starting from X N 5 produces a tensor of dimension 2 × T, 105, from which two outputs of dimension 1 × T are obtained X N, 0 and X N,1 which are respectively the voice channel and the noise channel.

[0018] These steps are repeated for each new frame. The parameters are learned from a sound database. The drawback of this method is that it does not use information from previous frames to process the current frame. This results in degraded quality and high latency in processing, due to the length of the frames.

[0019] One of the objectives of the present invention is to provide a method and a device for separating voices from background noise in an audio stream in real time, or for denoising voices in an audio stream, particularly by taking into account information from previous frames. This improves performance and reduces processing latency. The method thus enables the propagation of "global information" about the signal, its updating, and its processing from frame to frame. The invention is as defined in independent claims 1 and 6. Preferred aspects of the invention are defined in the dependent claims.

[0020] The invention relates to a method for separating voice from noise in an audio signal received on a receiver equipped with an audio sensor, characterized in that it comprises at least the following steps: The received audio stream is separated into N frames. X N , For each frame X N A tensor containing information about the entire audio stream is associated with it. The frame is then transmitted. X N to a first module M1 which generates a signal X N 1 , The tensor I N- 1 obtained during the previous iteration for frame processing X N - 1 is transmitted to an M3 module, such that I N = I N- 1 + λ ( A N - I N-1 ), λ being a factor in gain and A N being a tensor, in order to generate a signal X N 2 The M3 module takes the signal as input X N 2 , result of signal transformation X N 1 by a module M 2 and performs the concatenation of X N 2 And I N- 1 in order to generate a signal X N 3 of dimension 2F × T, The signal X N 3 is transmitted to an M4 module in order to generate a signal X N 4 which is combined with the signal X N 1 The signal resulting from the combination is decoded by an M5 decoder in order to generate a first voice signal X N, 0 and a second signal X N, 1.

[0021] To process a frame N we assume that the plot N - 1 has been processed previously and the resulting quantities have been stored. For frame 0, I 0 is fixed arbitrarily, for example it is identically zero.

[0022] The process may also include frame encoding X N by a 1D convolution network, which provides the tensor X N 1 .

[0023] The tensor I N and the tensor A N can be of dimension F × F.

[0024] The tensor A N can be defined by: A N = X N 2 ⋅ X N 2 t T , X N 2 ⋅ X N 2 t being the matrix product of X N 2 and its transposition. X N 3 may correspond to the concatenation of X N 2 And C N of dimension F × T, C N being defined by C N = B N ⋅ X N 2 and representing the matrix product between B N And X N 2 , with B N = Softmax ( I N -1).

[0025] The invention also relates to a device for separating voice from noise in an audio signal received by a receiver equipped with an audio sensor, said audio signal X having a length L. Advantageously, the receiver is configured to separate the received audio stream into N frames. X N , and to associate, with each frame X N , a tensor I N containing information about the entire audio stream, the receiver includes: A first module M 1 (100) adapted to receive several frames of a signal containing voice and noise and, for each frame received, to generate a signal X N 1 and transmit to a third module M3, a tensor I N-1 obtained during the previous iteration for frame processing X N - 1, such that I N = I N- 1 + λ ( A N - I N -1 ), λ being a factor in gain and A N being a tensor, a second module M2(101) configured to transform the signal X N 1 , in order to generate a signal X N 2 The third module, M3, is configured to concatenate the signal X N 2 and the tensor I N- 1 in order to generate a signal X N 3 of dimension 2F × T, where F is a given number of filters and T is a divisor of the length L, and transmit the signal X N 3 to a fourth module M4, the fourth module M4 (102) being configured to generate a signal X N 4 which is combined with the signal X N 1 , the device also includes an M decoder 5 (105) configured to decode the signal resulting from the combination, in order to generate a first voice signal X N, 0 and a second signal X N, 1 .

[0026] Other features, details and advantages of the invention will become apparent from the description provided with reference to the accompanying drawings, which are given by way of non-limiting example and represent, respectively: [ Fig.1 ], an illustration of earlier art, [ Fig.2 ], an example of a system enabling the implementation of the process according to the invention, [ Fig.3 ] an illustration of the steps implemented by the process according to the invention.

[0027] There figure 2 illustrates an example of a device enabling the implementation of the process according to the invention.

[0028] The signal from which the voice(s) must be extracted (separated) from the noise contained in the audio stream is received by an audio sensor 10. The audio sensor is connected to a set of hardware equipment or modules 20 configured to separate the voice from the noise, which will be detailed in the figure 3 .

[0029] There figure 3 illustrates a first implementation variant for separating a voice from noise in an audio signal, with processing performed at the level of assembly 20. This separation is performed in real time. The modules are similar to the schematic of the figure 1 They bear the same references. The set also includes an M3 module whose function is detailed below.

[0030] The audio signal received by the sensor is, in a first step, separated into N frames. X 1 .....X N . Each frame X N is associated with a tensor I N which is of constant dimension, independent of the frame index. The process will update the value of the tensor I N frame by frame and the combined use of X N And I N to estimate X N, 0 and X N, 1.

[0031] The plot X N is transmitted to a first module M 1 , 100, which generates a signal X N 1 . The tensor I N- 1 obtained during the previous step for frame processing X N - 1 is transmitted in an M 3, 201 module.

[0032] M 3 generates a tensor I N , 202, which will be used during frame processing X N+ 1.

[0033] The M3 encoder takes a signal as input X N 2 , 203, result of the signal transformation X N 1 by a module M 2 and performs the concatenation of X N 2 And I N , in order to generate a signal X N 3 of dimension 2F × T, M 3 : X N 2 I N − 1 − > X N 3 I N .

[0034] The signal X N 3 , 204, is transmitted to an M4 module in order to generate a signal X N 4 which is combined, 104, with the signal X N 1 , the signal resulting from the combination is decoded by an M 5, 105 decoder, in order to generate a first voice signal X N, 0 and a second noise signal X N, 1.

[0035] In one embodiment, the steps implemented by the process according to the invention are as follows: For all N, I N is of dimension F × F

[0036] A N is a tensor F × F defined by A N = X N 2 ⋅ X N 2 t T has. X N 2 ⋅ X N 2 t is the matrix product of X N 2 and its transposition

[0037] I N = I N- 1 + λ ( A N - I N -1) with λ a gain factor of 0 and 1 given by the user B N = Softmax I N − 1 a. The softmax function is classic in machine learning; to a vector of K numbers, ( v 1 ... v K ) it associates a vector of K number ( w 1 ... w K) with for all w k = exp v k ∑ l = 1 K exp v l , b. To calculate B N , the softmax function is applied independently to all rows of I N , CN=BN⋅XN2 is the matrix product between BN And XN2 its dimensions are F × T, XN3 is of dimension 2F × T, it is the concatenation of XN2 And CN.

[0038] The method and device according to the invention allow for the separation of voice from noise in an audio signal received on a sensor in real time and without degrading the parameters specific to the voice.

[0039] The numerical parameters defining the processing of the different modules are set in a preliminary learning phase on a database.

[0040] The invention allows real-time operation with a controllable latency / quality compromise, does not degrade the audio signal which does not contain noise, and allows the noise in a signal which does not contain speech (voice) to be enhanced.

[0041] The process allows, in particular, the preprocessing of the audio signal of speech to improve the quality of voice processing / enhancement building blocks (compression, analysis).

[0042] Adding an M3 module to the processing chain improves the quality of implementing a frame-by-frame strategy for real-time processing.

Claims

1. A method for separating in real-time voice from noise in an audio signal X received on a receiver equipped with an audio sensor, said audio signal X having a length L, characterised in that it contains at least the following steps: - the received audio stream is separated into N frames XN, - a tensor IN containing information on the entirety of the audio stream is associated with each frame XN, - each frame XN is transmitted to a first module M1, (100), configured to generate a signal X N 1 and a tensor IN-1, obtained in the preceding iteration of processing the frame XN - 1, is transmitted to a module M3, (201), such that IN = IN-1 + λ(AN - IN-1), with λ a gain factor and AN a tensor, - the signal X N 1 is transformed by a module M2 in order to generate a signal X N 2 , - the module M3 concatenates X N 2 and the tensor IN-1 in order to generate a signal X N 3 of 2F × T size, with F a given number of filters and T a divisor of the length L, - the signal X N 3 , (204), is transmitted to a module M4 configured to generate a signal X N 4 that is combined, (104), with the signal X N 1 , - the signal resulting from the combination is decoded by a decoder M5, (105), in order to generate a first voice signal XN,0 and a second signal XN,1.

2. The method according to claim 1, comprising a step of encoding frame XN via a 1D convolution network, thereby providing the tensor X N 1 .

3. The method according to any one of claims 1 and 2, wherein the tensor IN and the tensor AN are of F × F size.

4. The method according to claim 3, wherein the tensor AN is defined by: A N = X N 2 ⋅ X N 2 t T , X N 2 ⋅ X N 2 t being the matrix product of X N 2 and its transpose.

5. The method according to any one of claims 1 to 4, wherein X N 3 corresponds to the concatenation of X N 2 and CN of F × T size, with CN defined as C N = B N ⋅ X N 2 and representing the matrix product of BN and X N 2 , with BN = Softmax(IN-1).

6. A device for separating voice from noise in an audio signal received on a receiver equipped with an audio sensor, said audio signal X having a length L, characterised in that the received is configured to separate the received audio stream into N frames XN, and to associate, to each frame XN, a tensor IN containing information on the entirety of the audio stream, the receiver comprising: - a first module M1 (100) adapted to receive several frames of a signal containing voice and noise and to, for each received frame, to generate a signal X N 1 and transmit, to a third module M3, a tensor IN-1 obtained in the preceding iteration of processing the frame XN - 1, such that IN = IN-1 + λ(AN - IN-1), with λ a gain factor and AN a tensor, - a second module M2 (101) configured to transform the signal X N 1 in order to generate a signal X N 2 , - the third module M3 being configured to concatenate X N 2 and the tensor IN-1, in order to generate a signal X N 3 of 2F × T size, with F a given number of filters and T a divisor of the length L, and to transmit the signal X N 3 to a fourth module M4, - the fourth module M4 (102) being configured to generate a signal X N 4 that is combined with the signal X N 1 , the device further comprising a decoder M5 (105) configured to decode the signal resulting from the combination, in order to generate a first voice signal XN,0 and a second signal XN,1.

7. The device according to claim 6, wherein the module M1 is configured to encode frame XN via a 1D convolution network, thereby providing the tensor X N 1 .

8. The device according to any one of claims 6 and 7, wherein the tensor IN and the tensor AN are of F × F size.

9. The device according to claim 8, wherein the tensor AN is defined by A N = X N 2 ⋅ X N 2 t T , X N 2 ⋅ X N 2 t being the matrix product of X N 2 and its transpose.

10. The device according to any one of claims 6 to 9, wherein X N 3 corresponds to the concatenation of X N 2 and CN of F × T size, with CN defined as C N = B N ⋅ X N 2 and representing the matrix product of BN and X N 2 , with BN = Softmax(IN-1).

Images