Method and device for processing binaural recordings

Abstract

The present invention relates to a method and device for processing a first audio signal and a second audio signal representing an input binaural audio signal acquired by a binaural recording device. The present invention further relates to a method for rendering the binaural audio signal on a speaker system. The method for processing the binaural signal includes the steps of extracting audio information from the first audio signal, calculating band gains to reduce noise in the first audio signal, and applying the band gains to respective frequency bands of the first audio signal according to a dynamic scaling factor to provide a first output audio signal, where the dynamic scaling factor has a value between 0 and 1 and is selected to reduce quality degradation of the first audio signal.

Description

[Technical field]

[0001] The present invention relates to a method and device for processing a binaural audio signal. [Background technology]

[0002] In both the fields of user-generated content (UGC) and professional-generated content (PGC), binaural capture devices are often used to capture audio. Binaural audio is recorded, for example, by a pair of microphones that are respectively provided in a pair of earbuds worn by a user. Thus, the binaural capture device captures sound at each ear of the user wearing the binaural capture device. Thus, binaural capture devices are generally good at capturing the sound of a user or the audio perceived by a user. Thus, binaural capture devices are often used to record podcasts, interviews, or meetings.

[0003] A disadvantage of binaural capture devices is that they are very sensitive to environmental noise, which results in a poor playback experience when the captured binaural signal is rendered.

[0004] Another drawback of binaural capture devices is that audio sources of interest other than the voice of the user wearing the binaural capture device are picked up with very low signal strength, high noise and high reverberation, resulting in reduced intelligibility of other audio sources of interest that feature in the captured binaural audio signal.

[0005] To avoid these drawbacks, previous solutions use complex audio processing algorithms that are computationally intensive to perform, making these solutions particularly difficult to implement for low latency communications or UGC, where complex audio processing is difficult to implement. Summary of the Invention

[0006] Based on the above, it is therefore an object of the present invention to provide a method and a device for more efficient processing of binaural audio signals, together with a method for rendering the processed binaural audio signal.

[0007] According to a first aspect of the present invention, According to claim 1, A method is provided for processing a first audio signal and a second audio signal representative of an input binaural audio signal. The method of the first aspect of the invention is as set out in claim 1.

[0008] The present invention according to a first aspect is based at least in part on the realization that by dynamically scaling band gains of frequency bands, quality degradation of an output audio signal may be reduced. Regardless of the type of noise reduction method employed to calculate the noise reduction band gains, the audio signal to which the band gains have been applied contains undesirable audio artifacts introduced by the noise reduction process. To reduce these audio artifacts, the band gains are dynamically applied according to a dynamic scaling factor. A static or predetermined scaling factor cannot reduce the quality degradation for most of the possible audio signals by implementing the band gains to such a high degree that audio artifacts appear or to such a low degree that noise reduction is suppressed. The selection of the dynamic scaling factor may be based on the audio information and / or band gains of the audio signal, so as to allow the use of dynamic (non-static) scaling factors that are adjusted after a particular audio signal has been processed.

[0009] In some implementations, the dynamic scaling factor for each frequency band is based on a band gain associated with the corresponding frequency band in the current and previous time frames of the first audio signal.

[0010] A time frame means a partial time segment of the first audio signal. Thus, by analyzing the band gain for each frequency band of the current time frame and the previous time frame, the dynamic scaling factor is dynamically adjusted for the current first audio signal being processed. Thus, the dynamic scaling factor is optimized to provide a first output audio signal with reduced quality degradation.

[0011] In some implementations, the method further includes processing an additional audio signal from an additional recording device by synchronizing the additional audio signal with the binaural audio signal and providing an additional output audio signal based on the additional audio signal.

[0012] The additional recording device may be any device capable of recording at least a mono audio signal. The additional recording device may be, for example, the user's smartphone. The additional audio signal may enhance the audio from the user wearing the binaural recording device or from a second source of interest. The binaural recording device is prone to pick up noise and reverberation from the surroundings, and is therefore not suitable for recording audio from a source of interest other than the user wearing the binaural recording device, for example an interviewee having a conversation with the user. To this end, an additional recording device that records an additional audio signal may be employed and used as a microphone to record audio from the second source of interest. The additional audio signal is synchronized with the binaural signal, and the binaural signal combined with the synchronized additional audio signal may, for example, facilitate clearer dialogue reproduction.

[0013] Some implementations further include processing a bone vibration sensor signal acquired by a bone vibration sensor of the binaural recording device. A source of the detected sound may be determined based on the VAD probability and the bone vibration sensor signal by synchronizing the bone vibration sensor signal with the binaural audio signal and extracting a VAD probability of the additional audio signal. If the source is a wearer of the binaural recording device with the bone vibration sensor, the additional audio signal is processed with a first audio processing scheme. If the source is other than a wearer of the binaural recording device with the bone vibration sensor, the additional audio signal is processed with a second audio processing scheme. Processing the additional audio signal using different processing schemes may allow adaptive switching of gain levels and / or noise reduction processing depending on the source of the detected sound. This adaptive switching of audio processing schemes may be combined with the dynamic processing described above or may be implemented in other general forms of audio processing and / or noise reduction methods.

[0014] For example, as a second aspect of the present invention, a method for processing a first audio signal, a second audio signal and an additional audio signal is provided, where the first audio signal and the second audio signal represent an input binaural audio signal acquired by a binaural recording device, and the additional audio signal is recorded by the additional recording device. The method includes the steps of synchronizing the additional audio signal with the binaural audio signal, receiving a bone vibration sensor signal acquired by a bone vibration sensor of the binaural recording device, and also synchronizing the bone vibration sensor signal with the binaural audio signal. Furthermore, the method includes the steps of extracting a VAD probability of the additional audio signal, and determining a source of the detected sound based on the VAD probability and the bone vibration sensor signal. If the source is a wearer of the binaural recording device with the bone vibration sensor, the additional audio signal is processed using a first audio processing scheme. If the source is other than a wearer of the binaural recording device with the bone vibration sensor, the additional audio signal is processed using a second audio processing scheme. Additionally, an additional output audio signal is provided based on the processed additional audio signal, and a first output audio signal and a second output audio signal are provided based on the first audio signal and the second audio signal from which the binaural output audio signal is determined.

[0015] Providing the first output audio signal and the second output audio signal may include performing audio processing on the first audio signal and the second audio signal in accordance with an aspect of the present invention and / or performing other forms of audio processing, such as noise cancellation and / or equalization.

[0016] According to a third aspect of the present invention, According to claim 20, An audio processing device is provided.

[0017] The invention according to the second or third aspect features the same or equivalent embodiments and advantages as the invention according to the first aspect. Furthermore, any functionality described with respect to a processing method may have corresponding components featured in a processing device or corresponding code for performing such functionality in a computer program product. [Brief description of the drawings]

[0018] The invention will now be described in more detail with reference to the accompanying drawings, which show embodiments of the invention according to a first or second aspect. [Figure 1] 1 illustrates an exemplary binaural recording device and an additional recording device. [Diagram 2] 1 illustrates a binaural processing device according to some implementations. [Diagram 3] 1 is a flowchart illustrating a method for processing a first audio signal and a second audio signal according to an implementation of the present invention. [Figure 4a] 11 is a flow chart illustrating an alternative method for applying band gains according to dynamic scaling factors. [Figure 4b] 11 is a flow chart illustrating another alternative method for applying band gains according to dynamic scaling factors. [Diagram 5] 1 illustrates frequency bands for a series of time frames representing an audio signal. [Figure 6] 1 is a flowchart illustrating side and intermediate signal estimation and processing according to some implementations. [Figure 7] 4 is a flow chart illustrating a rendering method according to an aspect of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] FIG. 1 shows a user 4 wearing a binaural recording device 1. The binaural recording device 1 may include two wired (not shown) or wireless microphone pairs 2a, 2b, optionally provided in respective earpieces of a headset. The binaural recording device 1 records a binaural audio signal including two audio signals, e.g., a left audio signal and a right audio signal emanating from a left microphone 2a and a right microphone 2b in the respective earpieces. In some implementations, an additional recording device 31 records the additional audio signal and / or a bone vibration sensor 11 records the bone vibration signal. For example, the additional recording device 31 may be a microphone provided in a user device 3 (e.g., a smartphone, tablet or laptop), and the bone vibration sensor 11 may be provided as an integrated part of the binaural recording device 1 (e.g., integrated in the earpieces as shown) or may be provided externally (not shown). The additional recording device 31 may record a second sound source of interest, such as a second person having a conversation with the user 4. Alternatively, the additional recording device 31 may record the voice of the user 4 .

[0020] The bone vibration sensor signal from the bone vibration sensor 11 may indicate whether a user 4 wearing the binaural recording device 1 is speaking and / or the bone vibration sensor signal may be used to extract audio. Further, the bone vibration sensor signal may be used in conjunction with the first audio signal and / or the second audio signal to extract enhanced audio information.

[0021] The first and second audio signals recorded by the binaural recording device 1 may be synchronized in time by a binaural processing device 32 optionally provided in the user device 3, and the additional audio signal and / or bone vibration sensor signal may be synchronized with the binaural audio signal by the binaural processing device 32. In some implementations, the additional audio signal and / or bone vibration sensor signal are synchronized in time by the binaural processing device 32 using a software implementation. For example, the synchronization between the binaural audio signal and the additional audio signal and / or bone vibration sensor signal is achieved by the processing device determining a delay between the signals that is characterized by a maximum correlation between the signals. Alternatively, each recorded data block or time frame representing a portion of the binaural audio signal and the additional audio signal and / or bone vibration sensor signal is associated with a timestamp, and the signals are synchronized by comparing the timestamps of each block.

[0022] In addition to signal time synchronization, any audio processing described below may be performed by the binaural processing device 32. The binaural processing device 32 may be provided in whole or in part in the binaural recording device 1 and in wired or wireless (e.g., Bluetooth) communication with the binaural recording device 1. For example, the binaural processing device 32 of the user device 3 may receive, synchronize and process all audio signals from the binaural recording device 1, any bone vibration sensors 11 and any additional recording devices 31.

[0023] 2, a binaural processing device 32 according to some implementations is shown. The binaural processing device 32 is configured to receive two audio signals recorded by the binaural recording device 1, e.g. a binaural audio signal including a left audio signal L and a right audio signal R. In a synchronization module 321, the two audio signals L, R are synchronized. In some implementations, the synchronization module 321 is integrated into the binaural recording device 1, and further processing steps such as synchronization with any bone vibration signals and / or additional audio signals are performed by the user device 3.

[0024] The synchronization module 321 outputs the synchronized audio signal to an optional conversion module 322. The optional conversion module 322 may extract audio information and / or alternative representations of the synchronized audio signals L, R. The alternative representations of the audio signals (referred to as A1 and B1) are provided to respective processing modules 323a, 323b. Each processing module 323a, 323b is configured to perform audio processing including noise reduction of the audio signal representations A1, B1. In some implementations, the processing modules 323a, 323b perform processing equivalent to a first processing sequence and a second processing sequence described below.

[0025] The processed audio signals A2, B2 output by the signal processing modules 323a, 323b are provided to an inverse transformation module 324 that performs an inverse transformation to reproduce processed audio signals PL, PR corresponding to the audio signals received in the optional transformation module 322. In some implementations, the transformation module 322 and the inverse transformation module 324 are not used and the two audio signals L, R of the binaural recording device are processed in their original format.

[0026] The output stage 325 combines the first output audio signal PL and the second output audio signal PR into a binaural output audio signal representative of the two output audio signals.

[0027] In some implementations, the binaural processing device 32 takes into account the bone vibration sensor signal BV in the first processing module 323a and / or the second processing module 323b. Furthermore, the binaural processing device 32 may be further configured to receive an additional audio signal, synchronize and optionally convert the additional audio signal so that the additional audio signal is represented in at least one of the alternative representations of the first audio signal A1 and the second audio signal B1. Alternatively, a third processing module may be added in addition to the first processing module 323a and the second processing module 323b to process the additional audio signal and output the additional audio signal to an output stage 325 that generates a binaural output audio signal with side information representative of the processed additional audio signal.

[0028] 3 is a flow chart illustrating a method according to some implementations. At S1, an input binaural audio signal is received, represented by a first audio signal A1 and a second audio signal B1. The first and second audio signals may be alternative representations such as synchronized left and right audio signals or side and middle audio signals. The first audio signal A1 is passed to a first processing sequence S2a and the second audio signal B1 is passed to a second processing sequence S2b.

[0029] At S21, audio information is extracted from the first audio signal A1. The audio information includes a representation of at least a plurality of frequency bands, each frequency band representing a portion of the frequency content of the first audio signal A1. Furthermore, extracting the audio information from the first audio signal A1 may include extracting acoustic parameters describing the first audio signal A1.

[0030] Extracting the audio information in S21 may first include decomposing the first audio signal A1 into frequency spectral information. The frequency spectral information may be represented by a continuous or discrete frequency spectrum, such as a Fourier spectrum or a filter bank (such as QMF). The frequency spectral information may be represented by a number of bins, each bin containing values ​​such that the number of bins represents discrete samples of the frequency spectral information.

[0031] Second, the first audio signal A1 may be divided into multiple frequency bands, which may involve separate or overlapping groupings of bins representing frequency spectral information to form multiple frequency bands.

[0032] The frequency spectrum information may be used to extract band features such as Mel Frequency Cepstral Coefficients (MFCC) or Bark Frequency Cepstral Coefficients (BFCC) to be included in the audio information. Band harmonic features, a fundamental frequency (F0) of speech, a voice activity detection (VAD) probability and a signal-to-noise ratio (SNR) of the first audio signal A1 may be extracted by analyzing either the first audio signal A1 and / or the frequency spectrum information of the first audio signal A1. Thus, the audio information may include one or more of the band harmonic features, the fundamental frequency, the VAD probability and the SNR of each band of the first audio signal A1.

[0033] Based on at least the frequency bands representing the first audio signal A1 from the audio information extracted in S21, a band gain BGain for each frequency band is calculated in S22. The band gain BGain is calculated to reduce noise in the first audio signal A1. In some implementations, calculating the band gain BGain includes predicting the band gain BGain from the audio information using a trained neural network. The neural network may be a deep neural network and may include multiple neural network layers, each having multiple nodes. The neural network may be a fully connected neural network, a recurrent neural network, a convolutional neural network, or a combination thereof. A Wiener Filter may be combined with the neural network to provide a final prediction of the band gain. Given at least the frequency bands representing a portion of the first audio signal A1, the neural network is trained to predict an associated band gain BGain for reducing noise. In some implementations, the neural network (or a separate neural network) is further trained to also predict the VAD probability given at least a frequency band representing a portion of the frequency information of the first audio signal.

[0034] In S23, the band gain BGain of S22 is applied to the first audio signal A1 according to the dynamic scaling factor k from S24 to form a first audio output signal A2 with reduced quality degradation, where the dynamic scaling factor k is selected in S24 based on the band gain BGain calculated in S22 to reduce quality degradation. By selecting the dynamic scaling factor k to reduce quality degradation, the calculated band gain BGain for each frequency band may be adjusted according to the dynamic scaling factor k before being applied to the first audio signal A1 to provide the first output audio signal A2 with reduced quality degradation. The dynamic scaling factor k has a value between 0 and 1 and indicates how much of the calculated band gain is applied. In some implementations, the dynamic scaling factor k for each frequency band is based on at least one of the first audio signal A1, at least a portion of the audio information, and the calculated band gain BGain of each frequency band.

[0035] From the second audio signal B1 of the binaural audio signals, a second output audio signal B2 is provided by processing the second audio signal B1 in a second processing sequence S2b. For example, the second processing sequence S2b may comprise performing a separate processing of the second audio signal B1 (including, for example, a noise reduction processing) to form the second output audio signal B2. The separate processing of the second audio signal B1 may be equivalent to the processing of the first audio signal A1 in the first processing sequence S1a and may comprise steps corresponding to steps S21, S22, S23 and S24.

[0036] In some implementations, the processing of the first audio signal A1 and the second audio signal B1 in each processing sequence S2a, S2b is combined, for example, to apply a mono-channel noise reduction model. The mono-channel noise reduction model means that for each audio signal A1, B1, a respective set of noise reduction band gains BGain is calculated before these band gains BGain are reduced to a single common set. The common set of band gains can be determined as the maximum, minimum or average band gain for each band across all audio signals A1, B1. In other words, the calculated band gains BGain for each audio signal A1, B1 can first be represented by a matrix of band gains, denoted BGains(i,b), where i=1: the number of audio signals and b=1: the number of bands. Thus, each row of BGains(i,b) contains all band gains of the signal, and each column contains the band gain for a given band of each audio signal. In the mono-channel noise reduction matrix, a single row of band gains is extracted by merging each column into a single value, e.g., by finding the maximum value of each column. The same single row of band gains is then used for subsequent processing of all audio signals.

[0037] In S3, the first output audio signal A2 and the second output audio signal B2 are combined into a binaural output signal with reduced quality degradation.

[0038] 3 further illustrates a method according to some implementations in which the bone vibration sensor signal BV is used in processing the first audio signal A1. The recorded signal from the bone vibration sensor is more robust to environmental noise and the bone vibration sensor signal can be used to extract additional audio information and / or enhanced audio information and / or enhanced band gain.

[0039] In some implementations, the bone vibration sensor signal BV is used to extract a VAD probability for each time frame or each frequency band of each time frame, or to provide an enhanced VAD probability extracted from the first audio signal A1 and the bone vibration sensor signal BV. In S21 and S22, only the bone vibration sensor signal BV may be used, or the bone vibration sensor signal BV and the first audio signal A1 may be used to extract at least one of the frequency spectrum information, the band gain, the audio fundamental frequency, the SNR, and the VAD probability.

[0040] The bone vibration sensor signal BV may constitute a separate recording that complements the first audio signal A1 and the second audio signal A2 of the binaural audio signal, for example the bone vibration sensor signal BV may be treated as an additional audio signal and added to the binaural audio signal or provided as a separate output signal.

[0041] The enhanced first audio signal may be obtained from information in both the bone vibration sensor signal BV and the first audio signal A1. From the enhanced first audio signal, enhanced audio information (such as a more accurate representation of frequency content) may be extracted in S21, from which enhanced band gains may be calculated in S22. In some implementations, the bone vibration sensor signal BV is provided to a neural network in addition to the audio information for prediction of band gains and / or VAD probability in S22.

[0042] Similarly, a bone vibration sensor signal BV is provided and can be taken into account in the processing of a second audio signal B2 in a second processing sequence S2b.

[0043] FIG. 4a is a flow chart showing how the band gain BGain is applied to each frequency band according to the dynamic scaling factor k in S23a. The band gain BGain calculated in S22 is provided together with the first audio signal A1, and in S231, the calculated band gain is applied to the first audio signal A1 to form a noise-reduced first audio signal NA1. The noise-reduced first audio signal NA1 may exhibit undesirable audio artifacts introduced by applying the band gain in S231. In S24, a dynamic scaling factor k for reducing quality degradation is selected or calculated as described below. In S232, the noise-reduced first audio signal NA1 is mixed with the (original) first audio signal A1 in a mixing ratio corresponding to the dynamic scaling factor k selected in S24 to apply the band gain according to the dynamic scaling factor k. Thus, the first output audio signal A2 is obtained from the first audio signal A1, the noise-reduced first audio signal NA1 and the dynamic scaling factor k as follows:

number

[0044] 4b shows an alternative method for applying the band gains BGain according to the dynamic scaling factor k. In S23b, the calculated band gains for the first audio signal A1 from S22, the selected dynamic scaling factor k from S24 and the first audio signal A1 are available. The dynamic scaling factor k indicates how much of the band gains predicted in S22 should be applied, so that the first output audio signal A1 is a weighted sum of the first audio signal A1 and the first audio signal A1 with the band gains BGains applied. That is, the first output audio signal A2 may be calculated as follows:

number

[0045] 5 shows a time frame representation of an audio signal, for example a first audio signal. The audio signal is divided into a number of frames 101, 102, 103, 104 represented by columns, and each time frame includes a number of frequency bands represented by rows. For a particular frequency band 100, the calculated band gains (in linear units) are shown as 0.4, 0.6, 0.7 for the previous frames 101, 102, 103, and 0.8 for the current frame 104.

[0046] Methods are provided for determining a dynamic scaling factor k based on calculated band gains. For example, the dynamic scaling factor k is based on band gains calculated for a current (n+1) time frame 104 and previous (n, n-1, n-2) time frames 101, 102, 103 of an audio signal. In some implementations, the dynamic scaling factor k for a particular frequency band 100 of a current frame 104(n+1) is determined from a weighted sum of gains G(n+1), where the weighted sum G(n+1) is calculated as follows:

number

number

[0047] In some implementations, the dynamic scaling factor k for the current frame 104 is determined based on a predetermined threshold gain T Gain The gain can be affected only by the band gain of the previous frames 101, 102, 103, which exceeds a certain threshold gain T Gainmay be between 0.3 and 0.7, preferably about 0.5 (in linear units). This is because the calculated band gain BGain is Gain This can be achieved by updating the weighted sum of gains G(n+1) only in response to exceeding n. Thus, the weighted sum of gains G(n+1) for the current frame 104 is given by:

number

[0048] As an example, T Gain = 0.5, the calculated band gain for the frequency band 100 in the first frame 101 is 0.4 <T Gain Therefore, for a given threshold gain T Gain Then, if the initial value of the weighted sum of gains G is 1, the dynamic scaling factor k for the frequency band 100 of the first time frame 101 may be 0, since, according to the above, k=1−G. As a result, the band 100 of the first processed frame 101 is equal to the calculated band 100 of the first (noise reduced audio) audio signal. Each of the subsequent time frames 102, 103, 104 has a dynamic scaling factor k that is less than 1 but does not exceed a predetermined threshold gain T Gain Since the frame 102, 103, 104 is characterized by a calculated band gain exceeding , the processing of each subsequent frame 102, 103, 104 involves obtaining a lower value of G and, in response, obtaining a larger dynamic scaling factor k, which means that the applied band gains start to deviate from the calculated band gains and approach the original audio signal for band 100 of frames 102, 103, 104.

[0049] It will be appreciated that each frequency band represented by a row in FIG. 5 is associated with a respective weighted sum G of band gains that describes the band gains of the individual frequency bands in the current time frame 104 and previous time frames 101 , 102 , 103 .

[0050] Furthermore, the calculated band gain BGain(n+1) of the current frame 104 is compared to a predetermined threshold gain T Gain and the calculated band gain BGain(n+1) also exceeds 1 (in linear units), the calculated band BGain(n+1) may be set to a predetermined maximum numerical value before updating the weighted sum of the band gains G(n+1). The predetermined maximum value may be 1 (in linear units), meaning that the resulting dynamic mix ratio k is guaranteed to stay in the range of 0 to 1.

[0051] For offline processing, the dynamic scaling factor k for each frequency band of all time frames 101, 102, 103, 104 (represented by columns in FIG. 5) is calculated based on a predefined threshold gain T Gain to form a weighted sum G of the band gains or average band gain from which the dynamic scaling factor k is calculated.

[0052] In some implementations, the dynamic scaling factor may be further based on the VAD probability of each frequency band of each time frame 101, 102, 103, 104. Gain In addition to being the criterion for updating the weighted sum of band gains G, the VAD probability may define a further criterion. To this end, determining the dynamic scaling factor k is determined by determining whether the VAD probability for the frequency band 100 of the current frame 104 is greater than or equal to a predefined VAD probability threshold T VAD , and may further include determining whether the VAD probability exceeds a predetermined VAD probability threshold T VAD is between 0.4 (40%) and 0.6 (60%), preferably about 0.5 (50%). Thus, when the dynamic scaling factor k is determined for the current frame 104, only the band gains BGain of the previous frames 101, 102, 103 and the current frame 104, whose audio signals are likely to represent speech, are taken into account.

[0053] By taking into account the band gain and optionally the VAD probability for each band of the current time frame 104 and the previous time frames 101, 102, 103, the dynamic scaling factor k can be updated during online processing so that each frame (and each band) of the audio signal has an appropriate band gain BGain applied to reduce quality degradation given the available information. Thus, regardless of the audio signal being processed, the dynamic scaling factor can quickly approach a value suitable for reducing quality degradation for each additional processed time frame 101, 102, 103, 104.

[0054] In the case of offline processing, the band gains and optionally the audio information of the frequency bands 100 in all frames 101, 102, 103, 104 of the audio signal may be analyzed to determine a dynamic scaling factor k for each frequency band to direct the application of the band gains for all frames of the audio signal. The dynamic scaling factor for each frequency band in all time frames may be determined by a predefined probability threshold T VAD and a given threshold gain T Gain The band gain G may be determined by averaging all calculated band gains BGain above

[0055] In a further example illustrated by Figure 4, the band gain of a particular frequency band 100 for the current frame 104 is calculated as 0.8 (linear units), while the corresponding calculated band gains for the previous three frames 101, 102, 103 are 0.4, 0.6, 0.7 (linear units) respectively in increasing order of time. Gain In the situation where the threshold gain T is 0.5, the band gains of frames 102, 103, and 104 will affect the weighted sum of band gains G and the resulting dynamic scaling factor k for the current frame 104. When the previous frame 103 was processed, the band gain of frame 102 affected the weighted sum of band gains G, but not the threshold gain T GainThe band gains of frame 101 that are below the probability threshold T were ignored. The VAD probabilities calculated for each band of frames 101, 102, 103, 104 may result in different frame selections that influence the selection of the dynamic scaling factor k for the current frame 104. For example, if the previous frame 103 is below the probability threshold T VAD , then only frames 102 and 104 may influence the selection of the dynamic scaling factor k for the current frame 104, with frame 102 being ignored due to too low band gain and frame 103 being ignored due to too low VAD probability.

[0056] 6 shows a method for processing a binaural audio signal received at S1 according to some implementations. The audio signals of the binaural audio signal are left and right audio signals L and R, or are at least converted from alternative representations to left and right audio signals L and R, and provided to S12 (and optionally provided to S12 via S11, as described below).

[0057] The left audio signal L and the right audio signal R are combined in S12 to form intermediate audio signal M and side audio signal S, which are alternative representations of the left audio signal L and the right audio signal R. The intermediate audio signal M is estimated by the sum of the left audio signal L and the right audio signal R. For example, the intermediate audio signal M may be estimated as follows:

number

[0058] Similarly, the side audio signal S may be estimated by the difference between the left audio signal L and the right audio signal R. For example, the side audio signal S may be estimated as follows:

number

[0059] Each or one of the estimated intermediate audio signal M and side audio signal S constitutes the first audio signal and / or the second audio signal and may be processed according to the described implementations of the present disclosure. For example, both the side audio signal S and the intermediate audio signal M may be processed separately in the processing sequences S2a and S2b from FIG. 3. The audio processing of the side audio signal S may be different from the audio processing of the intermediate audio signal M. In one implementation, a more aggressive noise reduction is used in the processing of the side audio signal S in S2a compared to the processing of the intermediate audio signal M in S2b. Since a larger part of the recorded noise is assumed to be present in the side audio signal S, reducing the noise in the side audio signal S more improves the signal quality when the processed side audio signal PS and the processed intermediate audio signal PM are recombined to form the processed binaural audio signal.

[0060] To recreate processed versions of the original left and right audio signals L and R, i.e., the processed left audio signal PL and the processed right audio signal PR, the processed side audio signal PS and the processed intermediate audio signal PM may be recombined as a sum and a difference, in S28, to form the processed left audio signal PL and the processed right audio signal PR, respectively. For example, the processed left audio signal PL may be estimated as follows:

number

number

[0061] In some implementations, an additional audio signal from an additional recording device is received at S4. The additional audio signal is synchronized to the binaural audio signal and may be processed separately with respect to the first audio signal and the second audio signal, or may be processed in a joint manner (e.g., considered together with the first audio signal and the second audio signal to provide a mono-channel noise reduction model). For example, the processing of the additional audio signal may be equivalent to the processing of the first audio signal and the second audio signal in the first processing sequence S2a and the second processing sequence S2b. The processed additional audio signal PA may be provided as side information in the binaural output audio signal extracted at S28.

[0062] Alternatively, the additional audio signal is synchronized and mixed with the left audio signal L and the right audio signal R of the binaural audio signal in S11. The mixing of the additional audio signal A may be performed with the same predefined mixing ratio for each of the left audio signal L and the right audio signal R. For example, the mixing ratio of the additional audio signal A is 0.3 when mixed with the left audio signal L and 0.3 when mixed with the right audio signal R. If it is determined (e.g., by calculating the VAD probability) that the additional audio signal A is likely to contain speech, the predefined mixing ratio may be increased, for example, by applying a mixing gain, such that the resulting mixing ratio of the additional audio signal A is 0.7 when mixed with the left audio signal L and 0.7 when mixed with the right audio signal R. The additional audio signal A may undergo preprocessing, e.g., noise reduction or VAD probability extraction, before mixing with the left audio signal L and the right audio signal R. The resulting binaural output audio signal obtained in S3 may facilitate a more accurate reproduction of the audio from the second source of interest captured by the additional recording device.

[0063] In some implementations, frequency responses of the binaural recording device and the additional recording device are obtained. The frequency responses may be obtained by recording a measure representative of the energy captured by each device for each frequency band. Equalization information may be calculated and applied to at least one of the binaural audio signal (each of the first audio signal and the second audio signal) and the additional audio signal by comparing the frequency responses of equalization information associated with each device, which may be represented by an equalization curve, for example. For example, the equalization information may include a per-band gain extracted by comparing the per-band energy captured by the binaural recording device with the per-band energy captured by the additional recording device.

[0064] Because the binaural recording device and the additional recording device may be characterized by different frequency responses, application of equalization information, such as equalization curves, causes the tonality of the binaural recording device and the additional recording device to match, resulting in a more homogenous mix of the sound sources captured by each recording, improving the intelligibility of the sound sources captured by the recording devices.

[0065] In some implementations, the mixing gain of the additional audio signal from S4 and the binaural audio signal and / or the mixing gain of the binaural audio signal in S11 is adjusted based on the VAD probability. For example, a VAD probability for the additional audio signal may be extracted, and if the VAD probability indicates that the additional audio signal is likely to contain speech, a linear mixing gain greater than 1 may be applied to the additional audio signal when mixed with the binaural audio signals L, R in S11 to boost, for example, the speech of an interviewee closer to the additional recording device. Furthermore, if the VAD probability extracted for the intermediate audio signal indicates that the intermediate audio signal M is likely to contain speech, a linear gain greater than 1 may be applied to the intermediate audio signal M in S28 to boost, for example, the speech of a user wearing a binaural recording device.

[0066] In the processing of the binaural audio signal or in the processing of the binaural audio signal and the additional audio signal, the bone vibration sensor signal BV may be taken into account. Each processing sequence S2a, S2b may receive the bone vibration sensor signal BV according to the above.

[0067] Alternatively or additionally, the bone vibration sensor signal BV may be used to establish a VAD probability or an extended VAD probability for steering the mixing of the binaural audio signal and the additional signal A in S11. For example, if the bone vibration sensor signal BV indicates that the user of the binaural recording device is unlikely to be speaking, a linear mixing gain greater than one may be applied to boost the additional audio signal A in S11. In some implementations, the VAD estimated from the bone vibration sensor signal BV is used to determine whether speech is emanating from the user wearing the binaural recording device or from a second sound source of interest. For example, if the bone vibration sensor is worn by the user of the binaural recording device and the VAD probability extracted from the bone vibration audio signal BV indicates that speech audio is likely to be present, it is determined that the user wearing the binaural recording device is speaking. If the VAD probability extracted from the bone vibration audio signal BV indicates that speech audio is unlikely to be present, it may be determined that the user wearing the binaural recording device is not speaking. In response to determining that the user is not speaking, the additional audio signal and / or the side audio signal S is boosted to enhance any audio from the surroundings, such as if the interviewee is speaking, and in response to determining that the user is speaking, the intermediate audio signal is boosted to enhance the user's voice.

[0068] Instead of mixing the additional audio signal in the same mixing ratio to the left audio signal L and the right audio signal R, the middle audio signal M can be extracted solely or mainly from the additional audio signal and the side audio signal is extracted solely or mainly from the left audio signal L and the right audio signal R.

[0069] In some implementations, the bone vibration sensor signal emitted from the bone vibration sensor of the binaural recording device is used together with the extracted VAD probability of the additional audio signal to determine the source of the detected sound. For example, if the VAD of the additional audio signal is high but the bone vibration sensor signal shows little or no vibration, the source of the detected sound may be determined to be not the wearer of the binaural recording device. Alternatively, if the VAD probability of the additional audio signal is high and the bone vibration sensor signal shows bone vibration associated with speech, the source of the detected sound may be determined to be the wearer of the binaural recording device.

[0070] For this purpose, depending on the determined source of the detected sound, different noise reduction methods may be employed for the binaural audio signal and / or the additional audio signal. For example, if the sound originates from the wearer of the additional recording device, a first noise reduction technique may be employed that is specialized to suppress the noise added by the channel between the wearer of the binaural recording device and the additional recording device. If the sound originates from another sound source of interest, a different noise reduction technique may be more suitable to reduce the noise of the channel between the other sound source of interest and the additional recording device.

[0071] Additionally or alternatively, depending on the source of the detected sound, the relative gains of the binaural audio signal and the additional audio signal may be modulated accordingly, for example increasing the gain of the additional audio signal relative to the binaural audio if it is determined that the sound is emanating from another source of interest, or decreasing the gain of the additional audio signal relative to the binaural audio if it is determined that the sound is emanating from the wearer of the binaural audio signal.

[0072] FIG. 7 shows a flow chart illustrating a rendering method according to some implementations. In addition to playing binaural audio signals through headphones, using multiple speakers in a speaker system (e.g., a HiFi system or a surround sound system) or in a portable device is another common option. The portable device may be, for example, a tablet with four independent speakers, two upper speakers and two lower speakers, each speaker being fed through an individual power amplifier. For this purpose, a rendering method is provided for rendering binaural audio signals to at least four speakers.

[0073] In some implementations, the binaural audio signal includes a pair of audio signals, such as a processed left audio signal PL and a processed right audio signal PR. The rendering of the binaural audio signal is based on two cascaded procedures of applying the panning information obtained in S205 and the crosstalk cancellation information obtained in S210 to the binaural audio signal, which can be generally extended to render the binaural signal on an N-channel speaker system, where N is a natural number equal to or greater than 4, and at least two speakers of the speaker system form a left and right speaker pair. The N-channel rendering signal S can be determined as follows:

number

[0074] In S201, a binaural audio signal is obtained, and in S205, panning information (eg, a panning matrix M) is generated that indicates center panning for at least one left and right speaker pair of a speaker system.

[0075] In some implementations, in addition to the binaural audio signal having two audio signals obtained in S201 (the processed left audio signal PL and the processed right audio signal PR), a processed additional audio signal PA (originating from an additional audio signal A recorded by an additional recording device) is obtained in S202. The N-channel rendering signal S may be obtained in S220 as follows:

number

[0076] The parameters g1 and g2 indicate respective mixing coefficients of the binaural audio signal and the additional audio signal which set the signal power level of the binaural audio signal relative to the additional audio signal.

[0077] The crosstalk cancellation matrix X1 represents crosstalk cancellation information for at least one pair of speakers over which a binaural audio signal is rendered.

[0078] In accordance with the above, the binaural audio signal together with the processed additional audio signal may be rendered onto an N-channel speaker system in order to more clearly reproduce the voices of a user wearing the binaural recording device and a second sound source of interest (e.g. an interviewee located near the additional recording device).

[0079] Thus, the speaker system may render a binaural audio signal with an additional audio signal to enhance the audio from the second source of interest. By panning the additional audio signal to all speakers, the additional audio signal is clearly perceived, and the binaural signal is rendered on at least one speaker pair to provide an ambient audio effect.

[0080] In one embodiment, a system includes one or more computer processors and a non-transitory computer readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the operations of any one of the preceding method claims.

[0081] In one embodiment, a non-transitory computer readable medium stores instructions that, when executed by one or more computer processors, cause the one or more processors to perform the operations of any one of the preceding method claims.

[0082] According to exemplary embodiments of the present disclosure, the above-described processes may be implemented as a computer software program or on a computer-readable storage medium. For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine-readable medium, the computer program including program code for performing the method. In such embodiments, the computer program may be downloaded and mounted from a network via a communication unit and / or installed from a removable medium.

[0083] In general, various exemplary embodiments of the present disclosure may be implemented in hardware or dedicated circuits (e.g., control circuits), software, logic, or any combination thereof. For example, the above-mentioned units may be executed by a control circuit (e.g., a CPU in combination with other components), and thus the control circuit may perform the actions described in the present disclosure. While some aspects may be implemented in hardware, other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device (e.g., control circuit). Although various aspects of the exemplary embodiments of the present disclosure are illustrated and described as block diagrams, flow charts, or using any other graphical representation, it will be understood that the blocks, apparatus, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or any combination thereof, as non-limiting examples.

[0084] Additionally, the various blocks illustrated in the flowcharts may be considered as method steps and / or resulting operations of computer program code and / or as multiple combined logic circuit elements configured to perform the associated function(s). For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine-readable medium, the computer program including program code configured to perform the methods described above.

[0085] In the context of this disclosure, a machine-readable medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may be non-transitory and may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples of machine-readable storage media may include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0086] The computer program codes for carrying out the methods of the present disclosure may be written in any combination of one or more programming languages. These computer program codes may be provided to a processor of a general purpose computer, a special purpose computer, or other programmable data processing apparatus having control circuitry, so that the program code, when executed by the processor of the computer or other programmable data processing apparatus, causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on the computer, partially on the computer, as a stand-alone software package, partially on the computer and partially on a remote computer, or entirely on a remote computer or server, or distributed via one or more remote computers and / or servers.

[0087] Although this document contains many specific implementation details, these should not be construed as limitations on the scope of what may be claimed, but rather as descriptions of features that may be specific to a particular embodiment. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as acting in a particular combination and may even be initially claimed as such, one or more features from the claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination or variations of the subcombination. The logic flow depicted in the figures does not require the particular order or sequential order depicted to achieve the desired results. In addition, other steps may be provided or steps may be eliminated from the described flow, and other components may be added to or removed from the described system. Accordingly, other implementations are within the scope of the following claims.

Claims

1. 1. A method for processing a first audio signal and a second audio signal representative of an input binaural audio signal acquired by a binaural recording device, comprising: extracting audio information from the first audio signal, the audio information comprising a plurality of frequency bands representative of the first audio signal; calculating, for each frequency band of the first audio signal, a band gain for reducing noise in the first audio signal; calculating a voice activity detection (VAD) probability for each frequency band of the first audio signal; applying the band gains to respective frequency bands of the first audio signal according to respective dynamic scaling factors to provide a first output audio signal; the dynamic scaling factor has a value between 0 and 1, a value of 0 indicating that full-band gain is applied and a value of 1 indicating that no band gain is applied; the dynamic scaling factor is based on, for each frequency band, the band gains associated with corresponding frequency bands of a current time frame and a previous time frame of the first audio signal having a VAD probability exceeding a predetermined VAD probability threshold. Steps and performing a noise reduction process on the second audio signal to obtain a second output audio signal; determining a binaural output audio signal based on the first output audio signal and the second output audio signal; The method includes:

2. The method of claim 1 , wherein the noise reduction processing of the second audio signal comprises a separate processing step corresponding to the step of processing the first audio signal.

3. The step of providing a first output audio signal comprises: calculating a noise reduced audio signal by applying the band gains to each frequency band of the first audio signal; mixing each frequency band of the first audio signal with a corresponding frequency band of the noise reduced audio signal in a mixing ratio equal to the dynamic scaling factor to provide the first output audio signal; The method of claim 1 or 2, comprising:

4. The step of providing a first output audio signal comprises: calculating, for each band, a dynamic band gain as k+(1-k)Bgain, where k is the dynamic scaling factor and Bgain is the calculated band gain; applying the dynamic band gain to each band of a first audio signal to provide the first output audio signal; The method of claim 1 or 2, comprising:

5. The method of claim 1 , wherein the dynamic scaling factor for each frequency band is based on band gains of corresponding frequency bands in the current and previous time frames that exceed a predefined threshold gain.

6. The dynamic scaling factor is based on a weighted sum of band gains, the weighted sum including band gains from a previous time frame, and the method includes: determining that the band gain for a particular frequency band in the current time frame exceeds a predetermined threshold gain; if the band gain associated with the particular frequency band of the current frame exceeds the predetermined threshold gain, calculating a current weighted sum as a weighted sum of the band gains for the current time frame and a weighted sum including band gains from a previous time frame; If the band gain associated with the particular frequency band of the current frame is below a predetermined threshold gain, calculating the current weighted sum as a weighted sum including band gains from a previous time frame; The method of claim 1 , further comprising:

7. 7. The method of claim 1, wherein the dynamic scaling factor is determined as 1-G, where G is a weighted sum of band gains including at least a band gain from a frequency band of a previous time frame.

8. 8. The method of claim 1, wherein the step of determining the dynamic scaling factor for each frequency band is performed offline, and each dynamic scaling factor is based on the band gains associated with a corresponding frequency band of all time frames of the first audio signal.

9. determining a dynamic scaling factor for each frequency band of the first audio signal based on an average band gain from all frames, the band gain exceeding a predetermined threshold gain and the VAD probability exceeding a predetermined probability threshold; The method of claim 8 further comprising:

10. The first audio signal and the second audio signal are a left audio signal and a right audio signal, respectively, and the method comprises: - estimating the first audio signal as an intermediate audio signal, the intermediate audio signal being calculated from a sum of the left audio signal and the right audio signal; - estimating the second audio signal as a side audio signal, the side audio signal being calculated from a difference between the left audio signal and the right audio signal; estimating a left output audio signal by a sum of the intermediate audio signal and the side audio signal; and estimating a right output audio signal by a difference between the intermediate audio signal and the side audio signal. determining said binaural output audio signal by The method of claim 1 , further comprising:

11. The method may further include the step of processing an additional audio signal from an additional recording device, the first audio signal and the second audio signal being a left audio signal and a right audio signal, respectively, the method comprising: synchronizing the additional audio signal with the first audio signal and the second audio signal; mixing the additional audio signal with the first audio signal and the second audio signal; The method of claim 1 , further comprising:

12. The method further includes processing a bone vibration sensor signal acquired by the bone vibration sensor, the method comprising: synchronizing the bone vibration sensor signal with the first audio signal and the second audio signal; controlling a gain of the additional audio signal based on the bone vibration sensor signal; The method of claim 11 further comprising:

13. The method further includes processing a bone vibration sensor signal acquired by a bone vibration sensor of the binaural recording device, the method comprising: synchronizing the bone vibration sensor signal with the first audio signal and the second audio signal; extracting a VAD probability of said further audio signal; determining a source of the detected sound based on the VAD probability and the bone vibration sensor signal; if the source is a wearer of the binaural recording device having the bone vibration sensor, processing the additional audio signal with a first audio processing scheme adapted to suppress the noise in a channel between the wearer of the binaural recording device and the additional recording device; if the source is other than the wearer of the binaural recording device having the bone vibration sensor, processing the additional audio signal with a second audio processing scheme adapted to suppress the noise in the channel between another source and the additional recording device; The method of claim 12 further comprising:

14. The method of claim 13 , wherein the first audio processing scheme and the second audio processing scheme implement different signal gains for the additional audio signal.

15. The audio information is the SNR of the first audio signal; the fundamental frequency of the first audio signal; the VAD probability of the first audio signal; a bone vibration sensor signal acquired by the bone vibration sensor; a fundamental frequency extracted from the bone vibration sensor signal acquired by the bone vibration sensor; and VAD probabilities extracted from the bone vibration sensor signals acquired by the bone vibration sensor; The method of claim 1 , further comprising one or more of the following:

16. controlling a gain of the first audio signal based on the VAD probability extracted from the bone vibration sensor signal. The method of claim 15 further comprising:

17. 17. The method of claim 1, wherein calculating a band gain for each frequency band of the first audio signal comprises predicting the band gain from the audio information using a trained neural network.

18. A computer program comprising computer program code for performing the method according to any one of claims 1 to 17, when the computer program is run on a computer.

19. 1. An audio processing device, comprising: a receiver for receiving an input binaural audio signal captured by a binaural recording device, the input binaural audio signal including a first audio signal and a second audio signal; an extraction unit configured to receive the first audio signal from the receiver and extract audio information from the first audio signal, the audio information comprising a plurality of frequency bands representative of the first audio signal; a processing device configured to receive the audio information and to calculate, for each frequency band of the first audio signal, a band gain and a voice activity detection (VAD) probability for reducing noise in the first audio signal; an application unit configured to apply the band gains to respective frequency bands of the first audio signal according to a dynamic scaling factor to provide a first output audio signal, the dynamic scaling factor having a value between 0 and 1, a value of 0 indicating that a full band gain is applied and a value of 1 indicating that no band gain is applied, the dynamic scaling factor being based, for each frequency band, on the band gains associated with corresponding frequency bands of a current time frame and a previous time frame of the first audio signal for which a VAD probability exceeds a predefined VAD probability threshold; an additional processing module configured to perform noise reduction processing of the second audio signal to obtain a second output audio signal; an output stage configured to determine a binaural output audio signal based on the first output audio signal and the second output audio signal; 1. An audio processing device comprising:

Images