Steering binaural audio

The method addresses frequent switching issues in binauralization by using confidence values to dynamically adjust audio output, ensuring smooth transitions and accurate directional cues, thereby improving user experience in gaming and other audio applications.

JP7586573B2Active Publication Date: 2024-11-19DOLBY LABORATORIES LICENSING CORP
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2022509676
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-01-02
Filing Date
2020-08-19
Publication Date
2024-11-19
Estimated Expiration
2040-08-19

AI Technical Summary

Technical Problem

Existing binauralization techniques in gaming and other audio content often result in frequent switching between binauralized and non-binauralized audio, leading to a negative user experience due to jarring transitions and double binauralization, which can cause directional inaccuracies and instability in sound perception.

Method used

A method and system for steering binauralization that dynamically adjusts audio output based on confidence values derived from features like inter-channel level difference, phase difference, and coherence, using a binauralization detector and state determiner to smoothly transition between binauralized and non-binauralized states, avoiding abrupt changes and double binauralization.

Benefits of technology

The method reduces latency and frequency of switching, providing a smoother audio experience by minimizing abrupt transitions and ensuring accurate directional cues, thus enhancing user engagement and audio quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007586573000026
    Figure 0007586573000026
  • Figure 0007586573000027
    Figure 0007586573000027
  • Figure 0007586573000028
    Figure 0007586573000028
Patent Text Reader

Abstract

A method for steering binauralization of audio is provided, comprising: receiving (410) an audio input signal; calculating (430) a confidence value indicating a likelihood that a current audio frame of the audio input signal contains binauralized audio; determining (450) a state signal based on the confidence value; determining (460) a steering signal based on a first confidence value, the state signal, and an energy value of the audio frame; and generating (470) an audio output signal having steered binauralization by processing the audio input signal according to the steering signal.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to International Patent Application No. PCT / CN2019 / 101291, filed August 19, 2019, U.S. Provisional Patent Application No. 62 / 896,321, filed September 5, 2019, European Patent Application No. 19218142.8, filed December 19, 2019, and U.S. Provisional Patent Application No. 62 / 956,424, filed January 2, 2020, which are incorporated herein by reference.

[0002] Technical Field The present disclosure relates to the field of steering binauralization of audio. In particular, the present disclosure relates to a method, a non-transitory computer-readable medium, and a system for steering binauralization of audio. [Background technology]

[0003] Today, it is common to implement spatial audio techniques into audio content to provide an immersive user experience. One of the most common techniques is binauralization. Binauralization uses head-related transfer functions (HRTFs) to generate a virtual audio scene, which can be played by headphones or speakers. Binauralization is sometimes also referred to as virtualization. Audio generated by binauralization methods is sometimes referred to as binauralized audio or virtualized audio.

[0004] Electronic games have become increasingly popular with the rise of consumer entertainment devices such as smartphones, tablets, and personal computers. In gaming use cases, binauralization is widely used to provide additional information to the player. For example, binauralized gunshot sound clips in a first-person shooter game can provide directional information and indicate target location.

[0005] In gaming use cases, binauralized audio can be generated dynamically either on the content creation side or on the playback side. On the content creation side, various game engines offer binauralization methods that binauralize audio objects and mix them with [non-binauralized] background sounds. On the playback side, post-processing techniques may generate binauralized audio.

[0006] However, in all of the above cases, care should be taken with audio binauralization to avoid any adverse effects on the audio that may negatively impact the user's experience. Summary of the Invention [Means for solving the problem]

[0007] According to a first aspect, there is provided a method of steering binauralization of audio comprising the steps of: receiving an audio input signal comprising a plurality of audio frames; calculating a confidence value indicative of a likelihood that a current audio frame of the audio input signal comprises binauralized audio; determining a state signal based on the confidence value, the state signal indicating that the current audio frame is in a non-binauralized state or a binauralized state; and determining a steering signal, whereby when the state signal changes from indicative of the non-binauralized state to indicative of the binauralized state, the steering signal is varied to activate binauralization of audio by applying a head-related transfer function HRTF to the audio input signal resulting in a binauralized audio signal, and at least in part controlling the binauralization of the audio input signal. generating an audio output signal including a binauralized audio signal, setting a deactivation mode of binauralization to true when the state signal changes from indicating the binauralized state to indicating the non-binauralized state, and setting the deactivation mode of binauralization to false when the deactivation mode of binauralization is true and the confidence value of a current audio frame falls below a deactivation threshold and an energy value of a current audio frame is less than an energy value of a threshold number of audio frames of the audio input signal prior to a current audio frame, modifying the steering signal to deactivate or reduce binauralization of audio, and generating the audio output signal including at least a portion of the audio input signal.

[0008] By steering the binauralization according to such a method, frequent switching of the audio output signal between the binauralized audio input signal and the non-binauralized audio input signal is avoided, which is desirable because frequent switching can adversely affect the audio and lead to a negative user experience, for example, frequent switching can be jarring and cause discomfort to the user.

[0009] The steering also avoids double binauralization, such as post-binauralization processing of already binauralized audio, even if the audio input signal contains a mix of non-binauralized background and short-term binauralized sounds. It may be desirable to avoid double binauralization, as it can adversely affect the audio and lead to a negative user experience. For example, the direction of gunfire as perceived by a game player may be inaccurate if binauralization is applied twice.

[0010] The steering further has a properly designed switch point for checking that the energy value of the current audio frame is lower than the energy value of a threshold number of audio frames of the audio input signal preceding the current audio frame. This avoids negative user experience. For example, if a period of continuous gunfire is detected to be binauralized, the binauralizer should not be switched on immediately, as this would make the gunfire unstable. This instability problem can be noticeably perceived and detrimental to the overall audio quality.

[0011] According to an embodiment, when the steering signal is changed to activate audio binauralization, the step of generating the audio output signal comprises: mixing the binauralized audio signal and the audio input signal into a mixed audio signal over a first threshold time period and setting the mixed audio signal as the audio output signal, wherein the portion of the binauralized audio signal in the mixed audio signal is gradually increased during the first threshold period, and at the end of the first threshold period the audio output signal comprises only the binauralized audio signal.

[0012] A mixed audio signal is beneficial in that it smooths the transition from the audio input signal to the binauralised audio signal so that abrupt changes that may cause discomfort to the user are avoided.

[0013] The mixed audio signal optionally comprises the audio input signal and the binauralized audio signal as a linear combination with weights that sum to one, and may depend on the value of the steering signal. Weights that sum to one are beneficial in that the total energy content of the audio output signal is not affected by the mixing.

[0014] According to another embodiment, when the steering signal is changed to deactivate or reduce audio binauralization, generating the audio output signal comprises: mixing the binauralized audio signal and the audio input signal into a mixed audio signal over a second threshold time period and setting the mixed audio signal as an audio output signal, wherein the portion of the binauralized audio signal in the mixed audio signal is gradually reduced during the second threshold period, and at the end of the second threshold period the audio output signal comprises only the audio input signal.

[0015] The mixed audio signal is beneficial in that it provides a smooth transition from the binauralized audio signal to the audio input signal, avoiding abrupt changes that may cause discomfort to the user.

[0016] The mixed audio signal optionally comprises the audio input signal and the binauralized audio signal as a linear combination with weights that sum to one, and may depend on the value of the steering signal. Weights that sum to one are beneficial in that the total energy content of the audio output signal is not affected by the mixing.

[0017] According to yet another embodiment, calculating the confidence value comprises extracting features of a current audio frame of an audio input signal, the features of the audio input signal comprising at least one of inter-channel level differences (ICLD), inter-channel phase difference (ICPD), inter-channel coherence (ICC), mid / side Mel-Frequency Cepstral Coefficients (MFCC), and spectrogram peak / notch features; and calculating the confidence value based on the extracted features.

[0018] The extracted features are beneficial in that they allow a more precise calculation of the confidence value.

[0019] According to one or more embodiments, calculating the confidence value further includes: receiving features of a plurality of audio frames of the audio input signal preceding a current audio frame, the features corresponding to extracted features of the current audio frame; applying weights to features of the current and the plurality of previous audio frames of the audio input signal, the weights applied to the features of the current audio frame being greater than the weights applied to the features of the plurality of previous audio frames; and calculating the confidence value based on the weighted features.

[0020] The weights favor more recent frames, especially the current frame, which is beneficial in making the results more responsive to changes in the features computed from those frames.

[0021] According to yet another embodiment, the step of calculating the confidence value further comprises: weighting features of the current and a plurality of previous audio frames of the audio input signal according to an asymmetric window function.

[0022] Asymmetric window functions are beneficial in that they are a simple and reliable way to apply different weights to audio frames. An asymmetric window could be, for example, the first half of a Hamming window.

[0023] According to a second aspect, there is provided a non-transitory computer readable medium storing instructions that, when executed by one or more computer processors, cause the one or more processors to perform the method of the first aspect.

[0024] According to a third aspect, there is provided a system for steering binauralization of audio, the system comprising: an audio receiver for receiving an audio input signal comprising a plurality of audio frames; a binauralization detector for calculating a confidence value indicative of a likelihood that a current audio frame of the audio input signal comprises binauralized audio; a state determiner for determining a state signal based on the confidence value, the state signal indicating that the current audio frame is in a non-binauralized state or a binauralized state; and a switch determiner for determining a steering signal, wherein when the state determiner changes the state signal from indicative of the non-binauralized state to indicative of the binauralized state, the switch determiner changes the steering signal to activate binauralization of audio by applying a head-related transfer function HRTF to the audio input signal resulting in a binauralized audio signal, and at least partially modulates the steering signal to a binauralized audio signal. The binauralization control unit is configured to generate an audio output signal including the binauralized audio signal, and when the state determiner changes the state signal from one indicating the binauralization state to one indicating the non-binauralization state, the switching determiner is configured to set a deactivation mode of binauralization to true, and when the deactivation mode of binauralization is true and the confidence value of a current audio frame is below a deactivation threshold and an energy value of the current audio frame is less than an energy value of a threshold number of audio frames of the audio input signal prior to the current audio frame, the switching determiner is configured to set the deactivation mode of binauralization to false and modify the steering signal to deactivate or reduce binauralization of audio, and generate the audio output signal including at least in part the audio input signal.

[0025] The second and third aspects may generally have the same features and advantages as the first aspect. [Brief description of the drawings]

[0026] By way of example only, embodiments of the present disclosure will now be described with reference to the accompanying drawings, in which: [Figure 1] FIG. 1 is a block diagram of an exemplary system for binaural steering. [Diagram 2] FIG. 2 is a diagram of an exemplary four-state state machine. [Figure 3A] 4 shows an example of an exemplary confidence value. [Figure 3B] 4 illustrates an exemplary status signal. [Figure 3C] 4 illustrates an example steering signal. [Figure 4] 4 is a flowchart illustrating an example process for binauralization steering. [Diagram 5] 5 is a mobile device architecture for implementing the features and processes described with reference to FIGS. 1-4, according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, the present disclosure can be embodied in many different forms and should not be construed as being limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art. The terms used in the detailed description of the specific embodiments shown in the accompanying drawings are not intended to limit the present disclosure. In the drawings, like numbers refer to like elements.

[0028] Conventional binauralization techniques use a binauralization detection module and a mixing module to generate binauralized audio. This method is also effective for general entertainment content such as movies. However, it is not suitable for gaming use cases due to the differences between gaming content and other entertainment content (e.g., movies or music).

[0029] Typical gaming content contains many short-term binauralized sounds. This is due to the special binauralization method used for gaming content. Generally, binauralized movie content is obtained by applying a binauralizer to every audio frame (sometimes all at once). However, for gaming content, the binauralizer is usually applied to specific audio objects (gunshots, footsteps, etc.), which usually appear less frequently in time. That is, in contrast to other types of binauralized content, which have relatively long binauralized periods, gaming content has a mixture of non-binauralized background and short-term binauralized sounds.

[0030] The binauralization detection module is useful for the playback binauralization method to adaptively process binauralized or non-binauralized audio. This module typically uses Media Intelligence (MI) techniques and provides a confidence value that represents the probability that the signal is or is not binauralized. MI is a collection of techniques that derive information from multimedia signals using machine learning techniques and statistical signal processing.

[0031] The binauralization detection module can analyze audio data frame by frame in real time and output confidence scores associated with multiple types of audio (e.g., binauralization / dialogue / music / noise / VOIP) simultaneously. The confidence values ​​may be used to steer the binauralization method.

[0032] SUMMARY OF THE DISCLOSURE Accordingly, the present disclosure seeks to solve at least some of the problems discussed above and to eliminate or at least mitigate some of the shortcomings of prior art systems.

[0033] It is a further object of the present disclosure to provide a binauralization detection method that avoids relatively frequent switching.

[0034] Beginning with FIG. 1, a block diagram of an exemplary system 100 implementing a method for steering binauralization of audio is shown.

[0035] The input to the system 100 is an audio input signal 110. The audio input signal 110 includes a number of audio frames, which may include only foreground binaural audio, only background non-binaural audio, or a mix of both. The input signal 110 may be uncompressed or compressed. The compressed and / or encoded signal may be uncompressed and / or decoded (not shown in FIG. 1 ) prior to performing the method for steering the binauralization of the audio.

[0036] The audio input signal 110 is input to a binauralization detector 130. The binauralization detector 130 outputs a confidence value 135 indicating the likelihood that the input audio contains binauralized audio. The confidence value 135 is optionally normalized between zero and one, where zero indicates no likelihood that the audio input signal 110 contains binauralized audio and one indicates complete certainty that the audio input signal 110 contains binauralized audio.

[0037] The binauralization detector 130 may implement a step of calculating the confidence value 135 that involves extracting features of the audio input signal 110 that are indicative of the binauralized audio. The features are optionally extracted in the frequency domain, which implies that the features are transformed before extraction and inverse transformed after extraction. The transformation involves a domain transformation that decomposes the signal into several sub-bands (frequency bands).

[0038] According to one particular implementation, the binauralization detector 130 converts each frame of each channel into 64 complex quadrature mirror filter domain subbands, and further divides the lower three subbands into sub-subbands as follows: the first subband is divided into eight sub-subbands, the second and third subbands are each divided into four sub-subbands.

[0039] The features of the audio input signal indicative of the binauralized audio may include at least one of Inter-Channel Level Difference (ICLD), Inter-Channel Phase Difference (ICPD), Inter-Channel Coherence (ICC), Mid / Side Mel-Frequency Cepstral Coefficients (MFCC), and spectrogram peak / notch features.

[0040] Inter-Channel Level Difference (ICLD) is a measure proportional to the decibel difference in the subband acoustic energy of two different subbands. ICLD, ΔL(k) in the frequency domain, is given by:

number

[0041] Inter-channel phase difference (ICPD) is a measure of the phase difference between two subbands. ICPD, φ(k) in the frequency domain, is given by

number

[0042] Inter-channel coherence (ICC) is a measure of the coherence of two subbands. ICC, c(k) in the frequency domain, is given by

number

number

[0043] The mid and side Mel-frequency cepstral coefficients (MFCCs) may contain spectrogram modifications caused by HRTFs (Head Related Transfer Functions). The procedure for extracting these features includes: 1. The mid and side signals AM and AS are

number

[0044] HRTF filtering produces peaks and notches in the spectrogram in some frequency ranges (5-13 kHz). Such spectrogram peak and notch features can be useful to find the spectral modifications due to HRTFs. Spectral peak / notch features may be calculated for each channel by the following procedure:

[0045] 1. Find the local maxima and minima of the signal absolute value in the logarithmic domain and count the number of maxima and minima in a particular frequency range (e.g., 5-13 kHz). max , Num min Identify. The local maximum must satisfy the following conditions:

number

number

[0046] 2. To make it in the range [0,1], use Num max and Num min to the predefined value NUM norm_factor Normalize with .

number

[0047] Although these features are disclosed as being calculated for two subbands, any two subbands and / or sub-subbands may be selected, and optionally features are calculated for several pairs of subbands and / or sub-subbands, possibly combining them into a single mean or average metric. In an embodiment, these features are calculated for all subbands, and if a feature cannot be calculated exactly for at least one subband, such a subband is ignored.

[0048] In another embodiment, only certain ranges of subbands are used for certain features, and other ranges and non-computable subbands within those ranges are ignored. For example, with 77 Hybrid Complex Quadrature Mirror Filter (HCQMF) bands, only subbands 1-9 and 10-18 ranges can be used for ICC and ICPD calculations, and subbands 19-77 are ignored.

[0049] The extracted features of the audio input signal 110 indicative of the binauralized audio may be accumulated in a weighted histogram, which applies weights to the counts. In this embodiment, calculating the confidence value further comprises: accumulating features of the current audio frame and a predetermined number of previous audio frames of the audio input signal in a weighted histogram, which weights each subband used to calculate the features according to the total energy of that subband; and calculating the confidence value based on the mean or standard variance of the weighted histogram, for example by using them as input for a machine learning method as described below.

[0050] The weighted histogram includes features from a predetermined number of frames, such as 24, 48, 96, or any other suitable number. The frames are optionally sequential, starting with the current frame and counting backwards. The weighted histogram provides a good overview of the extracted features of the audio input signal from several different frames.

[0051] In one embodiment, two different weights are multiplied and applied to the histogram: one that weights the counts according to the ratio of each frequency band energy within a subband, and the other that weights the counts according to the ratio of each subband energy to the total subband energy of all subbands.

[0052] The weighted histogram is

number

number

number

number

[0053] The binauralization detector 130 may further implement a machine learning classifier that transforms the input as a function of at least one parameter estimated from training data and outputs a confidence value 135. The input may be the raw audio input signal or may be extracted features of the audio input signal, such as those exemplified above.

[0054] In one embodiment, the step of calculating the confidence value 135 includes: inputting the extracted features of the current speech frame of the audio input signal 110 and, if received or calculated, features of a number of speech frames of the audio input signal 110 preceding the current speech frame, into a machine learning classifier, which is trained to output the confidence value 135 based on the input.

[0055] The machine learning classifier may be trained to learn how to process inputs into confidence values ​​135, optionally supervised with the confidence values ​​135 as classes.

[0056] The machine learning classifier may be pre-trained or may be trained using a training set that is branched from the same data that is being input to the binauralization detector 130 .

[0057] A classifier is useful in providing greater precision in the calculation of the confidence value 135. A classifier may be implemented using, for example, AdaBoost, k-nearest neighbors, k-means clustering, support vector machines, regression, decision trees / forests / jungles, neural networks, and / or Naive Bayes algorithms.

[0058] The classifier may be, for example, an AdaBoost model. Real values ​​between [-∞,∞] may be obtained from the AdaBoost model, and therefore a sigmoid function may be used to map the obtained results to a range of confidence values ​​[0,1]. An example of such a sigmoid function is:

number

[0059] The binauralization detector 130 may apply weights to the audio input signal when calculating the confidence value 135, with the weight of the current audio frame being greater than the weight of the previous audio frame.

[0060] This may be implemented in that the step of calculating the confidence value 135 further includes: receiving features of a number of audio frames of the audio input signal 110 preceding the current audio frame, the features corresponding to the extracted features of the current audio frame; applying weights to the features of the current audio frame and the number of preceding audio frames of the audio input signal 110, the weights applied to the features of the current audio frame being greater than the weights applied to the features of the number of preceding audio frames; and calculating the confidence value based on the weighted features.

[0061] The received features of the multiple audio frames may for example be extracted from metadata or may be calculated in a similar manner as the features of the current audio frame.

[0062] A larger weight for the current audio frame than for the previous audio frame gives priority to newer frames, especially the current frame, which makes the binauralization detector 130 more sensitive to changes.

[0063] The weights may be implemented as constants or functions in the calculation of the confidence value 135. The weights may be implemented as an asymmetric window that includes the current audio frame and the most recent audio frames of the audio input signal 110.

[0064] Conventional binauralization detection methods calculate features based on statistics of a window that contains several consecutive frames. However, because it treats each frame equally, the delay is as much as half the window length, which is too large for gaming content. This is because if all frames of a window are weighted equally, at least half of the frames of the window will show binauralization upon arrival at the binauralization detector 130. Weighting the confidence values ​​as described herein reduces the latency of steering the binauralization of audio.

[0065] The weighting may be implemented in that the step of calculating the confidence value 135 further includes: applying weighting to features of the current audio frame and the plurality of previous audio frames of the audio input signal 110 according to an asymmetric window function.

[0066] The asymmetric window may be a Hamming window, a Hann window, or the first half of a triangular window.

[0067] The weights may be applied to a predetermined number of frames, e.g., 24, 48, 64, 96, or any other suitable number depending on the accuracy requirements of a particular embodiment. These frames are optionally sequential, starting with the current frame and counting backwards.

[0068] Thus, the binauralization detector 130 may be particularly suited to gaming content in that it has relatively low latency and is relatively adaptable to changes.

[0069] Some binaural audio events that may occur during a game have very short durations (e.g. gunshots). This poses a problem for feature-based classifiers with relatively long window lengths (audio clips). To handle this situation, shorter feature windows (shorter clips) can be used, but the general performance (e.g. latency) will be worse since the classifier will make decisions based on the shorter clips.

[0070] To address this issue, some embodiments of the present invention apply a dynamic frame feature weighting scheme. According to this approach, the frame feature weight is based on the frame energy ratio of the frame to the clip it belongs to. Thus, the weight is larger for high-energy frames.

[0071] Such dynamic weighting can be achieved by first determining whether the audio clip contains any impulse-like frames (i.e., frames that have significantly higher energy than other frames). In a two-channel implementation, this determination can be achieved as follows: 1. Calculate the average frame energy in the left and right channels for each frame i in a clip (N frames).

number

number

number

[0072] If a frame is found to be impulse-like, this may be indicated by setting flag P = 1. For clips without such frames, the weighting may be as described elsewhere. However, for clips that contain frames with flag P = 1, the dynamic weights may be determined according to the following: 1) Calculate the maximum and minimum values ​​of the average frame energy in the logarithmic domain, MinE(dB) and MaxE(dB). 2) Compute the frame feature weights for each frame i

number

number

[0073] Calculating the confidence value may optionally include inputting the calculated confidence value to a smoother 140. The smoothing stabilizes the confidence value such that abrupt changes are smoothed into less abrupt changes. This smoothing is beneficial in that abrupt changes have less impact on steering, which may otherwise cause sudden fluctuations that are unpleasant for the user.

[0074] This may be implemented in that the step of calculating the confidence value includes: receiving a confidence value of an audio frame immediately preceding the current audio frame; and adjusting the confidence value of the current audio frame using a one-pole filter, where the confidence value of the current audio frame and the confidence value of the audio frame immediately preceding the current audio frame are inputs to the one-pole filter, and the adjusted confidence value 145 is output from the one-pole filter.

[0075] One-pole filters are beneficial in that they are an efficient way of increasing speed and limiting the response time of smoothing. One technical effect of a one-pole filter is that only the confidence value of one previous frame is used, which reduces the number of frames that are checked, thereby reducing latency.

[0076] An example of a one-pole filter is: y(n)=ay(n-1)+(1-a)x(n), where y(n) is the smoothed confidence value 145 of the current frame, y(n-1) is the smoothed confidence value 145 of the previous frame, x(n) is the (unsmoothed) confidence value 135 of the current frame, and a is a constant. a is the sample rate of the audio signal F. s , and / or may depend on the period of smoothing τ, e.g.

number

[0077] The RC time constant is the charging or discharging rate of the processing circuit that performs the step of calculating the confidence value, ie the resistor-capacitor circuit which corresponds to the smoother 140 in this embodiment.

[0078] The one-pole filter may have a smoothing time shorter than the smoothing threshold, where the smoothing threshold is determined based on an RC time constant, to ensure that the period of smoothing is not too long and the response time of the smoothing 440 is relatively short.

[0079] The confidence value (smoothed 145 or unsmoothed 135) is input to a state determiner 150. The state determiner 150 performs steps to determine a state signal 155 of the method for steering the binauralization of the audio. The state signal 155 indicates whether the current audio frame is in a non-binauralized state or a binauralized state.

[0080] The state determiner 150 determines whether the state of the audio, i.e., binauralized or non-binauralized, has changed recently, which may include within a predetermined number of previous frames, such as the previous 1, 2, 3, 5, 10, or any suitable number of previous frames.

[0081] The state determiner 150 is optionally a four-state state machine as illustrated in FIG. 2 and further described below, where two states of the four-state state machine correspond to state signals 155 indicating that the current audio frame is in a non-binauralized state, and the remaining two states of the four-state state machine correspond to state signals 155 indicating that the current audio frame is in a binauralized state.

[0082] The four-state state machine includes an un-binauralized holding state (UBH) 210, a binauralized holding state (BH) 230, a binauralized release counting state (BRC) 240, and a binauralized attack counting state (BAC) 220, where UBH 210 and BAC 220 correspond to state signals 155 indicating that the current audio frame is in a un-binauralized state, and BH 230 and BRC 240 correspond to state signals indicating that the current audio frame is in a binauralized state.

[0083] BAC 220 implements a short-term accumulator with a slack counting rule to determine when the state signal transitions from BAC 220 to BH 230, i.e., from indicating that the current audio frame is in a non-binauralized state to indicating that the current audio frame is in a binauralized state. The accumulator continues, for example, counting any confidence values ​​that exceed a confidence threshold, until a predefined number is reached. The accumulator is short-term in that it is implemented over a relatively short, pre-set period, for example, 5 seconds. That is, the short-term accumulator optionally uses a slack counting rule so that it is relatively easy to get out of the BAC 220 state.

[0084] The BRC 240 implements a long-term monitor that uses a tight counting rule to determine when the state signal transitions from BRC 240 to UBH 210, i.e., from indicating that the current audio frame is in a binauralized state to indicating that the current audio frame is in a non-binauralized state. The monitor checks, for example, whether a predefined number of previous confidence values ​​are below a confidence threshold. The monitor is long-term in that it is implemented over a relatively long preset period, such as 20 seconds, i.e., the long-term monitor optionally uses a tight counting rule such that it is relatively difficult to get out of the BRC 240 state.

[0085] This difference between the short-term accumulator and the long-term monitor reduces the missing error in short-term binaural sound detection that is common in the prior art.

[0086] The four-state state machine is beneficial in that it further stabilizes the output 155 of the state determination stage, which avoids frequent switching between binauralized and non-binauralized states, which could otherwise be annoying to the user.

[0087] The four-state state machine is discussed further below with respect to FIG.

[0088] The input audio 110 may further be input to an energy analyzer 120, which analyzes the audio energy of the audio input signal and provides information for the switch decider 160. In another embodiment, the audio energy of the audio input signal 110 is received, for example, via metadata of the audio input signal 110.

[0089] The energy of a signal corresponds to the total magnitude of the signal. For an audio signal, it corresponds roughly to the volume of the signal. For example, the energy for an audio frame may be calculated as the sum of the squared absolute values ​​of the amplitudes normalized by the frame length.

[0090] In one embodiment, the energy value of the current frame is calculated by the energy analyzer 120. The root mean square of the energy value over a given number of frames N is:

number

[0091] In one embodiment, the short-term energy of the frame is calculated by the energy analyzer 120. The smoothed energy signal, also written as p(t), is

number

[0092] The energy value and / or the root mean square of the smoothed energy signal or any other suitable energy information is then output as an energy-oriented signal 125 to a switch decider 160 .

[0093] The switching decider 160 implements the stage of determining a steering signal 165 of the method for steering the binauralization of the audio. The switching decider 160 has as inputs the confidence values ​​135, 145 which are the result of the binauralization detector 130, the state signal 155 which is the result of the state decider 150, and the energy steering signal 125 which is the result of the energy analyzer 120 or is received through other means such as from metadata.

[0094] The step of determining the steering signal 165 includes, when the state signal 155 is changed from one indicating a non-binauralized state to one indicating a binauralized state, modifying the steering signal 165 to activate audio binauralization by applying a head-related transfer function HRTF to the audio input signal 110 resulting in a binauralized audio signal, and generating an audio output signal 175 including at least in part the binauralized audio signal.

[0095] The step of determining the steering signal 165 further includes the steps of: setting a binauralization deactivation mode to true when the state signal 155 is changed from one indicating a binauralization state to one indicating a non-binauralization state; and if the binauralization deactivation mode is true and the confidence value 135, 145 of the current audio frame is below a deactivation threshold and the energy value of the current audio frame is lower than the energy value of a threshold number of audio frames of the audio input signal 110 preceding the current audio frame: setting the binauralization deactivation mode to false and modifying the steering signal 165 to deactivate or reduce audio binauralization and generate an audio output signal 175 that at least in part includes said audio input signal 110.

[0096] The deactivation mode is beneficial in that the modification of the steering signal 165 that deactivates or reduces the binauralization of the audio does not occur immediately unless the confidence value 135, 145 of the current audio frame falls below the deactivation threshold and unless the energy value of the current audio frame falls below the energy value of a threshold number of audio frames of the audio input signal 110 preceding the current audio frame.

[0097] This avoids frequent switching between binauralized and non-binauralized states because the requirement of a deactivation threshold delays the switching, and also, for example, a sudden and temporary drop in the confidence value is ignored if it does not reach the threshold. The deactivation threshold may be pre-set or user-defined.

[0098] It also compares the energy values ​​of the current audio with the energy values ​​of previous audio frames, thus avoiding noticeable changes during high energy periods, which prevents an inconsistent listening experience.

[0099] Further details of determining the steering signal 165 are disclosed with respect to FIG. 3C.

[0100] In a final step of the method for steering audio binauralization implemented by the system 100 of Figure 1, generating an audio output 175 with steered binauralization is performed by audio processing 170. Generating the audio output is steered by the steering signal and may be performed by the switching decider 160 or a separate audio processor 170. The audio processing includes applying HRTFs to the audio input signal 110 when required (as described above), resulting in a binauralized audio signal.

[0101] FIG. 2 shows a four-state state machine according to an embodiment implementing the stage of determining the state signals of the method for steering the binauralization of audio.

[0102] The state signal is a binary function ranging from zero to 1. A value of zero in the state signal indicates that the audio input signal contains a non-binauralized state, while a value of one in the state signal indicates that the audio input signal contains a binauralized state. The state signal aims to prevent frequent switching between binauralized and non-binauralized states from the confidence values ​​by rounding the confidence values ​​to a stretch of one or zero.

[0103] The state of the state machine transitions from UBH 210 to BAC 220 when the confidence value exceeds a confidence threshold, from BAC 220 to BH 230 when a threshold number of frames have confidence values ​​above the confidence threshold while the state BAC 220 is reached, from BH 230 to BRC 240 when the confidence value falls below the confidence threshold, and from BRC 240 to UBH 210 when a predetermined number of consecutive frames have confidence values ​​below the confidence threshold.

[0104] In the following, a use case of the state machine of Fig. 2 is described. This is intended only as a non-limiting example to further explain the functionality of the different states. In this example, the initial state of the state machine is UBH 210, but for example, BH 230 could also be selected as the initial state.

[0105] If the final state is UBH 210 (which is also true if the UBH 210 state is the initial state), then the confidence value is the confidence threshold T high If it is less than this, the state is maintained (arrow a in FIG. 2) and the state signal is set to or maintained as zero. high is 0.6, but any other reasonable ratio is possible.

[0106] If the confidence value is greater than or equal to the confidence threshold, the state changes to the BAC 220 state (arrow b in FIG. 2) and the state signal is maintained as zero.

[0107] The short-term accumulator is active while the last state is the BAC 220 state. The accumulator is updated by a confidence threshold T medianLow Save the counts with higher confidence values. If the counts exceed a given count threshold N acc If the count is less than T, the accumulator continues to count while the state is maintained as the BAC 220 state (arrow c in FIG. 2) and the state signal is maintained at zero. medianLow is 0.45, but any other suitable ratio is possible. acc is the number of frames corresponding to 5 seconds, but any other number of frames is possible.

[0108] Once the accumulator count reaches a predetermined count threshold N acc If so, the state is changed to the BH 230 state (arrow d in Figure 2), while the state signal is set to 1 and the accumulator is reset.

[0109] If the last state is a BH 230 state, the confidence value is the confidence threshold T low If so, the state is held (arrow e in FIG. 2) and the state signal is held at 1. low is 0.25, but any other reasonable ratio is possible.

[0110] The confidence value is the confidence threshold T low If it is lower, the state changes to the BRC 240 state while the state signal is maintained as 1 (arrow f in FIG. 2).

[0111] The long-term monitor is active while the last state is the BRC 240 state. The monitor is activated when the most recent consecutive trust values ​​are all below the trust threshold T medianHigh Check if it is less than T medianHigh If the above confidence value occurs, the state transitions back to BH 230 (arrow g in FIG. 2) while the state signal is maintained as 1.

[0112] In one embodiment, 20 seconds of recent consecutive confidence values ​​are checked, but any other number of seconds is possible. medianHigh is 0.55, but any other reasonable ratio is possible.

[0113] The confidence value is the confidence threshold T medianHigh While it is less, the state is maintained as BRC 240 (arrow h in FIG. 2) and the monitor continues to wait until the full span of successive trust values ​​has been checked.

[0114] Once the monitors have consecutive trust values, all of them fall within the trust threshold T medianHigh If it is observed that the state is less than , then the state changes to UBH 210 (arrow i in Figure 2). During this time, the state is set to zero and the monitor is reset.

[0115] 3A illustrates example confidence values ​​over time 330. The illustrated confidence values ​​330 are smoothed confidence values, but may also be unsmoothed.

[0116] 3B illustrates an example state signal 350 resulting from the example confidence value 330 of FIG. 3A. Note that the state changes from zero to one only after a few seconds of high confidence value 330. This time occurs when the BAC 220 accumulator reaches a predefined count threshold N acc , which corresponds to changing state to BH 230. Furthermore, state signal 350 does not change from one to zero immediately as confidence value 330 drops because the continuity requirement of the long term monitor, which corresponds to the BRC 240 state, is not achieved, and therefore the state machine does not transition to the UBH 210 state until later.

[0117] Thus, the purpose of the state signal 350, which is to prevent frequent switching between binauralized and non-binauralized states, is achieved.

[0118] FIG. 3C illustrates an example steering signal 360 resulting from the example confidence value 330 of FIG. 3A and the example state signal 350 of FIG. 3B.

[0119] The steering signal 360 steers the audio processing. When the steering signal 360 is zero, no processing is performed. As a result, the audio input signal is output as is as the audio output signal. When the steering signal 360 is one, binauralization is performed by applying a head-related transfer function HRTF to the audio input signal, resulting in a binauralized audio signal as the audio output signal. If the steering signal 360 is between zero and one, mixing occurs and the mixed audio signal is output as the audio output signal. A steering signal 360 between zero and one may for example be caused by an intermediate ramp between the zero and one states, which will be described later.

[0120] In order to avoid double binauralization, which may have a negative effect on the audio and result in a negative user experience, it is an object of the invention to ensure that processing is only performed on audio frames of the audio input signal that do not already contain binauralized sounds.

[0121] Thus, many prior art steering signals correspond to the inverse of a confidence value or state signal. However, the inventors have come to realize that the switching points of the steering signal 360 from one to zero, and optionally vice versa, should be properly designed to avoid instability problems.

[0122] The switching points of the steering signal 360 should not be selected during periods of dense and loud binauralized sound, because quickly switching the HRTFs on and off during those periods would lead to an inconsistent listening experience.

[0123] Thus, determining a steering signal 360, such as the example steering signal 360 of FIG. 3C, goes beyond observing changes in the state signal 350 and includes comparing the confidence value 330 of the current audio frame to a deactivation threshold and comparing the energy value of the current audio frame to the energy value of the previous audio frame.

[0124] 3C avoids switching from one to zero in the middle of a block of high confidence values ​​330 despite changes in state signal 350. This is because the energy value of a current audio frame of the audio input signal is compared to the energy values ​​of a predetermined set of previous frames, and if the audio energy value remains relatively unchanged over the predetermined set of previous frames, steering signal 360 is maintained at its current value. The predetermined set may be, for example, the most recent 24, 48 or 96 audio frames.

[0125] In one particular example, the steering signal 360 is held at its current value if the energy value of the current audio frame is greater than or equal to the energy value of 90% of the last 48 audio frames. Other ratios are possible, such as 80%, 70%, etc., as are other numbers of audio frames, such as 10, 35, 42, etc.

[0126] Once the high confidence value 330 block is completed, the example steering signal of FIG. 3C switches from one to zero. The switching is implemented by applying a ramp function. During the ramp, the steering signal 360 has a value between zero and one, thus mixing the binauralized audio signal and the audio input signal into a mixed audio signal and setting the mixed audio signal as the audio output signal. This further avoids abrupt changes to binauralization that would lead to an inconsistent listening experience.

[0127] A ramp change may be implemented in that, when the steering signal 360 is changed to activate audio binauralization, the step of generating an audio output signal includes: mixing the binauralized audio signal and the audio input signal into a mixed audio signal over a first threshold time period and setting the mixed audio signal as the audio output signal, wherein a portion of the binauralized audio signal in the mixed audio signal gradually increases during the first threshold period, such that at the end of the first threshold period, the audio output signal includes only the binauralized audio signal.

[0128] Alternatively, when the steering signal 360 is changed to activate binauralization of the audio, generating the audio output signal includes setting the audio output signal as a binauralized audio signal, e.g., without a ramp-type change.

[0129] The ramp change may further be implemented in that when the steering signal 360 is changed to deactivate or reduce audio binauralization, the step of generating an audio output signal includes: mixing the binauralized audio signal and the audio input signal into a mixed audio signal over a second threshold time period and setting the mixed audio signal as the audio output signal, wherein a portion of the binauralized audio signal in the mixed audio signal gradually decreases during the second threshold period, such that at the end of the second threshold period, the audio output signal includes only the audio input signal.

[0130] Alternatively, when the steering signal 360 is changed to deactivate or reduce binauralization of the audio, generating the audio output signal includes setting the audio output signal as the audio input signal.

[0131] The example steering signal 360 of FIG. 3C is implemented according to the following three rules:

[0132] When state signal 350 switches from one to zero, steering signal 360 becomes:

number

[0133] When the state signal 350 switches from zero to one, the steering signal 360 is asserted if two conditions are met: the confidence value 330 c(t) of the current frame is greater than or equal to the deactivation threshold T switch and the smoothed energy signal p(t) is less than a threshold fraction R of the energy values ​​of a predetermined number M of previous frames, where

number

number

[0134] If the state signal 350 does not change, the steering signal 360 holds its last value.

[0135] In order to achieve a smooth transition between binauralization active and inactive, a mixing procedure is performed when w(t)∈(0,1), i.e., the audio output signal is a mixed audio signal. Given an audio input signal x(t), a generated binauralized audio signal B(t), and a steering signal 360 w(t), the output audio signal y(t) may be expressed as y(t)=w(t)B(t)+(1-w(t))x(t).

[0136] Thus, the binauralized audio signal and the audio input signal are mixed as a linear combination with weights that sum to one, and which depend on the value of the steering signal 360. If the steering signal 360 is closer to one than to zero, the weight of the binauralized audio signal is higher than the weight of the audio input signal, and vice versa.

[0137] 4 shows a flow chart illustrating a method 400 for steering binauralization of audio. The method 400 includes several steps, some of which are optional and some of which may be performed in any order. The method 400 shown in FIG. 4 is an exemplary embodiment and is not intended to be limiting.

[0138] The first step of the method 400 is receiving 410 an audio input signal. The audio input signal may be in any format and may or may not be compressed and / or encrypted. Preferably, receiving the audio input signal 410 includes decrypting any encrypted audio and / or decompressing any compressed audio before any other steps of the method 400 are performed. The audio input signal may contain several channels of audio, some of which may contain only binauralized sounds, some of which may contain only non-binauralized sounds, and some of which may contain a mix of binauralized and non-binauralized sounds. The audio input signal does not have to contain both binauralized and non-binauralized sounds, but in any other case the steering result is very simple.

[0139] Another step of the method 400 is analysing 420 energy values ​​of the audio input signal. This step 420 may involve calculating an energy value x(t) for the current frame t, for example by calculating the root mean square of the energy value and / or the smoothed energy signal p(t) or any other suitable energy information. This information is then output as a result of the step of analysing 420 energy values ​​of the audio input signal.

[0140] The stage 420 of analysing the energy values ​​of the audio input signal is optional and, if included, this stage 420 is performed before the stage 460 of determining the steering signal. As an alternative to this stage 420, the energy information may be extracted from another source, such as metadata.

[0141] Another step in the method 400 is calculating 430 a confidence value indicative of the likelihood that a current audio frame of the audio input signal contains binauralized audio.

[0142] This step 430 may be performed independently of the other steps of the method 400 .

[0143] The stage 430 may further comprise the steps of: extracting features of a current audio frame of the audio input signal, the features of the audio input signal including at least one of an Inter-Channel Level Difference (ICLD), an Inter-Channel Phase Difference (ICPD) and an Inter-Channel Coherence (ICC); calculating a confidence value based on the extracted features; receiving features of a number of audio frames of the audio input signal prior to the current audio frame, the features corresponding to the extracted features of the current audio frame; applying weights to features of the current audio frame and the number of previous audio frames of the audio input signal, the weights applied to the features of the current audio frame being greater than the weights applied to the features of the number of previous audio frames; and calculating a confidence value based on the weighted features.

[0144] This stage 430 may further comprise applying weights to the features of the current and multiple previous audio frames of the audio input signal according to an asymmetric window function, which may be the first half of a Hamming window.

[0145] This stage 430 further comprises accumulating features of the current audio frame and a predetermined number of previous audio frames of the audio input signal into a weighted histogram, which weights each sub-band used to compute the features according to the total energy in that sub-band; and calculating a confidence value based on the mean or standard variance of the weighted histogram.

[0146] This stage 430 may further include inputting the weighted features of the current and a plurality of previous audio frames of the audio input signal into a machine learning classifier, the machine learning classifier being trained to output a confidence value based on the input.

[0147] Another step of the method 400 is smoothing 440 the confidence values ​​into smoothed confidence values. This step 440 is optional, and if included, this step 440 is performed as part of the step of calculating 430 the confidence values, although steps 430, 440 may be implemented by different circuits / units. As a result, this step 440 may be performed independently from steps of the method 400 other than the step of calculating 430 the confidence values.

[0148] This step 440 may include receiving a confidence value of an audio frame immediately preceding the current audio frame; and adjusting the confidence value of the current audio frame using a one-pole filter, where the confidence value of the current audio frame and the confidence value of the audio frame immediately preceding the current audio frame are inputs to the one-pole filter, and the adjusted confidence value is an output from the one-pole filter.

[0149] This step 440 may further include the one-pole filter having a smoothing time less than a smoothing threshold, the smoothing threshold being determined based on an RC time constant.

[0150] Another step in the method 400 is determining 450 a status signal based on the confidence value.

[0151] The state signal is a binary function ranging from zero to 1. A value of zero in the state signal indicates that the audio input signal comprises a non-binauralized state, whereas a value of one in the state signal indicates that the audio input signal comprises a binauralized state.

[0152] Further steps of the method 400 are determining 460 a steering signal based on the energy values ​​of the audio frames analysed in a step 420 of analysing energy values ​​of the audio input signal or received through other means; depending on whether a step 440 of smoothing confidence values ​​has been performed and / or the confidence values ​​calculated in the step 430 of smoothing confidence values, and the state signal determined in a step 450 of determining a state signal.

[0153] The steering signal steers the stage for generating 470 an audio output signal. If the steering signal is zero, binauralization of the audio is deactivated or reduced. If the steering signal is one, binauralization of the audio is activated. If the steering signal is between zero and one, mixing occurs.

[0154] The step of generating 470 an audio output signal may or may not be performed in conjunction with the step of determining 460 a steering signal, and may or may not be performed by the same circuitry.

[0155] FIG. 5 illustrates a mobile device architecture for implementing the features and processes described with reference to FIGS. 1-4, according to an embodiment. The architecture 500 may be implemented in any electronic device, including, but not limited to, a desktop computer, a consumer audio / visual (AV) device, a radio broadcast device, or a mobile device (e.g., a smartphone, a tablet computer, a laptop computer, or a wearable device). In the illustrated exemplary embodiment, the architecture 500 is for a smartphone and includes a processor 501, a peripherals interface 502, an audio subsystem 503, a loudspeaker 504, a microphone 505, sensors 506 (e.g., accelerometer, gyro, barometer, magnetometer, camera), a position processor 507 (e.g., GNSS receiver), a wireless communication subsystem 508 (e.g., Wi-Fi, Bluetooth, cellular), and an I / O subsystem 509, including a touch controller 510 and other input controllers 511, a touch surface 512 and other input / control devices 513. Other architectures having more or fewer components may also be used to implement the disclosed embodiments.

[0156] Memory interface 514 is coupled to processor 501, peripherals interface 502, and memory 515 (e.g., Flash, RAM, ROM). Memory 515 stores computer program instructions and data, including, but not limited to, operating system instructions 516, communications instructions 517, GUI instructions 518, sensor processing instructions 519, telephony instructions 520, electronic messaging instructions 521, web browsing instructions 522, audio processing instructions 523, GNSS / navigation instructions 524, and applications / data 525. Audio processing instructions 523 include instructions for performing the audio processing described with reference to Figures 1-4.

[0157] Aspects of the system described herein may be implemented in a suitable computer-based sound processing network environment for processing digital or digitized audio files. Portions of the adaptive audio system may include one or more networks including any desired number of individual machines, including one or more routers (not shown) that serve to buffer and route data transmitted between computers. Such networks may be built on a variety of different network protocols and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0158] One or more of the components, blocks, processes, or other functional components may be implemented through a computer program that controls the execution of a processor-based computing device of the system. It should also be noted that the various functions disclosed herein may be described in terms of their behavior, register transfers, logical components, and / or other characteristics using any number of combinations of hardware, firmware, and / or data and / or instructions embodied in various machine-readable or computer-readable media. Computer-readable media in which such formatted data and / or instructions may be embodied include, but are not limited to, various forms of physical (non-transitory) non-volatile storage media, such as optical, magnetic, or semiconductor storage media.

[0159] Further embodiments of the present disclosure will be apparent to those skilled in the art after reviewing the above description. Although the present specification and drawings disclose embodiments and examples, the present disclosure is not limited to these specific examples. Numerous modifications and variations can be made without departing from the scope of the present disclosure defined by the appended claims. Any reference signs appearing in the claims shall not be understood as limiting the scope thereof.

[0160] Furthermore, variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the present disclosure, from a study of the drawings, the disclosure, and the appended claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0161] The above-described systems and methods may be implemented as software, firmware, hardware, or a combination thereof. For example, aspects of the present application may be embodied, at least in part, in an apparatus, a system including multiple apparatuses, a method, a computer program product, etc. In a hardware implementation, the division of tasks among the functional units described above does not necessarily correspond to a division into physical units. Conversely, one physical component may have multiple functions, and one task may be performed by multiple cooperating physical components. Some or all of the components may be implemented as software executed by a digital signal processor or microprocessor, or as hardware, or as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those skilled in the art, the term "computer storage media" includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by a computer. Additionally, communication media typically embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and is well known to those skilled in the art to include any information delivery media.

Claims

1. 1. A method for steering binauralization of audio, the method comprising: receiving an audio input signal comprising a plurality of audio frames; calculating a confidence value indicative of the likelihood that a current audio frame of the audio input signal contains binauralized audio; determining a state signal based on the confidence value, the state signal indicating that the current audio frame is in a non-binauralized state or a binauralized state; determining a steering signal when the state signal changes from indicative of the non-binauralization state to indicative of the binauralization state; activating audio binauralization by applying a head-related transfer function (HRTF) to the audio input signal, thereby varying the steering signal to result in a binauralized audio signal; generating an audio output signal comprising at least in part the binauralized audio signal; setting a binauralization deactivation mode to true when the state signal changes from indicating the binauralization state to indicating the non-binauralization state; If the deactivation mode of binauralization is true, and the confidence value of a current audio frame is below a deactivation threshold, and the energy value of the current audio frame is less than the energy value of a threshold number of audio frames of the audio input signal prior to the current audio frame: setting the deactivation mode of binauralization to false; modifying the steering signal to deactivate or reduce binauralization of audio; generating the audio output signal comprising at least in part the audio input signal; method.

2. When the steering signal is changed to activate binauralization of audio, generating the audio output signal comprises: mixing the binauralized audio signal and the audio input signal into a mixed audio signal over a first threshold time period and setting the mixed audio signal as an audio output signal, wherein a portion of the binauralized audio signal in the mixed audio signal is gradually increased during the first threshold time period, and at the end of the first threshold time period, the audio output signal comprises only the binauralized audio signal. The method of claim 1.

3. When the steering signal is changed to deactivate or reduce binauralization of audio, generating the audio output signal comprises: mixing the binauralized audio signal and the audio input signal into a mixed audio signal over a second threshold time period and setting the mixed audio signal as an audio output signal, wherein a portion of the binauralized audio signal in the mixed audio signal is gradually reduced during the second threshold time period, and at the end of the second threshold time period, the audio output signal includes only the audio input signal.

3. The method according to claim 1 or 2.

4. 2. The method of claim 1, wherein when the steering signal is changed to activate binauralization of audio, generating the audio output signal comprises setting the audio output signal as the binauralized audio signal.

5. 5. The method of claim 1, wherein when the steering signal is changed to deactivate or reduce binauralization of audio, generating the audio output signal comprises setting the audio output signal as the audio input signal.

6. Calculating a confidence value comprises extracting features of a current audio frame of the audio input signal; and calculating the confidence value based on the extracted features, wherein the features are: at least one of inter-channel level difference (ICLD), inter-channel phase difference (ICPD), inter-channel coherence (ICC), mid / side mel-frequency cepstral coefficients (MFCC), and spectrogram peak / notch features; 6. The method according to any one of claims 1 to 5.

7. The step of calculating the confidence value further comprises: receiving features of a plurality of audio frames of the audio input signal prior to a current audio frame, the features corresponding to extracted features of the current audio frame; applying weights to features of the current and the plurality of previous audio frames of the audio input signal, the weights applied to the features of the current audio frame being greater than the weights applied to the features of the plurality of previous audio frames; and calculating the confidence value based on the weighted features. The method of claim 6.

8. The step of calculating the confidence value further comprises: applying weights to features of the current and the plurality of previous audio frames of the audio input signal according to an asymmetric window function; The method of claim 7.

9. The method of claim 8 , wherein the asymmetric window is the first half of a Hamming window.

10. determining whether a current audio frame and the plurality of previous audio frames contain an impulse-like signal; if so, applying dynamic weights to features of the current audio frame and the plurality of previous audio frames; the dynamic weights are based on a ratio of frame energies; The method of claim 7.

11. The determining step comprises: [Equation 25] The frame energy ratio R for each frame is calculated according to i Calculating E i is the average energy of all channels in frame i, and R i is greater than the first threshold, and E i determining that frame i is impulse-like if The method of claim 10.

12. The step of calculating the confidence value further comprises: accumulating features of the current and a predetermined number of previous audio frames of the audio input signal into a weighted histogram, the weighted histogram weighting each subband used to calculate the features according to the total energy within that subband; and calculating the confidence value based on the mean or standard variance of the weighted histogram.

12. The method according to any one of claims 7 to 11.

13. The steps for calculating the confidence value are: inputting the extracted features of the current audio frame of the audio input signal and, if received, features of a plurality of audio frames of the audio input signal prior to the current audio frame into a machine learning classifier; the machine learning classifier is trained to output a confidence value based on the input; 13. The method according to any one of claims 6 to 12.

14. The steps for calculating the confidence value are: receiving a confidence value of an audio frame immediately preceding a current audio frame; and adjusting the confidence value of the current audio frame using a one-pole filter; a confidence value of a current audio frame and a confidence value of an audio frame immediately preceding the current audio frame are inputs to the one-pole filter, and an adjusted confidence value is output from the one-pole filter; 14. The method according to any one of claims 1 to 13.

15. The step of determining the status signal comprises: applying a four-state state machine, two states of which correspond to the state signal indicating that the current audio frame is not binauralized, and two remaining states of which correspond to the state signal indicating that the current audio frame is binauralized.

15. The method according to any one of claims 1 to 14.

16. 16. The method of claim 15, wherein the one-pole filter has a smoothing time less than a smoothing threshold, the smoothing threshold being determined based on an RC time constant.

17. the four-state state machine includes a non-binauralized hold state (UBH), a binauralized hold state (BH), a binauralized release count state (BRC), and a binauralized attack count state (BAC); wherein UBH and BAC correspond to the state signal indicating that the current audio frame is in a non-binauralized state, and BH and BRC correspond to the state signal indicating that the current audio frame is in a binauralized state; The state transitions from UBH to BAC when the confidence value exceeds a confidence threshold, the state transitions from BAC to BH when a threshold number of frames have confidence values ​​higher than the confidence threshold while the state is BAC being reached, the state transitions from BH to BRC when the confidence value falls below the confidence threshold, and the state transitions from BRC to UBH when a predetermined number of consecutive frames have confidence values ​​lower than the confidence threshold.

17. The method of claim 15 or 16.

18. A non-transitory computer readable medium storing instructions that, when executed by one or more computer processors, cause the one or more processors to perform the method of any one of claims 1 to 17.

19. 1. A system for steering binauralization of audio, the system comprising: an audio receiver for receiving an audio input signal including a plurality of audio frames; a binauralization detector that calculates a confidence value indicative of the likelihood that a current audio frame of the audio input signal contains binauralized audio; a state determiner that determines a state signal based on the confidence value, the state signal indicating that the current audio frame is in a non-binauralized state or a binauralized state; a switching determiner for determining a steering signal, wherein when the state determiner changes the state signal from one indicative of the non-binauralization state to one indicative of the binauralization state, the switching determiner: activating audio binauralization by applying a head-related transfer function (HRTF) to the audio input signal, thereby varying the steering signal to result in a binauralized audio signal; configured to generate an audio output signal comprising at least in part the binauralized audio signal; when the state determiner changes the state signal from one indicating the binauralization state to one indicating the non-binauralization state, the switching determiner sets a binauralization deactivation mode to true; If the deactivation mode of binauralization is true, and the confidence value of a current audio frame is below a deactivation threshold, and an energy value of the current audio frame is less than an energy value of a threshold number of audio frames of the audio input signal before the current audio frame, the switching decider: setting the deactivation mode of binauralization to false; modifying the steering signal to deactivate or reduce binauralization of audio; configured to generate the audio output signal comprising at least in part the audio input signal; system.

20. one or more computer processor circuits; a non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 17. system.

Citation Information

Patent Citations

  • Audio processing method and apparatus

    JP2010511910A

  • Processing of a monophonic signal in a 3D audio decoder, delivering a binaural content

    WO2019122580A1