Perceptual enhancement for binaural audio recordings

Abstract

A method of audio processing includes capturing a binaural audio signal, calculating a noise reduction gain using a machine learning model, and generating a modified binaural audio signal. The method may further include performing various corrections to the audio taking into account video captured by different cameras, such as a front camera and a rear camera. The method may further include performing a smooth transition of the binaural audio when switching between the front camera and the rear camera. The method may reduce noise in the binaural audio and improve a user's perception of the combined video and binaural audio.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Patent Application No. 63 / 139,329, filed January 20, 2020, and U.S. Provisional Patent Application No. 63 / 287,730, filed December 9, 2021, and PCT Application No. PCT / CN2020 / 138221, filed December 22, 2020, all of which are incorporated by reference in their entirety herein.

[0002] The present disclosure relates to audio processing, and in particular to noise suppression. [Background technology]

[0003] Unless otherwise noted herein, the approaches described in this section are not prior art to the claims of this application and are not admitted to be prior art by inclusion in this section.

[0004] Devices for audiovisual capture are becoming increasingly popular among consumers. Such devices include portable cameras, such as Sony Action Cam™ and GoPro™ cameras, and mobile phones with integrated camera functionality. Typically, the devices capture audio simultaneously as they capture video, for example using mono or stereo microphones. Audiovisual content sharing systems, such as the YouTube™ and Twitch.tv™ services, are also becoming increasingly popular. Users then broadcast the captured audiovisual content simultaneously as they are captured, or upload the captured audiovisual content to the content sharing system. Because this content is generated by users, it is called user-generated content (UGC) as opposed to professionally generated content (PGC), which is usually generated by professionals. UGC differs from PGC in that UGC is often created using consumer equipment that is less expensive and has fewer features than professional equipment. Another difference between UGC and PGC is that UGC is often captured in an uncontrolled environment, such as outdoors, whereas PGC is often captured in a controlled environment, such as a recording studio.

[0005] Binaural audio involves audio recorded using two microphones located at the user's ears. The captured binaural audio creates an immersive listening experience when played back through headphones. Compared to stereo audio, binaural audio also includes the shadows of the user's head and ears, resulting in interaural time differences and interaural level differences when binaural audio is captured. Summary of the Invention

[0006] Existing audiovisual capture systems have several problems. One problem is that many existing capture devices only include mono or stereo microphones, making binaural audio particularly difficult to capture. Another problem is that UGC audio often has stationary and non-stationary noise that is not present in PGC audio, because PGC is often captured in a controlled environment. Another problem is that separate audio and video capture devices can result in audio and video streams that are inconsistent with human perception using the eyes and ears.

[0007] Embodiments relate to capturing video simultaneously with binaural audio and performing perceptual enhancements, such as noise reduction, on the captured binaural audio, where the resulting binaural audio, when subsequently consumed in combination with the captured video, is perceived differently than stereo or mono audio.

[0008] According to one embodiment, a computer-implemented method of audio processing includes capturing, by an audio capture device, an audio signal having at least two channels, including a left channel and a right channel. The method further includes calculating, by a machine learning system, a plurality of noise reduction gains for each channel of the at least two channels. The method further includes calculating a plurality of shared noise reduction gains based on the plurality of noise reduction gains for each channel. The method further includes generating a modified audio signal by applying the plurality of shared noise reduction gains to each channel of the at least two channels.

[0009] As a result, noise can be reduced in the captured binaural audio.

[0010] The machine learning system may use a mono model, a binaural model, or both mono and binaural models.

[0011] The method may further include capturing a video signal by a video capture device simultaneously with capturing the audio signal. The method may further include switching between a front camera and a rear camera, where the switching includes smoothing a left / right correction of the audio signal using a first smoothing parameter and smoothing a front / back correction of the audio signal using a second smoothing parameter. Capturing the video signal simultaneously with capturing the audio signal may include performing a correction on the audio signal, where the correction includes at least one of a left / right correction, a front / back correction, and a stereo image width control correction. The stereo image width control correction may include generating a center channel and a side channel from a left channel and a right channel of the audio signal, attenuating the side channel by a width adjustment factor, and generating a modified audio signal from the center channel and the attenuated side channel.

[0012] According to another embodiment, an apparatus includes a processor. The processor is configured to control the apparatus to implement one or more of the methods described herein. The apparatus may further include similar details to one or more of the methods described herein.

[0013] According to another embodiment, a non-transitory computer readable medium stores a computer program that, when executed by a processor, controls an apparatus to perform processes including one or more of the methods described herein.

[0014] The following detailed description and the accompanying drawings provide a further understanding of the nature and advantages of various implementations. [Brief description of the drawings]

[0015] [Figure 1]FIG. 1 is a stylized overhead view of an audio-visual capture system 100.

[0016] [Diagram 2] FIG. 2 is a block diagram of an audio processing system 200.

[0017] [Diagram 3] FIG. 3 is a block diagram of an audio processing system 300.

[0018] [Figure 4] FIG. 4 is a block diagram of an audio processing system 400.

[0019] [Diagram 5] FIG. 5 is a block diagram of an audio processing system 500.

[0020] [Figure 6] FIG. 2 is a stylized overhead view showing binaural audio capture in selfie mode using video capture system 100 (see FIG. 1).

[0021] [Figure 7] 1 is a graph showing an example of the magnitude response of a high shelf filter implemented using a bi-quad filter.

[0022] [Figure 8] FIG. 13 is a stylized overhead view showing various audio capture angles in selfie mode.

[0023] [Figure 9] 1 is a graph of the attenuation coefficient α for different focal lengths f.

[0024] [Figure 10]FIG. 2 is a stylized overhead view showing binaural audio capture in normal mode using video capture system 100 (see FIG. 1).

[0025] [Figure 11] 11 is a device architecture 1100 for implementing the features and processes described herein, according to one embodiment.

[0026] [Figure 12] 12 is a flow chart of a method 1200 of audio processing.

[0027] [Figure 13] 13 is a flow chart of a method 1300 of audio processing. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] Herein, techniques related to audio processing are described. In the following description, for purposes of explanation, numerous examples and specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that the present disclosure, as defined by the claims, may include some or all of the features of these examples, either alone or in combination with other features described below, and may further include modifications and equivalents of the features and concepts described herein.

[0029] The following description details various methods, processes, and procedures. While certain steps may be described in a certain order, such order is primarily for convenience and clarity. Certain steps may be repeated multiple times, may occur before or after other steps, or may occur in parallel with other steps, even if the other steps are described in a different order. A second step need only follow a first step if the first step must be completed before the second step can begin. Such situations will be specifically pointed out if not clear from the context.

[0030] In this document, the terms "and", "or" and "and / or" are used. Such terms are to be interpreted as having an inclusive meaning. For example, "A and B" may mean at least: "both A and B", "at least both A and B". As another example, "A or B" may mean at least: "at least A", "at least B", "both A and B", "at least both A and B". As another example, "A and / or B" may mean at least: "A and B", "A or B". If an exclusive-or is intended, this will be specifically stated, e.g., "either A or B", "at most one of A and B", etc.

[0031] This document describes various processing functions in the context of structures such as blocks, elements, components, circuits, etc. Generally, these structures may be implemented by a processor controlled by one or more computer programs.

[0032] 1 is a stylized overhead view of an audiovisual capture system 100. A user typically uses the audiovisual capture system 100 to capture audio and video in an uncontrolled environment, such as to capture UGC. The audiovisual capture system 100 includes a video capture device 102, a left earphone 104, and a right earphone 106.

[0033] The video capture device 102 generally includes a camera that captures video data. The video capture device 102 may include two cameras, referred to as a front camera and a rear camera. The front camera, also referred to as a selfie camera, is generally located on one side of the video capture device 102, e.g., the side that includes the display screen or touch screen. The rear camera is generally located on the side opposite the side of the front camera. The video capture device 102 may be a mobile phone, and as such may have several additional components and functions, such as a processor, volatile and non-volatile memory and storage, a radio, a microphone, a speaker, etc. For example, the video capture device 102 may be a mobile phone, such as an Apple iPhone® mobile phone, a Samsung Galaxy™ mobile phone, etc. The video capture device 102 may generally be held by a user, attached to a user's selfie stick or tripod, attached to a user's shoulder mount, attached to an aerial drone, etc.

[0034] The left earphone 104 is placed on the user's left ear, includes a microphone, and typically captures a left binaural signal. The left earphone 104 provides the left binaural signal to the video capture device 102 for capturing audio data simultaneously with the video data. The left earphone 104 may wirelessly connect to the video capture device 102 via an IEEE 802.15.1 standard protocol, such as the Bluetooth protocol. Alternatively, the left earphone 104 may be connected to another device, not shown, that receives both the captured audio data and the captured video data from the video capture device 102.

[0035] The right earphone 106 is placed at the user's right ear, includes a microphone, and typically captures a right binaural signal. The right earphone 104 provides a right binaural signal to the video capture device 102 in a manner similar to that described above with respect to the left earphone 104. The right earphone 106 may be similar to the left earphone 104 in other respects.

[0036] An example use case for the audiovisual capture system 100 is for a user to walk down the street and capture binaural audio using the earphones 104 and 106 while simultaneously capturing video using the video capture device 102. The audiovisual capture system 100 then broadcasts the captured content or stores the captured content for later editing or uploading. Another example use case is recording podcasts, interviews, news reports, and speeches during conferences and events. In such situations, binaural recording can provide a desirable sense of spaciousness; however, the presence of environmental noise and the distance of other sources of interest from the person wearing the earphones 104 and 106 often results in a less than optimal playback experience due to the overwhelming presence of noise. Properly reducing excess noise while maintaining the spatial cues of the recording is difficult, but very valuable in practice.

[0037] The following sections detail additional audio processing techniques implemented by audio-visual capture system 100, for example, to perform noise reduction on the captured binaural audio.

[0038] 1. Noise reduction for captured binaural audio

[0039] 2 is a block diagram of an audio processing system 200. The audio processing system 200 may be implemented as a component of the audiovisual capture system 100 (see FIG. 1), for example as one or more computer programs executed by a processor of the video capture device 102. The audio processing system 200 includes a conversion system 202, a noise reduction system 204, a mixing system 206, and an inverse conversion system 208.

[0040] The conversion system 202 receives a left input signal 220 and a right input signal 222 and performs a signal conversion to generate a converted left signal 224 and a converted right signal 226. The left input signal 220 generally corresponds to the signal captured by the left earphone 104, and the right input signal 222 generally corresponds to the signal captured by the right earphone 106. That is, the input signals 220 and 222 correspond to binaural signals, with the left input signal 220 corresponding to the left binaural signal and the right input signal 222 corresponding to the right binaural signal. The converted left input signal 224 corresponds to the converted left input signal 220, and the converted right input signal 226 corresponds to the converted right input signal 222.

[0041] The signal transformation generally converts the input signal from a first signal domain to a second signal domain. The first signal domain may be the time domain. The second signal domain may be the frequency domain. The signal transformation may be one or more of the following: a Fourier transform, such as a fast Fourier transform (FFT), a short-time Fourier transform (STFT), a discrete-time Fourier transform (DTFT), a discrete Fourier transform (DFT), a discrete sine transform (DST), a discrete cosine transform (DCT), a quadrature mirror filter (QMF) transform; a complex quadrature mirror filter (CQMF) transform; a hybrid complex quadrature mirror filter (HCQMF) transform; and the like. The transformation system 202 may perform framing of the input signal before performing the transformation, and the transformation is performed on a frame-by-frame basis. The frame size may be between 5 ms and 15 ms, e.g., 10 ms. The transformation system 202 may output the transformed signals 224 and 226 grouped into bands in the transformation domain. The number of bands may be between 15 and 25, e.g., 20 bands.

[0042] The noise reduction system 204 receives the transformed left signal 224 and the transformed right signal 226 and performs gain calculations to generate a left gain 230 and a right gain 232. The noise reduction system 204 generally implements one or more machine learning systems to calculate the noise reduction gains 230 and 232. In particular, the left gain 230 corresponds to the noise reduction gain applied to the transformed left signal 224 and the right gain 232 corresponds to the noise reduction gain applied to the transformed right signal 226. The noise reduction gains may be shared noise reduction gains applied to both left / right signals, e.g., a single set of gains applied to both signals. Details of the machine learning systems and noise reduction gains are provided below, with particular reference to Figures 3-5.

[0043] The mixing system 206 receives and mixes the transformed left signal 224, the transformed right signal 226, the left gain 230, and the right gain 232 to generate a mixed left signal 234 and a mixed right signal 236. The mixing system 206 generally mixes the transformed left signal 224 with the left gain 230 to generate a mixed left signal 234, and mixes the transformed right signal 226 with the right gain 232 to generate a mixed right signal 236. Further details of mixing are provided below, with particular reference to Figures 3-5.

[0044] The inverse transformation system 208 receives the mixed left signal 234 and the mixed right signal 236 and performs an inverse signal transformation to generate a modified left signal 240 and a modified right signal 242. The inverse signal transformation generally corresponds to the inverse of the signal transformation performed by the transformation system 202, converting the signal from the second signal domain back to the first signal domain. For example, the inverse transformation system 208 may transform the mixed signals 234 and 236 from the QMF domain to the time domain. As a result, the modified left signal 240 corresponds to a noise reduced version of the left input signal 220, and the modified right signal 242 corresponds to a noise reduced version of the right input signal 222.

[0045] The audiovisual capture system 100 may then output the modified left signal 240 and the modified right signal 242 along with the captured video signal as part of the generation of the UGC. Additional details of the audio processing system 200 are provided below, with particular reference to Figures 3-5.

[0046] 3 is a block diagram of an audio processing system 300. The audio processing system 300 is a more specific embodiment of the audio processing system 200 (see FIG. 2). The audio processing system 300 may be implemented as a component of the audiovisual capture system 100 (see FIG. 1), for example, as one or more computer programs executed by a processor of the video capture device 102. The audio processing system 300 includes transformation systems 302a and 302b, noise reduction systems 304a and 304b, a gain calculation system 306, mixing systems 308a and 308b, and inverse transformation systems 310a and 310b.

[0047] The conversion systems 302a and 302b receive a left input signal 320 and a right input signal 322, perform signal conversion, and generate a converted left signal 324 and a converted right signal 326. In particular, the conversion system 302a generates the converted left signal 324 based on the left input signal 320, and the conversion system 302b generates the converted right signal 326 based on the right input signal 322. The input signals 320 and 322 correspond to the binaural signals captured by the earphones 104 and 106 (see FIG. 1). The signal conversion performed by the conversion systems 302a and 302b generally corresponds to the signal conversion as described above with respect to the conversion system 202 (see FIG. 2).

[0048] The noise reduction systems 304a and 304b receive the transformed left signal 324 and the transformed right signal 326, perform gain calculations, and generate a left gain 330 and a right gain 332. In particular, the noise reduction system 304a generates the left gain 330 based on the transformed left signal 324, and the noise reduction system 304b generates the right gain 332 based on the transformed right signal 326. The noise reduction system 304a receives the transformed left signal 324, performs feature extraction on the transformed left signal 324 to extract a set of features, processes the set of features by inputting the set of features into a trained model, and generates the left gain 330 as a result of processing the set of features. Inputting the features into a trained model and processing them may also be referred to as "classification." The noise reduction system 304b receives the transformed right signal 326, performs feature extraction on the transformed right signal 326 to extract a set of features, processes the set of features by inputting the set of features into a trained model, and generates a right gain 332 as a result of processing the set of features.

[0049] The features may include one or more of temporal features, spectral features, time-frequency features, etc. The temporal features may include one or more of autocorrelation coefficients (ACC), linear predictive coding coefficients (LPCC), zero-crossing rate (ZCR), etc. The spectral features may include one or more of spectral centroid, spectral roll-off, spectral energy distribution, spectral flatness, spectral entropy, Mel-frequency cepstral coefficients (MFCC), etc. The time-frequency features may include one or more of spectral flux, chroma, etc. The features may also include statistics of the other features mentioned above. These statistics may include mean, standard deviation, and higher order statistics, e.g., skewness, kurtosis, etc. For example, the features may include the mean and standard deviation of the spectral energy distribution.

[0050] The trained model may be implemented as part of a machine learning system. The machine learning system may include one or more neural networks, such as a recurrent neural network (RNN), a convolutional neural network (CNN), etc. The trained model receives the extracted features as input, processes the extracted features, and outputs a gain as a result of processing the extracted features. It is noted that both noise reduction systems 304a and 304b use the same trained model, e.g., each noise reduction system implements a copy of the trained model. The trained model is trained offline using mono training data, as described further below.

[0051] The gain calculation system 306 receives the left gain 330 and the right gain 332 and combines the gains 330 and 332 according to a mathematical function to generate the shared gain 334. The mathematical function may be one or more of a maximum, an average, a range function, a comparison function, etc. As an example, let the left gain 330, the right gain 332, and the shared gain 334 each be a gain vector of gains, e.g., a vector of 20 bands. For the maximum, the gain of band 1 of the shared gain 334 is the maximum of the gain of band 1 of the left gain 330 and the gain of band 1 of the right gain 332; similarly for the other 19 bands. For the average, the gain of band 1 of the shared gain 334 is the average of the gain of band 1 of the left gain 330 and the gain of band 1 of the right gain 332; similarly for the other 19 bands.

[0052] The range function applies a different function to each band based on the range of gains in each band of gains 330 and 332. For example, if the gain in band 1 of each of gains 330 and 332 is less than X1, calculate the maximum. If the gain is between X1 and X2, calculate the average. If the gain is greater than X2, calculate the maximum.

[0053] The difference function applies a different function to each band based on a comparison of the difference between the gains in each band, 330 and 332. For example, if the gain difference in band 1, 330 and 332, is less than X1, calculate the average; if the gain difference is greater than or equal to X1, calculate the maximum.

[0054] Instead of applying the left gain 330 to the transformed left signal 324 and the right gain 332 to the transformed right signal 326, the audio processing system 300 uses a shared gain 334 to reduce artifacts that may be present in quick-attack sounds. Binaurally captured quick-attack sounds may cross frame boundaries of the input signals 320 and 322 (as part of the operation of the conversion systems 302a and 302b) due to interaural time differences between the left / right microphones. In such cases, the gain of the quick-attack sound is processed at frame X in one channel and frame X+1 in the other channel, which may result in artifacts. Calculating the shared gain, e.g., the maximum value of the gain of a particular band in each channel, results in a reduced perception of the artifacts.

[0055] Noise reduction systems 304a and 304b, and gain calculation system 306 may otherwise be similar to noise reduction system 204 (see FIG. 2).

[0056] Mixing systems 308a and 308b receive the transformed left signal 324, the transformed right signal 326, and a shared gain 334 and apply the shared gain 334 to the signals 324 and 326 to generate a mixed left signal 336 and a mixed right signal 338. In particular, mixing system 308a applies the shared gain 334 to the transformed left signal 324 to generate the mixed left signal 336, and mixing system 308b applies the shared gain 334 to the transformed right signal 326 to generate the mixed right signal 338. For example, the transformed left signal 324 may have 20 bands, the shared gain 334 may be a gain vector with 20 bands, and the magnitude value of a given band of the mixed left signal 336 results from multiplying the magnitude value of a given band of the transformed left signal 324 by the gain value of the given band of the shared gain 334. The mixing systems 308a and 308b may otherwise be similar to the mixing system 206 (see FIG. 2).

[0057] The inverse transformation systems 310a and 310b receive the mixed left signal 336 and the mixed right signal 338 and perform an inverse signal transformation to generate a modified left signal 340 and a modified right signal 342. In particular, the inverse transformation system 310a performs an inverse signal transformation on the mixed left signal 336 to generate a modified left signal 340, and the inverse transformation system 310b performs an inverse signal transformation on the mixed right signal 338 to generate a modified right signal 342. The inverse transformation performed by the inverse transformation systems 310a and 310b generally corresponds to the inverse transformation of the transformation performed by the transformation systems 302a and 302b to transform the signal from the second signal domain back to the first signal domain. The modified left signal 340 then corresponds to a noise-reduced version of the left input signal 320, and the modified right signal 342 corresponds to a noise-reduced version of the right input signal 322. Inverse transformation systems 310a and 310b may otherwise be similar to inverse transformation system 208 (see FIG. 2).

[0058] Mono Model Training

[0059] As mentioned above, the noise reduction systems 304a and 304b use a trained model to generate the left gain 330 and the right gain 332 from the transformed left signal 324 and the transformed right signal 326. The trained model is trained offline using mono training data. The offline training process may also be called a training phase, which is contrasted with the operational phase, where the trained model is used by the audio processing system 300 during normal operation. The training phase generally has four steps:

[0060] First, a set of training data is generated. The set of training data may be generated by mixing various mono audio data source samples with various noise samples at various signal-to-noise ratios (SNRs). The mono audio data source samples generally correspond to noise-free audio data, also referred to as clean audio data, including speech, music, etc. The noise samples correspond to noisy audio data, including traffic noise, fan noise, airplane noise, construction noise, sirens, baby crying, etc. The training data may result in a corpus of approximately 100-200 hours from mixing approximately 1-2 hours of source samples with 15-25 noise samples at 5-10 SNRs. Each source sample may be between 15-60 seconds long and the SNR may range from -45 to 0 dB. For example, a given source sample of speech may be 30 seconds long and may be mixed with noise samples of traffic noise at 5 SNRs of -40, -30, -20, -10, and 0 dB, resulting in 600 seconds of training data in the corpus of training data.

[0061] Second, features are extracted from the set of training data. In general, the feature extraction process is the same as that used during operation of an audio processing system, such as 200 (see FIG. 2) and 300 (see FIG. 3), such as performing a transform and extracting features in the second signal domain. The extracted features also correspond to the features used during operation of the audio processing system.

[0062] Third, the model is trained on a set of training data. Roughly speaking, training is done by adjusting the weights of the nodes in the model in response to a comparison between the model's output and an ideal output, which corresponds to the gain required to adjust a noisy input to a noiseless output.

[0063] Finally, once the model is sufficiently trained, the resulting model is provided to an audio processing system, such as 200 in FIG. 2 or 300 in FIG. 3, for use in the operational phase.

[0064] As mentioned above, the training data is mono training data. This mono training data results in a single model that the audio processing system 300 uses for each input channel. In particular, the noise reduction system 304a uses a model trained with the transformed left signal 324 as input, and the noise reduction system 304b uses a model trained with the transformed right signal 326 as input; for example, the systems 304a and 304b may each implement a copy of the trained model. The models may also be trained using binaural training data, as described below with respect to Figures 4-5.

[0065] 4 is a block diagram of an audio processing system 400. The audio processing system 400 is a more specific embodiment of the audio processing system 200 (see FIG. 2). The audio processing system 400 may be implemented as a component of the audiovisual capture system 100 (see FIG. 1), for example, as one or more computer programs executed by a processor of the video capture device 102. The audio processing system 400 is similar to the audio processing system 300 (see FIG. 3), with differences related to trained models, as described in more detail below. The audio processing system 400 includes conversion systems 402a and 402b, a noise reduction system 404, mixing systems 406a and 406b, and inverse conversion systems 408a and 408b.

[0066] Conversion systems 402a and 402b receive a left input signal 420 and a right input signal 422 and perform a signal conversion to generate a converted left signal 424 and a converted right signal 426. Conversion systems 402a and 402b operate in a similar manner to conversion systems 302a and 302b (see FIG. 3), and the description thereof will not be repeated for the sake of brevity.

[0067] The noise reduction system 404 receives the transformed left signal 424 and the transformed right signal 426, performs gain calculations, and generates a joint gain 430. The joint gain 430 is based on both the transformed left signal 424 and the transformed right signal 426. The noise reduction system 404 performs feature extraction on the transformed left signal 424 and the transformed right signal 426 to extract a joint set of features, processes the joint set of features by inputting the joint set of features into a trained model, and generates the joint gain 430 as a result of processing the joint set of features. The joint gain 430 thus corresponds to a shared gain, and may also be referred to as a shared gain 430. The noise reduction system 404 is otherwise similar to the noise reduction systems 304a and 304b (see FIG. 3), and the description thereof will not be repeated for the sake of brevity. For example, the joint set of features may be similar features to those described above with respect to the noise reduction systems 304a and 304b. The trained models are similar to those described above with respect to noise reduction systems 304a and 304b, except that the trained models implemented by noise reduction system 404 are trained offline using binaural training data, as described below.

[0068] It should be noted that, unlike audio processing system 300 (see FIG. 3), audio processing system 400 does not need to have a gain calculation system since noise reduction system 404 outputs shared gain 430 as a result of training using binaural training data.

[0069] Mixing systems 406a and 406b receive a transformed left signal 424, a transformed right signal 426, and a shared gain 430, and apply the shared gain 430 to signals 424 and 426 to generate a mixed left signal 434 and a mixed right signal 436. In particular, mixing system 406a applies the shared gain 430 to the transformed left signal 424 to generate a mixed left signal 434, and mixing system 406b applies the shared gain 430 to the transformed right signal 426 to generate a mixed right signal 436. Mixing systems 406a and 406b are otherwise similar to mixing systems 308a and 308b (see FIG. 3), and the description thereof will not be repeated for the sake of brevity.

[0070] The inverse transformation systems 408a and 408b receive the mixed left signal 434 and the mixed right signal 436 and perform an inverse signal transformation to generate a modified left signal 440 and a modified right signal 442. In particular, the inverse transformation system 408a performs an inverse signal transformation on the mixed left signal 434 to generate a modified left signal 440, and the inverse transformation system 408b performs an inverse signal transformation on the mixed right signal 436 to generate a modified right signal 442. The inverse transformations performed by the inverse transformation systems 408a and 408b generally correspond to the inverse transformations performed by the transformation systems 402a and 402b to transform the signals from the second signal domain back to the first signal domain. As a result, the modified left signal 440 corresponds to a noise-reduced version of the left input signal 420, and the modified right signal 442 corresponds to a noise-reduced version of the right input signal 422. The inverse transformation systems 408a and 408b may otherwise be similar to the inverse transformation systems 310a and 310b (see FIG. 3).

[0071] Binaural Model Training

[0072] As mentioned above, the noise reduction system 404 uses a trained model to generate the shared gain 430 from the transformed left signal 424 and the transformed right signal 426. The trained model is trained offline using binaural training data. The use of binaural training data is in contrast to the use of mono training data used when training the models of the noise reduction systems 304a and 304b (see FIG. 3). Training the model using binaural training data is generally similar to training the model using mono training data as described above with respect to FIG. 3, and the training phase generally has four steps.

[0073] First, a set of training data is generated: the audio data source samples are binaural audio data source samples instead of the mono audio data source samples described above with respect to Figure 3. Mixing the binaural audio data source samples with noise samples at various SNRs results in a similar corpus of about 100-200 hours.

[0074] Second, features are extracted from the set of training data. Features are extracted from the binaural channels in combination, e.g., left and right channels in combination. Extracting features from the binaural channels in combination is in contrast to extraction from a single channel, as used when training the models of the noise reduction systems 304a and 304b (see FIG. 3).

[0075] Third, the model is trained on a set of training data. The training process is generally similar to the training process used when training the models of the noise reduction systems 304a and 304b (see FIG. 3).

[0076] Finally, once the model is sufficiently trained, the resulting model is provided to an audio processing system, such as 400 in FIG. 4, for use in the operational phase.

[0077] FIG. 5 is a block diagram of an audio processing system 500. The audio processing system 500 is a more specific embodiment of the audio processing system 200 (see FIG. 2). The audio processing system 500 may be implemented as a component of the audiovisual capture system 100 (see FIG. 1), for example, as one or more computer programs executed by a processor of the video capture device 102. The audio processing system 500 is similar to both the audio processing system 300 (see FIG. 3) and the audio processing system 400 (see FIG. 4), with differences related to trained models, as described in more detail below. The audio processing system 500 includes transformation systems 502a and 502b, noise reduction systems 504a, 504b and 504c, a gain calculation system 506, mixing systems 508a and 508b, and inverse transformation systems 510a and 510b.

[0078] Conversion systems 502a and 502b receive a left input signal 520 and a right input signal 522 and perform a signal conversion to generate a converted left signal 524 and a converted right signal 526. Conversion systems 502a and 502b operate in a similar manner as conversion systems 302a and 302b (see FIG. 3) or 402a and 402b (see FIG. 4), and the description thereof will not be repeated for the sake of brevity.

[0079] The noise reduction systems 504a, 504b, and 504c receive the transformed left signal 524 and the transformed right signal 526, perform gain calculations, and generate a left gain 530, a right gain 532, and a combined gain 534. In particular, the noise reduction system 504a generates the left gain 530 based on the transformed left signal 524, the noise reduction system 504b generates the right gain 532 based on the transformed right signal 326, and the noise reduction system 504c generates the combined gain 534 based on both the transformed left signal 524 and the transformed right signal 526. The noise reduction system 504a receives the transformed left signal 524, performs feature extraction on the transformed left signal 524 to extract a set of features, processes the set of features by inputting the set of features into a trained mono model, and generates the left gain 530 as a result of processing the set of features. The noise reduction system 504b receives the transformed right signal 526, performs feature extraction on the transformed right signal 526 to extract a set of features, processes the set of features by inputting the set of features into a trained mono model, and generates a right gain 532 as a result of processing the set of features. The noise reduction system 504c receives the transformed left signal 524 and the transformed right signal 526, performs feature extraction on the transformed left signal 524 and the transformed right signal 526 to extract a set of features, processes the set of features by inputting the set of features into a trained binaural model, and generates a combined gain 534 as a result of processing the set of features. The noise reduction systems 504a and 504b are otherwise similar to the noise reduction systems 304a and 304b (see FIG. 3), and the noise reduction system 504c is otherwise similar to the noise reduction system 404 (see FIG. 4), the description of which will not be repeated for the sake of brevity.

[0080] In summary, the noise reduction systems 504a and 504b implement a machine learning system using a mono model similar to the audio processing system 300 (see FIG. 3), and the noise reduction system 504c implements a machine learning system using a binaural model similar to the audio processing system 400 (see FIG. 4). Thus, the audio processing system 500 can be seen as a combination of the audio processing systems 300 and 400.

[0081] Gain calculation system 506 receives left gain 530, right gain 532, and combined gain 534 and combines gains 530, 532, and 534 according to a mathematical function to generate a shared gain 536. The mathematical function may be one or more of a maximum, average, range function, comparison function, etc. Gains 530, 532, and 534 may be gain vectors of band gains, and the mathematical function is applied to each given band of gains 530, 532, and 534. Gain calculation system 506 may otherwise be similar to gain calculation system 306 (see FIG. 3), and for the sake of brevity, a description thereof will not be repeated.

[0082] Mixing systems 508a and 508b receive a transformed left signal 524, a transformed right signal 526, and a shared gain 536 and apply the shared gain 536 to signals 524 and 526 to generate a mixed left signal 540 and a mixed right signal 542. In particular, mixing system 508a applies the shared gain 536 to the transformed left signal 524 to generate a mixed left signal 540, and mixing system 508b applies the shared gain 536 to the transformed right signal 526 to generate a mixed right signal 542. Mixing systems 508a and 508b may be otherwise similar to mixing systems 308a and 308b (see FIG. 3), and a description thereof will not be repeated for the sake of brevity.

[0083] The inverse transformation systems 510a and 510b receive the mixed left signal 540 and the mixed right signal 542 and perform an inverse signal transformation to generate a modified left signal 544 and a modified right signal 546. In particular, the inverse transformation system 510a performs an inverse signal transformation on the mixed left signal 540 to generate a modified left signal 544, and the inverse transformation system 510b performs an inverse signal transformation on the mixed right signal 542 to generate a modified right signal 546. The inverse transformation performed by the inverse transformation systems 510a and 510b generally corresponds to the inverse transformation of the transformation performed by the transformation systems 502a and 502b to transform the signal from the second signal domain back to the first signal domain. As a result, the modified left signal 544 corresponds to a noise-reduced version of the left input signal 520, and the modified right signal 546 corresponds to a noise-reduced version of the right input signal 522. Inverse transformation systems 510a and 510b may otherwise be similar to inverse transformation systems 310a and 310b (see FIG. 3) or 408a and 408b (see FIG. 4).

[0084] Model Training

[0085] As previously mentioned, the noise reduction systems 504a, 504b, and 504c use trained mono and binaural models to generate gains 530, 532, and 534 from the transformed left signal 524 and the transformed right signal 526. The training of the mono model is generally similar to the training of the model used by the noise reduction systems 304a and 304b (see FIG. 3), and the training of the binaural model is generally similar to the training of the model used by the noise reduction system 404 (see FIG. 4), and the description thereof will not be repeated for the sake of brevity.

[0086] 2. Combined binaural audio and video capture

[0087] As mentioned above, UGC often includes combined audio and video capture. Simultaneous capture of video and binaural audio is particularly challenging. One such challenge is when binaural audio capture and video capture are performed by separate devices, e.g., capturing video with a mobile phone and binaural audio with earphones. Mobile phones typically include two cameras: a front camera, also called a selfie camera, and a back camera, also called a main camera. When the back (main) camera is in use, this is sometimes called normal mode. When the front (selfie) camera is in use, this is sometimes called selfie mode. In normal mode, the user with the video capture device is behind the scene captured on video. In selfie mode, the user with the video capture device is present in the scene captured on video.

[0088] When binaural audio capture and video capture are performed on separate devices, there may be a mismatch between the captured video data and the captured binaural audio data as compared to human perception of the environment with the eyes and ears. One example of such a mismatch includes the perception of binaural audio captured simultaneously with the video in normal mode versus the perception of binaural audio captured simultaneously with the video in selfie mode. Another example of such a mismatch includes the discontinuities that occur when switching between normal mode and selfie mode. The following sections describe various processes for compensating for these mismatches.

[0089] 3. Binaural audio capture in selfie mode

[0090] FIG. 6 is a stylized overhead view illustrating binaural audio capture in selfie mode using video capture system 100 (see FIG. 1). Video capture device 102 is in selfie mode and uses a front camera to capture a video including a user in a scene. The user is wearing earphones 104 and 106 to capture binaural audio of the scene. Video capture device 102 is between about 0.5 m and 1.5 m in front of the user, depending on whether the user is holding video capture device 102 in their hand or using a selfie stick to hold video capture device 102 in their hand. Video capture device 102 also captures other people who are close to the user, e.g., a person behind the user to the left of the user, referred to as the left person, and a person behind the user to the right of the user, referred to as the right person. Because the audio is captured binaurally, the listener perceives sounds made by the person on the left as originating from behind and to the left, and the listener perceives sounds made by the person on the right as originating from behind and to the right. This involves some corrections in selfie mode.

[0091] 3.1 Left / Right Correction

[0092] The opposite orientation of the user wearing earphones 104 and 106 and the front (selfie) camera of video capture device 102 results in a left / right inversion of the captured binaural audio content: a consumer of the captured audiovisual content perceives sound coming from the right earphone as coming from a source appearing on the left side of the video, and sound coming from the left earphone as coming from a source appearing on the right side of the video, which is inconsistent with our experience of seeing with our eyes and hearing with our ears.

[0093] Left / right compensation involves taking the left channel from the input and sending it to the right channel of the output, or expressed as an equation: R'=L; it involves taking the right channel from the input and sending it to the left channel of the output, or expressed as an equation: L'=R.

[0094] 3.2 Before / after correction

[0095] To record other speakers in the same scene when using a front (selfie) camera, a user wearing earphones 104 and 106 and with video capture device 102 often stands slightly in front of the other speakers, i.e., closer to the camera. Thus, in the case of captured binaural audio, the speech of the other speakers comes from behind the listener consuming the content, whereas in the captured video, all speakers appear in front.

[0096] To generally correct for this and increase perceptual consistency between audio and video, embodiments may implement front / back correction, which operates to modify the spectral shape of sounds coming from behind the listener so that the sounds are perceived in a similar manner as sounds coming from the front.

[0097] The embodiments disclosed herein may implement the spectral shape modification using a high-shelf filter. The high-shelf filter may be constructed in various ways. For example, it may be implemented using an infinite impulse response (IIR) filter, such as a biquad filter.

[0098] Figure 7 is a graph showing an example of the magnitude response of a high shelf filter implemented using a biquad filter. In Figure 7, the x-axis is frequency (kHz) and the y-axis is the magnitude of the loudness adjustment the filter applies to the signal. The high shelf frequency in this example is around 3 kHz, a typical value given the shading effects of the human head. Because rear captured audio is attenuated at higher frequencies such as 5 kHz and above as shown in Figure 7, the filter implements a high shelf that boosts these frequencies when the audio is corrected forward.

[0099] The embodiments disclosed herein may also perform spectral shape modification using an equalizer, which boosts or attenuates the input audio in one or more bands with different gains and may be implemented by an IIR filter or a finite impulse response (FIR) filter. An equalizer can shape the spectrum with greater precision, with a typical configuration being an 8 to 12 dB boost in the 3 to 8 kHz frequency range for pre / post correction.

[0100] 3.3 Stereo image width control

[0101] 8 is a stylized overhead view showing various audio capture angles in selfie mode. Angle θ1 corresponds to the angle of the right person's sound captured by the microphone of video capture device 102 (see FIG. 6), and angle θ2 corresponds to the angle of the right person's sound captured by right earphone 106. Compared to when there is a microphone on video capture device 102, earphones 104 and 106 are usually closer to the line where other speakers would usually stand, so θ2>θ1, which means that the speech of other speakers comes from a direction closer to the side, whereas based on the video scene, the viewer expects the speech to come from a direction closer to the center.

[0102] To address this issue, embodiments may implement stereo image width control to improve consistency between video and binaural audio recordings by compressing the perceived width of the binaural audio. In one implementation, compression is achieved by attenuating the side components of the binaural audio. First, the input binaural audio is converted to a middle-side representation according to equations (1.1) and (1.2).

number

[0103] In equations (1.1) and (1.2), L and R are the left and right channels of the input audio, e.g., the left and right input signals 220 and 222 in FIG. 2, while M and S are the center and side components resulting from the transformation.

[0104] The side channel S is then attenuated by the attenuation factor α and the processed output audio L′ and R′ are given by equations (2.1) and (2.2).

number

[0105] The attenuation coefficient α can be a function of the focal length f of the front (selfie) camera and is given by equation (3):

number

[0106] In formula (3), f c is the expected focal length = 1, i.e. the focal length at which no attenuation of the side component S is applied, also called the baseline focal length; γ is an aggressiveness factor, which will be explained in more detail with reference to FIG. 9.

[0107] FIG. 9 is a graph of the attenuation coefficient α for different focal lengths f. In FIG. 9, the x-axis is focal length f ranging from 10 mm to 35 mm, and the y-axis is the attenuation coefficient α, the baseline focal length f c is 70mm, and the aggressiveness factor γ can be selected from [1.2 1.5 2.0 2.5]. The aggressiveness factor γ can be selected by the device manufacturer to provide different options for the camera. For a typical front (selfie) camera of a smartphone with F=30mm, α is in the range of 0.5 to 0.7.

[0108] In summary, when video is captured at a smaller focal length, the video appears zoomed out and the captured audio for the left and right people all appear to originate from the center of the video, so the width control compensates for the captured audio by shrinking the audio scene to fit the video scene.

[0109] 4. Binaural audio capture in normal mode

[0110] FIG. 10 is a stylized overhead view showing binaural audio capture in normal mode using video capture system 100 (see FIG. 1). Video capture device 102 is in normal mode and uses a rear camera to capture video that does not include the user in the scene. The user is wearing earphones 104 and 106 to capture binaural audio of the scene. In contrast to selfie mode (see FIGS. 6 and 8) where the user is often captured in the video scene, in normal mode the user is not often captured in the video scene. In normal mode, the user wearing earphones 104 and 106 and with video capture device 102 is usually behind the video scene. Other people are usually in front, as captured in the video, as shown with people on the left and right. Angle θ1 corresponds to the angle of the sound of the right person captured by the microphone of video capture device 102, and angle θ2 corresponds to the angle of the sound of the right person captured by the right earphone 106.

[0111] In normal mode, the audio processing system does not need to perform either left / right correction or front / back correction as may be done in selfie mode. For stereo image width control, the binaural audio may be perceived a little wider in this mode since θ2<θ1 since earphones 104 and 106 are usually farther away from the line where other talkers usually stand compared to when the microphone is at video capture device 102. However, the difference between θ1 and θ2 is not that large compared to selfie mode, so for simplicity, a typical approach is to leave the binaural audio as is.

[0112] 5. Switch between normal and selfie mode

[0113] Different audio processing is often applied in normal mode compared to selfie mode. For example, left / right correction is performed in selfie mode but not in normal mode. When a user switches between modes, it is beneficial for the audio processing system to perform the switch smoothly. The switch may occur during real-time operations, such as capturing content for broadcast or streaming, and during non-real-time operations, such as capturing content for later processing or uploading.

[0114] 5.1 Smoothing for left / right compensation and stereo image width control

[0115] Recall from section 3 that the equations for left / right correction are L'=R and R'=L. These can be rewritten as equations (4.1-4.4):

number

[0116] In equations (4.1-4.4), the attenuation coefficient α for left / right correction in selfie mode is -1.

[0117] No left / right correction is needed for normal mode, so α=-1 in that mode. Thus, during switching between normal mode and selfie mode, α switches between 1 and -1. To ensure that the switching is smooth, α should change its value gradually. An example of an equation for performing a smooth transition is given in equation (5):

number

[0118] In equation (5), t s is the time at which the switch is performed, and a transition time of 1 second works well for left / right compensated switches. Therefore, the transition is performed at t s Start with -0.5, t s +0.5. In the real-time case, equation (5) is t s and ending at 1 second. The value of 1 second can be adjusted as needed, for example, to a range of 0.5 to 1.5 seconds.

[0119] The stereo image width control uses a similar set of equations, as expressed in equations (6.1-6.4).

number

[0120] However, in equations (6.1-6.4), the attenuation coefficient α ranges from 0.5 to 0.7 in selfie mode and 1.0 in normal mode.

[0121] That is, the stereo image width control involves generating a center channel M and a side channel S, attenuating the side channel by a width adjustment factor α, and generating modified audio signals L' and R' from the attenuated center and side channels. The width adjustment factor is calculated based on a focal length of the video capture device, and the width adjustment factor may be updated in real time as the video capture device changes its focal length in real time.

[0122] Combining the stereo image width control and left / right smoothing correction, we have α ranging from -0.5 to -0.7 for selfie mode and 1.0 for normal mode. Assuming α = -0.5 as an example results in equation (7):

number

[0123] In equation (7), t s is the time at which the switch occurs, and a transition time of 1 second works well for a combination of left / right compensation switching and stereo image width control switching. Therefore, in the non-real-time case, the transition is t s Starting at -0.5, t s +0.5. In the real-time case, equation (7) is t s and ending at 1 second. The value of 1 second can be adjusted as needed, for example, to a range of 0.5 to 1.5 seconds.

[0124] 5.2 Pre / Post Correction Smoothing

[0125] As explained in sections 3 and 4, in selfie mode, pre / post correction is applied as a spectral transformation, while in normal mode, no pre / post correction is applied.

[0126] x org Let x be the input for pre / post correction. fb Let denote the output of pre / post correction. Then, the smoothed output of pre / post correction is given by equation (8).

number

[0127] In equation (8), α=0 for selfie mode and α=1 for normal mode. An example of a smooth transition equation is given by equation (9).

number

[0128] In equation (9), t s is the time when the switch occurs, and a transition time of 6 seconds works well for pre / post correction. Therefore, the transition is performed at t s Start with -3, t s +3. In the real-time case, equation (9) is t s and ending at 6 seconds. The value of 6 seconds can be adjusted as needed, for example, to a range of 3 to 9 seconds.

[0129] Front / back smoothing uses longer transition times (e.g. 6 seconds) than those used for left / right and stereo image width smoothing (e.g. 1 second), because front / back transitions involve timbre changes that are less perceptible when longer transition times are used.

[0130] 6. Exemplary Device Architecture

[0131] 11 is a device architecture 1100 for implementing the features and processes described herein, according to one embodiment. The architecture 1100 may be implemented in any electronic device, including, but not limited to, desktop computers, consumer audio / visual (AV) equipment, radio broadcasting equipment, mobile devices such as smartphones, tablet computers, laptop computers, wearable devices, etc. In the exemplary embodiment shown, the architecture 1100 is for a mobile phone. The architecture 1100 includes processor(s) 1101, peripherals interface 1102, audio subsystem 1103, speaker 1104, microphone 1105, sensors 1106, such as accelerometer, gyro, barometer, magnetometer, camera, etc., location processor 1107, such as a GNSS receiver, etc., wireless communication subsystem 1108, such as Wi-Fi, Bluetooth, cellular, etc., and I / O subsystem(s) 1109, including touch controller 1110 and other input controllers 1111, touch surface 1112 and other input / control devices 1113. Other architectures having more or fewer components may be used to implement the disclosed embodiments.

[0132] Memory interface 1114 is coupled to processor 1101, peripherals interface 1102, and memory 1115, e.g., Flash, RAM, ROM, etc. Memory 1115 stores computer program instructions and data, including, but not limited to, operating system instructions 1116, communications instructions 1117, GUI instructions 1118, sensor processing instructions 1119, telephony instructions 1120, electronic messaging instructions 1121, web browsing instructions 1122, audio processing instructions 1123, GNSS / navigation instructions 1124, and applications / data 1125. Audio processing instructions 1123 include instructions for performing the audio processing described herein.

[0133] According to one embodiment, the architecture 1100 may correspond to a mobile phone that captures video data and connects to earphones that capture binaural audio data (see FIG. 1).

[0134] Figure 12 is a flowchart of a method 1200 of audio processing. The method 1200 may be performed by a device, e.g., a laptop computer, a mobile phone, etc., having components of the architecture 1100 of Figure 11 to implement the functionality of the video capture system 100 (see Figure 1), the audio processing system 200 (see Figure 2), etc., by executing one or more computer programs.

[0135] At 1202, an audio signal is captured by an audio capture device. The audio signal has at least two channels, including a left channel and a right channel. For example, the left earphone 104 (see FIG. 1) may capture the left channel (e.g., 220 in FIG. 2) and the right earphone 106 may capture the right channel (e.g., 222 in FIG. 2).

[0136] At 1204, a noise reduction gain for each channel of the at least two channels is calculated by a machine learning system. The machine learning system may perform feature extraction, process the extracted features by inputting the extracted features into a trained model, and output a noise reduction gain as a result of processing the features. The trained model may be a mono model, a binaural model, or both a mono model and a binaural model. At 1206, a shared noise reduction gain is calculated based on the noise reduction gain for each channel.

[0137] Steps 1204 and 1206 may be performed as individual steps or as sub-steps of a combined operation. For example, the noise reduction system 204 (see FIG. 2) may calculate the left gain 230 and the right gain 232 as a shared noise reduction gain. As another example, the noise reduction system 304a (see FIG. 3) may generate the left gain 330 and the noise reduction system 304b may generate the right gain 332; the gain calculation system 306 may then generate the shared gain 334 by combining the gains 330 and 332 according to a mathematical function. As another example, the noise reduction system 404 (see FIG. 4) may calculate the combined gain 430 as a shared noise reduction gain. As another example, noise reduction system 504a (see FIG. 5) may generate a left gain 530, noise reduction system 504b may generate a right gain 532, and noise reduction system 504c may generate a combined gain 534; gain calculation system 506 may then generate a shared gain 536 by combining gains 530, 532, and 534 according to a mathematical function.

[0138] At 1208, a modified audio signal is generated by applying a plurality of shared noise reduction gains to each channel of the at least two channels. For example, the mixing system 206 (see FIG. 2) may generate the mixed left signal 234 and the mixed right signal 236 by applying the left gain 230 and the right gain 232 to the transformed left signal 224 and the transformed right signal 226. As another example, the mixing system 308a (see FIG. 3) may generate the mixed left signal 336 by applying the shared gain 334 to the transformed left signal 324, and the mixing system 308b may generate the mixed right signal 338 by applying the shared gain 334 to the transformed right signal 326. As another example, mixing system 406a (see FIG. 4) may generate a mixed left signal 434 by applying a shared gain 430 to the transformed left signal 424, and mixing system 406b may generate a mixed right signal 436 by applying a shared gain 430 to the transformed right signal 426. As another example, mixing system 508a (see FIG. 5) may generate a mixed left signal 540 by applying a shared gain 536 to the transformed left signal 524, and mixing system 508b may generate a mixed right signal 542 by applying a shared gain 536 to the transformed right signal 526.

[0139] Method 1200 may include additional steps corresponding to other functions of the audio processing system described herein. One such function is converting an audio signal from a first signal domain to a second signal domain, performing audio processing in the second signal domain, and converting the processed audio signal back to the first signal domain, e.g., using conversion system 202 and inverse conversion system 208 of FIG. 2. Another such function is simultaneous video capture and audio capture, e.g., including one or more of front / back correction, left / right correction, and stereo image width control correction, as described in Sections 3-4. Another such function is smoothly switching between selfie mode and normal mode, e.g., including smoothing the left / right correction using a first smoothing parameter and smoothing the front / back correction using a second smoothing parameter, as described in Section 5.

[0140] 7. Alternative Embodiments

[0141] Many of the features are described above in combination, but this is primarily due to the synergistic effects that arise from the combination. Many of the features may be implemented independently of other features and still provide advantages over existing systems.

[0142] 7.1 Single camera system

[0143] Although some of the features are described here in the context of a video capture device with two cameras, many of the features are also applicable to video capture devices with a single camera: for example, single camera systems also benefit from the binaural adjustments made in normal mode as described in Section 4.

[0144] 7.2 Smooth switching of video capture modes

[0145] FIG. 13 is a flow chart of a method 1300 of audio processing. Method 1200 (see FIG. 12) performs noise reduction with smooth switching as an additional feature as described in Section 5, but smooth switching can be performed independently of noise reduction. Method 1300 describes performing smooth switching independent of noise reduction. Method 1300 can be performed by a device, e.g., a laptop computer, a mobile phone, etc., having components of architecture 1100 of FIG. 11 to implement the functionality of video capture system 100 (see FIG. 1), e.g., by executing one or more computer programs.

[0146] At 1302, an audio signal is captured by an audio capture device. The audio signal has at least two channels, including a left channel and a right channel. For example, the left earphone 104 (see FIG. 1) may capture the left channel (e.g., 220 in FIG. 2) and the right earphone 106 may capture the right channel (e.g., 222 in FIG. 2).

[0147] At 1304, a video signal is captured by the video capture device simultaneously with capturing the audio signal (see 1302). For example, video capture device 102 (see FIG. 1) may capture a video signal simultaneously with capturing a binaural audio signal at earphones 104 and 106.

[0148] At 1306, the audio signal is corrected to generate a corrected audio signal. The correction may include one or more of front / back correction, left / right correction, and stereo image width correction.

[0149] At 1308, the video signal is switched from the first camera mode to the second camera mode. For example, the video capture device 102 (see FIG. 1) may switch from a selfie mode (see FIGS. 6 and 8) to a normal mode (see FIG. 10) or from a normal mode to the selfie mode.

[0150] At 1310, smooth switching of the compensated audio signal is performed simultaneously with switching the video signal (see 1308). The smooth switching may use a first smoothing parameter for smoothing one type of compensation (e.g., left / right smoothing using equation (5) or combined left / right and stereo image width smoothing using equation (7)) and a second smoothing parameter for smoothing another type of compensation (e.g., pre / post compensation using equation (9)).

[0151] Implementation details

[0152] An embodiment may be implemented in hardware, executable modules stored on a computer-readable medium, or a combination of both, such as a programmable logic array. Unless otherwise specified, steps performed by an embodiment need not be inherently related to a particular computer or other apparatus, even if they are in a particular embodiment. In particular, various general-purpose machines may be used with programs written in accordance with the teachings herein, or it may be more convenient to construct more specialized apparatus, such as, for example, integrated circuits, to perform the required method steps. Thus, an embodiment may be implemented in one or more computer programs running on one or more programmable computer systems, each having at least one processor, at least one data storage system including volatile and non-volatile memory and / or storage elements, at least one input device or port, and at least one output device or port. The program code is applied to input data to perform the functions described herein and to generate output information. The output information is applied to one or more output devices, in a known manner.

[0153] Each such computer program is preferably stored or downloaded onto a general-purpose or special-purpose programmable computer-readable storage medium or device, such as a solid-state memory or medium, magnetic or optical medium, etc., for configuring and operating a computer when the storage medium or device is read by a computer system to perform the procedures described herein. The system of the present invention may also be implemented as a computer-readable storage medium configured with a computer program, and the storage medium so configured may be considered to cause a computer system to operate in a particular, predetermined manner to perform the functions described herein. Software itself, and intangible or transitory signals, are excluded to the extent that they are non-patentable subject matter.

[0154] Aspects of the systems described herein may be implemented in any suitable computer-based sound processing network environment for processing digital or digitized audio files. Part of an adaptive audio system may include one or more networks having any number of individual machines, including one or more routers (not shown) that serve to buffer and route data transmitted between computers. Such networks may be built on a variety of different network protocols and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0155] One or more components, blocks, processes, or other functional components may be implemented through a computer program that controls the execution of a processor-based computing device of the system. It should also be noted that the various functions disclosed herein may be described using any number of combinations of hardware, firmware, and / or in terms of their operations, register transfers, logic components, and / or other characteristics as data and / or instructions embodied in various machine-readable or computer-readable media. The computer-readable media on which such formatted data and / or instructions may be embodied include, but are not limited to, various forms of physical, non-transitory, non-volatile storage media, such as optical, magnetic, or semiconductor storage media.

[0156] The above description illustrates various embodiments of the present disclosure, along with examples of how aspects of the disclosure may be implemented. The above examples and embodiments should not be considered as the only embodiments, but are presented to illustrate the flexibility and advantages of the present disclosure as defined by the following claims. Based on the above disclosure and the following claims, other configurations, embodiments, implementations and equivalents will be apparent to those skilled in the art and may be adopted without departing from the spirit and scope of the disclosure as defined by the claims.

Claims

1. 1. A computer-implemented method of audio processing, the method comprising: capturing, by an audio capture device, an audio signal having at least two channels, including a left channel and a right channel; calculating, by a machine learning system, a plurality of noise reduction gains for each channel of the at least two channels; calculating a plurality of shared noise reduction gains based on the plurality of noise reduction gains for each of the channels; and generating a modified audio signal by applying the plurality of shared noise reduction gains to each channel of the at least two channels; method.

2. converting the audio signal from a first signal domain to a second signal domain, the first signal domain being a time domain, and the plurality of noise reduction gains being calculated based on the audio signal converted to the second signal domain; and converting the modified audio signal from the second signal domain to the first signal domain. The method of claim 1.

3. the calculating the plurality of noise reduction gains, the calculating the plurality of shared noise reduction gains, and the generating the modified audio signal are performed simultaneously with capturing the audio signal. The method according to claim 1 or 2.

4. storing the captured audio signal; the calculating the plurality of noise reduction gains, the calculating the shared noise reduction gain, and the generating the modified audio signal are performed on the stored audio signal. The method according to claim 1 or 2.

5. Calculating the plurality of noise reduction gains by the machine learning system includes: performing feature extraction on each of the at least two channels to generate a plurality of features for each channel; processing a plurality of features for each channel, the processing comprising inputting the plurality of features for each channel into a machine learning model; and outputting the plurality of noise reduction gains from the machine learning system as a result of inputting the plurality of features into the machine learning model.

5. The method according to any one of claims 1 to 4.

6. the machine learning model is a mono model trained offline using mono audio training data; the plurality of features includes a first plurality of features corresponding to the left channel and a second plurality of features corresponding to the right channel; the plurality of noise reduction gains including a first plurality of noise reduction gains corresponding to the first plurality of features and a second plurality of noise reduction gains corresponding to the second plurality of features; The method according to claim 5.

7. the machine learning model is a binaural model trained offline using binaural audio training data; the plurality of features being a combined plurality of features corresponding to both the left channel and the right channel; the plurality of shared noise reduction gains result from the plurality of combined features corresponding to both the left channel and the right channel; The method according to claim 5.

8. the machine learning models include a mono model trained offline using mono audio training data and a binaural model trained offline using binaural speech training data; the plurality of features includes a first plurality of features corresponding to the left channel, a second plurality of features corresponding to the right channel, and a combined plurality of features corresponding to both the left channel and the right channel; the plurality of noise reduction gains include a first plurality of noise reduction gains corresponding to the first plurality of features, a second plurality of noise reduction gains corresponding to the second plurality of features, and a combination plurality of noise reduction gains corresponding to the combination plurality of features; The method according to claim 5.

9. the audio capture device having a first earphone for capturing the left channel and a second earphone for capturing the right channel; the plurality of noise reduction gains includes a first plurality of noise reduction gains and a second plurality of noise reduction gains; calculating the plurality of shared noise reduction gains includes combining the first plurality of noise reduction gains and the second plurality of noise reduction gains according to a mathematical function; 9. The method according to any one of claims 1 to 8.

10. The mathematical function includes one or more of an average, a maximum, a range function, and a comparison function.

10. The method of claim 9.

11. the first plurality of noise reduction gains correspond to a first gain vector for a plurality of bands of the left channel and the second plurality of noise reduction gains correspond to a second gain vector for a plurality of bands of the right channel; calculating the plurality of shared noise reduction gains includes selecting a maximum gain for each band of the plurality of bands from the first gain vector and the second gain vector; 10. The method of claim 9.

12. the plurality of noise reduction gains further comprising a combination of a plurality of noise reduction gains; calculating the plurality of shared noise reduction gains includes combining the first plurality of noise reduction gains, the second plurality of noise reduction gains, and the combined plurality of noise reduction gains according to the mathematical function.

10. The method of claim 9.

13. capturing a video signal by a video capture device simultaneously with capturing the audio signal; The video capture device includes a mobile phone, the mobile phone including a front camera and a rear camera.

13. The method according to any one of claims 1 to 12.

14. and switching from a first mode using one of the front camera and the rear camera to a second mode using the other of the front camera and the rear camera, the switching including smoothing left / right correction of the audio signal using a first smoothing parameter and smoothing pre / post correction of the audio signal using a second smoothing parameter. The method of claim 13.

15. capturing the video signal simultaneously with capturing the audio signal includes performing a correction to the audio signal, the correction including at least one of a left / right correction, a front / back correction, and a stereo image width control correction.

15. The method according to claim 13 or 14.

16. Performing the stereo image width control correction comprises: generating a center channel and a side channel from the left channel and the right channel of the audio signal; attenuating the side channel by a width adjustment factor; and generating a modified audio signal from the center channel and the attenuated side channel; The method of claim 15.

17. the width adjustment factor is calculated based on a focal length of the video capture device; 17. The method of claim 16.

18. the width adjustment factor is updated in real time in response to the video capture device changing the focal length in real time.

20. The method of claim 17.

19. A non-transitory computer readable medium storing a computer program which, when executed by a processor, controls an apparatus to carry out processes including the method of any one of claims 1 to 18.

20. 1. An apparatus for audio processing, comprising: A processor configured to control the device to perform a process including the method of any one of claims 1 to 18. Device.

Images