Rendering audio captured with multiple devices

Abstract

An audio processing method includes: receiving user-generated content having two audio sources; extracting an audio object and a residual signal; adjusting the audio object and the residual signal according to the head movement of a listener; and mixing the adjusted audio signal to generate a binaural audio signal. In this manner, the binaural signal is adjusted according to the head movement of the listener without the need for a perfect audio object.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to International Patent Application No. PCT / CN2022 / 114596 filed on August 24, 2022, U.S. Provisional Patent Application No. 63 / 432,385 filed on December 14, 2022, and U.S. Provisional Patent Application No. 63 / 509,121 filed on June 20, 2023, each of which is incorporated by reference in its entirety. Technical Field

[0003] The present disclosure relates to audio processing, and in particular to processing audio captured by binaural microphones and additional microphones. Background Art

[0004] Unless otherwise indicated herein, the approaches described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.

[0005] Devices for audio-visual capture are becoming increasingly popular with consumers. Such devices include portable cameras (such as the Sony Action Cam TM Camera and GoPro TM Cameras) and mobile phones with integrated camera functionality. Typically, the device captures audio at the same time as the video, for example by using a mono or stereo microphone. Audiovisual content sharing systems such as YouTube TM Services and Twitch.tv TM Services) are also becoming increasingly popular. Users upload the captured audiovisual content to the content sharing system, or broadcast the captured audiovisual content while capturing. Because the content is generated by the user, it is called user-generated content (UGC) compared to professional-generated content (PGC) that is usually generated by professionals. The difference between UGC and PGC is usually that UGC is created using consumer equipment that may be cheaper and have fewer features than professional equipment. Another difference between UGC and PGC is that UGC is usually captured in an uncontrolled environment (such as outdoors), while PGC is usually captured in a controlled environment (such as a recording studio).

[0006] Another difference between UGC and PGC is that PGC can use perfect audio objects, while UGC may not. For example, a PGC content creator can position high-resolution audio at a specific object location, and the PGC system can generate an audio object that exactly corresponds to the content creator's intent; such an audio object is called a perfect audio object. In contrast, UGC content creators generally cannot use perfect audio objects.

[0007] Binaural audio includes audio recorded using two microphones located at the user's ear positions. The captured binaural audio (which may be referred to as immersive audio) produces an immersive listening experience when replayed via headphones. Compared with stereo audio, binaural audio also includes the head shadow effect of the user's head and ears, resulting in inter-aural time differences and inter-aural sound level differences when capturing binaural audio. Binaural audio is also different from stereo in that stereo audio may involve speaker crosstalk between speakers. PGC binaural audio can be captured in a studio environment with controllable sound sources and acoustics. UGC binaural audio can be captured by earplugs and may include unwanted sounds from the surrounding environment.

[0008] Head tracking (head tracking or headtracking) generally refers to tracking the orientation of a user's head to adjust the input to a system or the output of a system. For audio, head tracking refers to changing the audio signal according to the listener's head orientation. Summary of the invention

[0009] Existing audio-visual capture systems for UGC have many problems. One problem is that UGC does not typically use perfect audio objects because consumer capture devices typically cannot capture perfect audio objects. Another problem is that UGC content creators can typically use a mobile phone to capture audio-visual content or can use binaural earbuds to capture binaural audio content; however, UGC content creators do not have a good way to integrate the output of these two devices.

[0010] In view of these issues, embodiments are directed to processing audio from multiple sources and adjusting the audio based on a listener's head movement.

[0011] According to an embodiment, a computer-implemented audio processing method includes receiving, by one or more playback devices, user generated content (UGC) captured by capture devices connected to each other. Each audio source of the UGC corresponds to a corresponding characteristic in an audio scene. The method further includes receiving, by the one or more playback devices, information indicating a listener behavior of a user of the one or more playback devices from one or more sensors of the one or more playback devices. The listener behavior may include a listener's head movement. The method further includes adapting the UGC according to the listener behavior, including compensating for characteristics of an audio source according to the listener behavior. The method further includes rendering the adapted UGC to provide a listener with an interactive experience about the audio scene.

[0012] As a result, even for user-generated content, the output audio responds to the listener's head movements without requiring perfect audio objects or professionally generated content.

[0013] According to another embodiment, an apparatus includes a processor. The processor is configured to control the apparatus to implement one or more of the methods described herein. The apparatus may further include details similar to the details of one or more of the methods described herein.

[0014] According to another embodiment, a non-transitory computer-readable medium stores a computer program that, when executed by a processor, controls an apparatus to perform a process including one or more of the methods described herein.

[0015] The following detailed description and accompanying drawings provide a further understanding of the nature and advantages of various embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1A to Figure 1B is the view of a user with a UGC capture device.

[0017] Figure 2 is a block diagram of a system 200 for interactive rendering of UGC captured with multiple devices.

[0018] Figure 3 FIG. 2 is a diagram showing the HRTF adjuster 220 (see FIG. Figure 2 ) with additional details.

[0019] Figure 4 The rebalancer 230 (see Figure 2 ) with additional details.

[0020] Figure 5 The mixer 240 (see Figure 2 ) with additional details.

[0021] FIG. 6 is a device architecture 600 for implementing the features and processes described herein, according to an embodiment.

[0022] Figure 7 is a flow chart of an audio processing method 700 . DETAILED DESCRIPTION

[0023] This article describes technologies related to audio processing. In the following description, for the purpose of explanation, many examples and specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it is apparent to those skilled in the art that the present disclosure as defined by the claims may include some or all of the features in these examples alone or in combination with other features described below, and may further include modifications and equivalents of the features and concepts described herein.

[0024] In the following description, various methods, processes, and procedures are described in detail. Although particular steps may be described in a certain order, such order is primarily for convenience and clarity. Particular steps may be repeated more than once, may occur before or after other steps even if the steps are otherwise described in another order, and may occur in parallel with other steps. A second step need only be performed after a first step if the first step must be completed before starting the second step. When this is not clear from the context, this will be specifically noted.

[0025] In this document, the terms "and", "or" and "and / or" are used. Such terms should be understood to have an inclusive meaning. For example, "A and B" may at least mean the following meanings: "both A and B", "at least both A and B". As another example, "A or B" may at least mean the following meanings: "at least A", "at least B", "both A and B", "at least both A and B". As another example, "A and / or B" may at least mean the following meanings: "A and B", "A or B". When exclusive or is intended to be used, this will be clearly indicated, for example, "either A or B", "at most one of A and B", etc.

[0026] This document describes various processing functions associated with structures such as blocks, elements, components, circuits, etc. Generally, these structures can be implemented by processors controlled by one or more computer programs.

[0027] Figure 1A to Figure 1B is the view of a user with a UGC capture device. Figure 1A is a side perspective view, and Figure 1B It is a top view. Figure 1A to Figure 1B A user 102 is shown holding a mobile phone 104 and wearing earbuds 106a and 106b (collectively 106). Mobile phone 104 typically includes a camera, microphone, screen, speaker, processor, volatile and non-volatile memory and storage devices, radio, and other components. Examples of mobile phones 104 include the Apple iPhone TM Mobile phone, Samsung Galaxy TMMobile phones, etc. The earbuds 106 can communicate with each other, for example, via IEEE 802.15.1 standard protocols (such as Bluetooth TM The earbuds 106 are wirelessly connected to the mobile phone 104 using a Bluetooth® protocol. The earbuds 106 typically include a speaker, a microphone, a processor, volatile and non-volatile memory and storage devices, a radio, and other components.

[0028] The user 102 uses these devices to capture UGC of the surrounding environment (referred to as the audio-visual scene 110). The user 102 may hold the mobile phone 104 in his hand or on a selfie stick to capture UGC; for example, using the phone's screen (in the front, facing the user) to frame the video scene in front of the user, using the phone's camera (in the back, facing the video scene) to capture video, and using the phone's microphone to capture audio (e.g., a single microphone captures mono audio, two microphones capture stereo audio, etc.). The user may use earbuds 106 to capture binaural audio of the audio-visual scene 110 while using the mobile phone 104 to capture audio and video.

[0029] As discussed in the background, there is no simple way for existing UGC devices to integrate the output of these two devices, especially because both are capturing audio. For example, when playing back content, the listener may have to choose between the audio captured by the mobile phone 104 (which is not binaural audio) and the binaural audio captured by the earplug 106. Subsequent sections describe how the system described herein integrates these two audio sources. As another example, the listener itself may have earplugs with head tracking capabilities, but there is no simple way to adjust the captured UGC binaural audio to take into account the listener's head movement. Subsequent sections also describe how the system described herein solves this problem.

[0030] Figure 2 1 is a block diagram of a system 200 for interactive rendering of UGC captured with multiple devices. The system 200 can be implemented using multiple devices, including two or more capture devices (e.g., earbuds and a mobile phone), a server device, a playback device, etc. The device implementing the system 200 may include circuits such as a microprocessor that execute a computer program that implements the functions of the system 200. The system 200 includes an object extractor 210, a head-related transfer function (HRTF) adjuster 220, a rebalancer 230, a mixer 240, and a remixer 250.

[0031] The object extractor 210 receives an audio signal 262 and a binaural audio signal 264, performs object extraction, and generates one or more binaural objects 266 and a residual signal 268. The audio signal 262 is captured by a UGC audio-visual capture device such as a mobile phone, wherein the audio signal 262 is captured simultaneously with the video data. The audio signal 262 has N channels, which generally correspond to the number of microphones of the UGC audio-visual capture device. For example, the audio signal 262 may have 1 channel (for captured mono audio), 2 channels (for captured stereo audio), etc. The mobile phone used to capture the audio signal 262 may have two microphones (e.g., at the bottom and top, at the left and right, at the back and front, etc.), three microphones (at the bottom, top, left, right, front, back or omnidirectional microphones, etc.), etc. The binaural audio signal 264 is captured by a UGC audio capture device such as binaural earbuds. The binaural audio signal 264 generally has 2 channels.

[0032] Binaural objects 266 generally correspond to audio data that object extractor 210 has localized to an identified location in the audio-visual scene. For example, bird chirping or airplane noise can be extracted to generate a height object. Similarly, recognized sounds originating from the left side of the capture device can be extracted to generate a second audio object, and recognized sounds originating from the right side of the capture device can be extracted to generate a third audio object.

[0033] The residual signal 268 corresponds to the audio signal 262 and the binaural audio signal 264 excluding the binaural object 266. The residual signal 268 has N+2 channels, corresponding to the N-channel residual of the audio signal 262 (excluding the binaural object 266) and the 2-channel residual of the binaural audio signal 264 (excluding the binaural object 266).

[0034] The object extractor 210 may implement a machine learning system for extracting binaural objects 266 from the audio input. Typically, the machine learning system has a model that has been trained using training data during a training phase. During an operation phase, the machine learning system uses the model as part of processing the input data in order to generate an output of the machine learning system.

[0035] Based on an embodiment, the machine learning system implements a model trained based on the signal-to-noise ratio (SNR) of a training audio data set. The model can have multiple sub-models or layers and can be configured to process sparse objects. In operation, the machine learning system performs feature extraction on the audio input (e.g., including SNR features), classifies the extracted features, and uses the model as part of generating a binaural object 266 based on the extracted features. The machine learning system can reduce or remove leakage from the classified objects by processing the extracted objects based on at least one of the SNR or audio-visual context to generate a binaural object 266. Additional details of this embodiment of the machine learning system are provided in International Patent Application No. PCT / CN2022 / 114613.

[0036] The object extractor 210 may be implemented by a capture device of the UGC content creator, such as a mobile phone (e.g., the mobile phone 104 of FIG. 1 ). For example, the mobile phone may use its microphone to capture the audio signal 262, which may be received from a user via, for example, Bluetooth. TM An earbud (eg, earbud 106 ) wirelessly connected to the mobile phone receives the binaural signal 264 and may locally generate a binaural object 266 .

[0037] Alternatively, the object extractor 210 can be implemented by a server device. Instead of generating the binaural object 266 locally, the capture device (e.g., mobile phone 104) transmits the audio signal 262 and the binaural signal 264 to the server that generates the binaural object 266. The UGC content creator can then receive the binaural object 266 from the server for local playback on the capture device or on another device. Additionally, other users can receive the binaural object 266 from the server for playback using their own playback devices (together with the captured video when the captured video is also transmitted to the server device).

[0038] Alternatively, the object extractor 210 may be implemented by a computer. Instead of generating the binaural object 266 locally or uploading the captured signal to a server, the UGC content creator may connect a mobile phone to a personal computer that generates the binaural object 266. The UGC content creator may then use a computer or other device for local playback to play back the binaural object 266. Additionally, the UGC content creator may upload the captured video and received binaural object 266 to a server for other users to play back using their own playback devices.

[0039] HRTF adjuster 220 receives binaural object 266 and head orientation information 270, adjusts binaural object 266 according to head orientation information 270, and generates adjusted binaural object 272. Head orientation information 270 may be generated by a playback device of the listener, such as a binaural head mounted device having a gyroscope for tracking movement of the head mounted device as the listener's head moves. Thus, adjusted binaural object 272 corresponds to binaural object 266 adjusted using HRTF based on head orientation information 270. Figure 3 Further details of the HRTF adjuster 220 are provided.

[0040] The rebalancer 230 receives the adjusted binaural object 272 and the head orientation information 270, rebalances the adjusted binaural object 272 according to the head orientation information 270, and generates a rebalanced binaural object 274. In general, the rebalancer 230 performs sound level adjustment and timbre adjustment based on the listener's head movement (as indicated by the head orientation information 270). Figure 4 Further details of the rebalancer 230 are provided.

[0041] The mixer 240 receives the residual signal 268 and the head orientation information 270, mixes the residual signal 268 according to the head orientation information 270, and generates a residual signal 276. The residual signal 276 has 2 channels, while the residual signal 268 has N+2 channels. Figure 5 Further details of mixer 240 are provided.

[0042] The remixer 250 receives the binaural object 274 and the residual signal 276, mixes the audio corresponding to the binaural object 274 with the residual signal 276, and generates a modified binaural signal 278. Typically, the remixer 250 renders the binaural object 274 as an intermediate binaural signal having two channels, to which it adds the residual signal 276, also having two channels, thereby producing a modified binaural signal 278 having two channels.

[0043] As discussed above, the functionality of system 200 can be implemented by multiple devices. As an example, the UGC content creator itself can use its capture device as a playback device. In this embodiment, the UGC content creator's mobile phone is used to perform object extraction, and the UGC content creator's earbuds are used to capture their head movements during playback. As another example, the UGC content creator can provide the captured video and processed audio to the listener, and the listener's device plays back the audio as modified by the listener's current head movement. In this embodiment, the UGC content creator's mobile phone is used to perform object extraction, and the listener's earbuds are used to capture the listener's current head movement. As another example, the server can perform object extraction to play back the audio as modified by its current head movement by the UGC content creator or another listener.

[0044] Figure 3 FIG. 2 is a diagram showing the HRTF adjuster 220 (see FIG. Figure 2 ) is a block diagram of additional details of the HRTF adjuster 220. The HRTF adjuster 220 includes a direction estimator 302, a ΔHRTF generator 304, a ΔHRTF calculator 306, and an object adjuster 308. In general, the HRTF adjuster 220 primarily adjusts for azimuth (left and right) variations of the listener's head orientation, but also adjusts for elevation (up and down) variations.

[0045] The direction estimator 302 receives the binaural object 266, estimates the direction of arrival (DOA) of the sound represented by the object, and generates a weighting vector 320. The weighting vector 320 can correspond to the time difference of arrival (TDOA) of the sound represented by the object. The direction estimator 302 can implement one or more techniques for estimating the DOA, including digital signal processor (DSP)-based techniques, machine learning (ML)-based techniques, etc. DSP-based techniques can operate on the sound level difference and time difference of the sound represented by the object. An example of a DSP-based technique is described in Kwon, Byoungho, Youngjin Park, and Youn-sik Park, “Analysis of the GCC-PHAT technique for multiple sources”, ICCAS 2010, pp. 2070-2073 (IEEE, 2010) <doi:10.1109 / ICCAS.2010.<5670137>. Another example of a DSP-based technique is described in Dmochowski, Jacek P., Jacob Benesty, and Sofiene Affes, “A generalized steered response power method for computationally viable source localization”, IEEE Transactions on Audio, Speech, and Language Processing 15, no. 8 (2007): 2510-2526 <doi:10.1109 / TASL.2007.906694>. An example of an ML-based technique is the adaptive boosting technique such as AdaBoost. Another example of an ML-based technique is a neural network having multiple layers between the input layer and the output layer, such as a deep neural network (DNN).

[0046] The ΔHRTF generator 304 receives the head orientation information 270, adjusts the HRTFs calculated for a plurality of predefined positions according to the head orientation information 270, and generates ΔHRTFs 322. The number of predefined positions can be four, corresponding to the front, back, left, and right. Then, the ΔHRTFs 320 correspond to the HRTFs of the predefined positions adjusted according to the head orientation information 270. Using predefined positions reduces the computational complexity of generating HRTFs based on the listener's head movement. The number of predefined positions can be adjusted as needed.

[0047] Equation (1) describes the operation of the ΔHRTF generator 304 .

[0048]

[0049] In equation (1), ω is the angular frequency because HRTF is frequency-dependent. i and are the azimuth and elevation angles of the i-th predefined position, respectively. Δθ and are respectively the azimuth change and the elevation change due to the head rotation according to the head rotation information 270. In other words, the ΔHRTF 322 of the given predefined position is proportional to a function of at least one of the azimuth and elevation of the given predefined position after the head rotation, and is inversely proportional to a function of at least one of the azimuth and elevation of the given predefined position.

[0050] ΔHRTF calculator 306 receives weighting vector 320 and ΔHRTF 322, applies weighting vector 320 to ΔHRTF 322 at a predefined location, and generates weighted ΔHRTF 324. The operation of ΔHRTF calculator 306 is described by equation (2).

[0051] dHRTF(ω)=∑ i W i dHRTF i (ω) (2)

[0053] In equation (2), W i is the weighting vector for the ith predefined position. In other words, the weighted ΔHRTF 324 is the sum of the weighting vectors 320 of the ΔHRTF 322 applied to each predefined position over a set of predefined positions.

[0054] The object adjuster 308 receives the binaural objects 266 and the weighted ΔHRTF 324, applies the weighted ΔHRTF to each object, and generates an adjusted binaural object 272. Thus, the adjusted binaural object 272 corresponds to the binaural object 266 rotated according to the head orientation information 270. The operation of the object adjuster 308 is described by equation (3).

[0055] Y obj_HA (ω) = X obj (ω)dHRTF(ω) (3)

[0057] In equation (3), X obj (ω) is the frequency domain representation of a given object obj, dHRTF(ω) is the weighted ΔHRTF 324, and Y obj_HA(ω) is the frequency domain representation of the object after HRTF adjustment according to the head orientation information 270. In other words, the adjusted binaural object 272 is proportional to the weighted ΔHRTF 324.

[0058] In summary, the HRTF adjuster 220 calculates the DOA for each object generated by the object extractor 110 (using the direction estimator 302 ), and then uses the object adjuster 308 to weight the HRTF for each object between two predefined locations.

[0059] The playback device (e.g., a listener's mobile phone) may implement all components of the HRTF adjuster 220. Alternatively, a capture device (e.g., a UGC content creator's mobile phone) may implement the direction estimator 302, with the playback device implementing the other components. In such an embodiment, the capture device may provide the weighting vector 320 and the binaural object 266 to the playback device, for example as metadata.

[0060] Figure 4 The rebalancer 230 (see Figure 2 ). Rebalancer 230 includes direction estimator 402, rebalancing factor calculator 404, level adjuster 406, and timbre adjuster 408. In general, rebalancer 230 adjusts for elevation (upward and downward) changes in the listener's head orientation (e.g., related to high objects (airplanes, birds chirping, etc.)).

[0061] Direction estimator 402 receives adjusted binaural object 272, estimates the direction of arrival (DOA) of the sound represented by the object, and generates weight vector 420. Direction estimator 402 may be a directivity vector of direction estimator 302 (see Figure 3 ) is the same component as direction estimator 402, in which case weighting vector 420 corresponds to weighting vector 320 calculated based on binaural object 266. Alternatively, direction estimator 402 is a different component than direction estimator 302, in which case weighting vector 420 is calculated based on adjusted binaural object 272. In either case, direction estimator 402 can perform direction estimation using techniques similar to those of direction estimator 302, such as DSP-based techniques and ML-based techniques.

[0062] Rebalancing factor calculator 404 receives weighting vector 420 and head orientation information 270 and generates steering factor 422. The operation of rebalancing factor calculator 404 is described in equation (4).

[0063]

[0064] In equation (4), g dis the direction factor for the given object as calculated by direction estimator 402 , and corresponds to weighting vector 420 . is the elevation change caused by the listener's head movement and corresponds to the head orientation information 270. σ(·) is an activation function; when the listener moves his head upward, the rebalancer 230 may use the activation function to rebalance the height objects, for example, to apply a gain. g Rebalance corresponds to steering factor 422. In other words, steering factor 422 is proportional to the weights in weight vector 420 and the activation function (associated with head orientation information 270).

[0065] The sound level adjuster 406 receives the adjusted binaural object 272 and the manipulation factor 422, adjusts the sound level of the adjusted binaural object 272 according to the manipulation factor 422, and generates a binaural object 424 with adjusted sound level. The sound level adjuster 406 can use the dynamic range control (DRC) applied to the object to implement the sound level adjustment. For example, if the height object is already loud (before adjustment), when the listener moves his head direction upward, no more additional gain needs to be applied. In this case, the manipulation factor 422 controls the intensity of the DRC. The sound level adjuster 406 can adjust the amount of the DRC based on psychoacoustic principles.

[0066] The timbre adjuster 408 receives the binaural object 424 adjusted by the sound level and the control factor 422, adjusts the timbre of the binaural object 424 adjusted by the sound level according to the control factor 422, and generates the rebalanced binaural object 274. The timbre adjuster 408 can implement the timbre adjustment using equalization applied to certain frequency bands. For example, when the listener moves his head upward, the timbre of the sound changes, and the timbre adjustment can improve certain frequency bands in this case. In other words, when the listener perceives a height object and moves his head upward, the timbre adjustment makes it perceived that the listener is looking directly at the height object, rather than perceiving that the height object is above the listener. In this case, the control factor 422 controls the intensity of the EQ. The frequency bands adjusted by the timbre adjuster 408 can be selected based on psychoacoustic principles.

[0067] The playback device (e.g., a listener's mobile phone) may implement all components of the rebalancer 230. Alternatively, a capture device (e.g., a UGC content creator's mobile phone) may implement the direction estimator 402, with the playback device implementing the other components. In such an embodiment, the capture device may provide the weighting vector 420 and the binaural object 266 to the playback device, for example as metadata.

[0068] Figure 5 The mixer 240 (see Figure 2) is a block diagram of additional details of FIG. 240. The mixer 240 includes a decorrelator 502, a mixing ratio calculator 504, and a mixer 506.

[0069] The decorrelator 502 receives the residual signal 268, performs decorrelation on the residual signal 268, and generates a decorrelated residual signal 520 resulting from the decorrelation. The residual signal 268 has N+2 channels, and the decorrelated residual signal 520 has M channels, where M≥N+2. When M=N+2, the decorrelation operation can be skipped. However, M is usually increased at the expense of increased processing time to provide a more special perception to the listener. The decorrelator can be implemented using a set of delay lines. Another example implementation of the decorrelator 502 is given in Kendall, Gary S., "The decorrelation of audio signals and its impact on spatial imagery", Computer Music Journal 19, No. 4 (1995): 71-87.

[0070] The mixing ratio calculator 504 receives the head orientation information 270 and generates a mixing matrix 522 based on the head orientation information 270. The mixing matrix 522 has a size of M×2 and corresponds to the azimuth change Δθ and the elevation change due to the head rotation as indicated by the head orientation information 270.

[0071] The mixer 506 receives the decorrelated residual signal 520 and the mixing matrix 522, performs mixing, and generates the residual signal 276. The mixer 506 may perform mixing as described by equation (5).

[0072]

[0073] In equation (5), Y res (ω) is the frequency domain representation of the residual signal 276, is the mixing matrix 522, and X res (ω) is the frequency domain representation of the decorrelated residual signal 520. In other words, the residual signal 276 is proportional to the decorrelated residual signal 520 (which is based on the residual signal 268) and to the mixing matrix 522 (which is based on the azimuth and elevation changes due to the head rotation).

[0074] Example device architecture

[0075] FIG6 is a device architecture 600 for implementing the features and processes described herein, according to an embodiment. The architecture 600 may be implemented in any electronic device including, but not limited to, a desktop computer, a consumer audio / video (AV) device, a radio broadcast device, a mobile device (e.g., a smartphone, a tablet computer, a notebook computer, a wearable device, etc.). In the example embodiment shown, the architecture 600 is used in a mobile phone. The architecture 600 includes (multiple) processors 601, a peripheral device interface 602, an audio subsystem 603, one or more speakers 604, one or more microphones 605, a sensor 606 (e.g., an accelerometer, a gyroscope, a barometer, a magnetometer, a camera, etc.), a position processor 607 (e.g., a GNSS receiver, etc.), a wireless communication subsystem 608 (e.g., Wi-Fi, Bluetooth, cellular, etc.), and (multiple) I / O subsystems 609, including a touch controller 610 and other input controllers 611, a touch surface 612 and other input / control devices 613. Other architectures with more or fewer components may also be used to implement the disclosed embodiments.

[0076] The memory interface 614 is coupled to the processor 601, the peripheral device interface 602, and the memory 615 (e.g., flash memory, RAM, ROM, etc.). The memory 615 stores computer program instructions and data, including but not limited to: operating system instructions 616, communication instructions 617, GUI instructions 618, sensor processing instructions 619, telephony instructions 620, electronic messaging instructions 621, web browsing instructions 622, audio processing instructions 623, GNSS / navigation instructions 624, and application / data 625. The audio processing instructions 623 include instructions for performing the audio processing described herein.

[0077] According to an embodiment, the architecture 600 may correspond to one or more playback devices, such as a mobile phone and earbuds. In such an embodiment, the device architecture 600 corresponds to a mobile phone, and the audio subsystem 603 communicates wirelessly with a speaker 604 implemented in the earbud. The sensor 606 generates head orientation information, for example, by tracking the movement of the earbud. The earbud itself may include components similar to those of the architecture 600 and output binaural signals 278. The processor(s) 601 implement various functions of the system 200, such as the HRTF adjuster 220, the rebalancer 230, the mixer 240, the remixer 250, etc.

[0078] Similarly, architecture 600 can correspond to one or more capture devices, such as a mobile phone and earbuds. In such an embodiment, device architecture 600 corresponds to a mobile phone, and audio subsystem 603 communicates wirelessly with microphone 605 implemented in the earbuds. The earbuds themselves may include components similar to those of architecture 600, and capture binaural signals 264. Processor(s) 601 implement various functions of system 200, such as object extractor 210.

[0079] Similarly, architecture 600 may correspond to a computer system implementing a cloud service. In such an embodiment, device architecture 600 corresponds to a computer system implementing object extractor 210. The computer system receives audio signals 262 and 264 from a capture device and transmits binaural objects 266 and residual signals 268 to a playback device.

[0080] Figure 7 6 , to implement the system 200 (see FIG. 6 ). Figure 2 ) functions, etc.

[0081] At 702, one or more playback devices receive user generated content (UGC) captured by capture devices connected to each other. Each audio source of the UGC corresponds to a corresponding feature in the audio scene. For example, the mobile phone 104 (see FIG. 1) and the earbuds 106 can be wirelessly connected, wherein the mobile phone 104 captures UGC video and UGC audio, and the earbuds 106 capture UGC binaural audio. The playback device (e.g., the mobile phone and the listener's earbuds) can receive the captured UGC.

[0082] At 704, the one or more playback devices receive information indicative of listener behavior of a user of the one or more playback devices from one or more sensors of the one or more playback devices. For example, the playback device may be implemented by architecture 600 (see FIG. 6 ), wherein sensor 606 includes a gyroscope that generates head orientation information corresponding to head movement of the listener.

[0083] At 706, the UGC is adapted based on the listener behavior, including compensating for characteristics of the audio source based on the listener behavior. For example, the playback device may implement system 200 (see Figure 2 ), the system adjusts the captured UGC audio (audio signal 262 and binaural audio signal 264) according to the head orientation information 270.

[0084] At 708, the adapted UGC (see 706) is rendered to provide the listener with an interactive experience with the audio scene. For example, the playback device may implement the remixer 250 (see Figure 2 ), the remixer renders the result of adapting the captured UGC audio and generates a modified binaural signal 278.

[0085] Method 700 may include additional steps corresponding to other functions of the audio processing system as described herein. One such function is object modification, such as using HRTF adjuster 220 or rebalancer 230 (see Figures 2 to 4 ). HRTF adjustment may include extracting one or more objects from a given audio portion of the UGC, such as described herein with respect to the object extractor 210 (see Figure 2 ) as described herein. HRTF adjustment may include calculating the HRTF difference before and after head rotation for a set of predefined positions, for example as described herein with respect to the ΔHRTF generator 304 (see Figure 3 ) as described herein. The HRTF adjustment may include obtaining the HRTF difference for a particular object by applying different weights to the HRTF differences of the set of predefined positions according to the corresponding direction of each of the one or more objects, such as described herein with respect to the direction estimator 302 and the ΔHRTF calculator 306 (see Figure 3 ) as described herein. The HRTF adjustment may include repositioning a specific object to a new position after the head rotation, including applying the obtained HRTF difference to the specific object, for example as described herein with respect to the object adjuster 308 (see Figure 3 ) described.

[0086] Rebalancing may include extracting one or more objects from a given audio portion of the UGC, such as described herein with respect to the object extractor 210 (see Figure 2 ) as described herein. Rebalancing may include determining a respective orientation of each of the one or more objects, such as described herein with respect to the direction estimator 402 and the rebalancing factor calculator 404 (see Figure 4 ) as described herein. Rebalancing may include rebalancing the object by applying at least one of a sound level adjustment and a timbre adjustment according to the head orientation information, such as described herein with respect to the sound level adjuster 406 and the timbre adjuster 408 (see Figure 4 ) described.

[0087] Another such function is residual blending, for example using mixer 240 (see Figure 2 and Figure 5 ). The residual mixing may include obtaining a residual by removing an object from a given audio portion of the UGC to generate a residual signal 268, for example as described herein with respect to the object extractor 210 (see Figure 2) as described herein. The residual mix may include creating one or more additional channels by decorrelation, for example as described herein with respect to decorrelator 502 (see Figure 5 ) as described herein. Residual mixing may include mixing the residuals of different audio channels from different capture devices by applying a mixing ratio to each channel according to the head orientation information, for example as described herein with respect to the mixing ratio calculator 504 and the mixer 506 (see Figure 5 ) described.

[0088] Implementation details

[0089] Embodiments may be implemented in hardware, executable modules stored on computer-readable media, or a combination of the two (e.g., programmable logic arrays, etc.). Unless otherwise specified, the steps performed by the embodiments need not be inherently related to any particular computer or other device, although they may be related in certain embodiments. Specifically, various general-purpose machines may be used with programs written according to the teachings herein, or more specialized devices (e.g., integrated circuits, etc.) may be more conveniently constructed to perform the required method steps. Therefore, embodiments may be implemented in one or more computer programs executed on one or more programmable computer systems, each of which includes at least one processor, at least one data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device or port, and at least one output device or port. Program code is applied to input data to perform the functions described herein and generate output information. Output information is applied to one or more output devices in a known manner.

[0090] Each such computer program is preferably stored on or downloaded to a storage medium or device (e.g., solid-state memory or medium, magnetic or optical medium, etc.) readable by a general or special purpose programmable computer for use in configuring and operating the computer system to execute the procedures described herein when the storage medium or device is read by the computer system. The system of the present invention may also be considered to be implemented as a computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer system to operate in a specific and predefined manner to perform the functions described herein. Software per se and intangible or transient signals are excluded in the sense that they are non-patentable subject matter.

[0091] Aspects of the systems described herein can be implemented in a suitable computer-based sound processing network environment to process digital or digitized audio files. Portions of the adaptive audio system may include one or more networks comprising any desired number of independent machines including one or more routers (not shown) for buffering and routing data transmitted between computers. Such networks may be constructed on a variety of different network protocols and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0092] One or more components, blocks, processes or other functional components may be implemented by a computer program executed by a processor-based computing device that controls the system. It should also be noted that the various functions disclosed herein may be described in terms of their behavior, register transfers, logic components and / or other features using any number of combinations of hardware, firmware and / or data and / or instructions embodied in various machine-readable or computer-readable media. Computer-readable media that may embody such formatted data and / or instructions include, but are not limited to, various forms of physical, non-transitory, non-volatile storage media, such as optical, magnetic or semiconductor storage media.

[0093] The above description illustrates various embodiments of the present disclosure and examples of how various aspects of the present disclosure may be implemented. The above examples and embodiments should not be considered to be the only embodiments, but are presented to illustrate the flexibility and advantages of the present disclosure as defined by the appended claims. Based on the above disclosure and the appended claims, other arrangements, embodiments, implementations, and equivalents will be apparent to those skilled in the art and may be adopted without departing from the spirit and scope of the present disclosure as defined by the claims.

Claims

1. A computer-implemented audio processing method, the method comprising: Receiving, by one or more playback devices, user generated content (UGC) captured by capture devices connected to each other, each audio source of the UGC corresponding to a corresponding feature in an audio scene; receiving, by the one or more playback devices from one or more sensors of the one or more playback devices, information indicative of listener behavior of a user of the one or more playback devices; adapting the UGC based on the listener behavior, including compensating for characteristics of an audio source based on the listener behavior; as well as The adapted UGC is rendered to provide the listener with an interactive experience with respect to the audio scene.

2. The method of claim 1, wherein: The UGC includes a video stream and an immersive audio stream.

3. The method according to any one of claims 1 to 2, wherein: The one or more playback devices include a device for video playback and a connection device for audio playback.

4. The method according to any one of claims 1 to 3, wherein: The capture device includes a first device that captures at least one channel of a video and audio stream, and a connection device that captures a binaural audio stream.

5. The method according to any one of claims 1 to 4, wherein: The listener behavior includes at least head orientation relative to a screen of the one or more playback devices configured for video playback.

6. The method according to any one of claims 1 to 5, wherein: Adapting the UGC includes at least one of object modification and residual blending.

7. The method of claim 6, wherein: The subject modification includes at least one of a head related transfer function (HRTF) adjustment and a subject rebalancing.

8. The method of claim 7, wherein: The HRTF adjustment includes the following actions: extracting one or more objects from a given audio portion of the UGC; Calculating the HRTF difference before and after head rotation for a set of predefined positions; obtaining an HRTF difference for a specific object by applying different weights to the HRTF differences of the set of predefined positions according to a corresponding direction of each of the one or more objects; as well as Repositioning the specific object to a new position after the head rotation includes applying the obtained HRTF difference to the specific object.

9. The method of claim 7, wherein: The object rebalancing includes actions including: extracting one or more objects from a given audio portion of the UGC; determining a respective orientation of each of the one or more objects; The one or more objects are rebalanced by applying at least one of a sound level adjustment and a timbre adjustment based on the head orientation information.

10. The method of claim 6, wherein: The residual mixing includes actions including: Obtaining a residual by removing an object from a given audio portion of the UGC; creating one or more additional channels by decorrelation; as well as The residuals of different audio channels from different capture devices are mixed by applying a mixing ratio to each channel according to the head orientation information.

11. The method according to any one of claims 1 to 5, wherein: Adapting the UGC includes: receiving one or more objects and a residual signal, wherein the one or more objects and the residual signal have been generated based on a given audio portion of the UGC, wherein the given audio portion of the UGC comprises a first audio signal having at least one channel and a second audio signal being a binaural audio signal; performing object modification on the one or more objects based on the head orientation information, wherein performing the object modification includes generating one or more modified objects; Performing residual mixing on the residual signal based on the head orientation information, wherein performing the residual mixing comprises generating a mixed residual signal; and Remixing the one or more modified objects and the mixed residual signal includes generating a modified binaural signal based on the one or more modified objects and the mixed residual signal.

12. The method of claim 11, further comprising: The one or more objects and the residual signal are extracted from a given audio portion of the UGC.

13. The method of claim 11, wherein: Performing the object modification includes: performing direction estimation to calculate a direction of arrival for each of the one or more objects; performing HRTF adjustment on each of the one or more objects to adjust the given object for at least one of azimuth changes and elevation changes in the head orientation information based on the corresponding direction of arrival; and Object rebalancing is performed on each of the one or more objects to adjust the given object for elevation changes in the head orientation information based on the corresponding direction of arrival.

14. The method of claim 13, wherein: The HRTF adjustment is based on a ratio that is proportional to a function of at least one of the azimuth angle of the predefined position after the azimuth angle in the head orientation information changes and the elevation angle of the predefined position after the elevation angle in the head orientation information changes, and is inversely proportional to a function of at least one of the azimuth angle and the elevation angle of the predefined position.

15. The method of claim 13, wherein: The object rebalancing is proportional to a weighting vector and to an activation function, wherein the weighting vector is based on the orientation of the given object and wherein the activation function is based on the elevation change in the head orientation information.

16. The method of claim 11, wherein: Performing the residual blending includes: performing decorrelation on the residual signal, wherein performing the decorrelation comprises generating a decorrelated residual signal; generating a mixing matrix based on azimuth changes and elevation changes in the head orientation information; and Mixing the decorrelated residual signal with the mixing matrix in the frequency domain includes generating the mixed residual signal.

17. The method of claim 16, wherein: The mixed residual signal is proportional to the mixing matrix and to the decorrelated residual signal. 18 . A non-transitory computer-readable medium storing a computer program, which, when executed by a processor, controls a device to perform a process including the method according to claim 1 .

19. An apparatus for audio processing, the apparatus comprising: A processor, wherein the processor is configured to control the device to perform a process including the method according to any one of claims 1 to 17.

20. The device of claim 19, wherein: The one or more playback devices include: a mobile phone comprising the processor; and A set of earplugs for both ears.

Images