Context aware audio capture and rendering
Through situation-aware audio capture and rendering technology, the audio artifact problem in UGC processing is solved, and effective sound processing in different audio environments is achieved, ensuring the creative goals and audio quality of content creators.
Patent Information
- Application Number
- CN202380029774.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-29
- Filing Date
- 2023-03-17
- Publication Date
- 2024-11-08
AI Technical Summary
Existing UGC processing technology is difficult to effectively deal with different sound types in different audio environments while maintaining the creative goals of content creators, resulting in audio artifact problems.
Using situation-aware audio capture and rendering technology, by capturing multi-channel input audio signals, generate noise-reduced target sound events and environmental noise of interest, determine the event type, and select a rendering scheme based on the event type and speaker layout to render the audio signal. deal with.
It achieves the goal of capturing realistic binaural soundscapes in different audio environments, reducing audio artifacts, and maintaining the creative goals of content creators, providing appropriate audio processing configurations to suit different situations.
Smart Images

Figure CN118923135A_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to International Patent Application No. PCT / CN2022 / 083675 filed on March 29, 2022 and U.S. Provisional Application No. 63 / 336,424 filed on April 29, 2022, all of which are incorporated herein by reference in their entirety. Technical Field
[0003] The present disclosure relates generally to audio signal processing, and more particularly to the creation and playback of user generated content (UGC). Background Art
[0004] UGC is usually created by consumers and can contain any form of content (e.g., images, videos, text, audio). UGC is usually published by its creators to online platforms, which include but are not limited to social media, blogs, Wiki™, etc. A trend related to UGC is to share personal moments in various environments (e.g., indoors, outdoors, at the seaside) by recording video and audio using personal mobile devices (e.g., smartphones, tablets, wearable devices). Due to consumer hardware limitations and non-professional recording environments, most UGC content contains audio artifacts. Traditional UGC processing methods are based on audio signal analysis or artificial intelligence (AI)-based noise reduction and enhancement processing. One of the difficulties in UGC processing is how to handle different sound types in different audio environments while maintaining the creative goals of the content creator. Summary of the invention
[0005] Embodiments for context-aware audio capture and rendering are disclosed. In one embodiment, an audio processing method includes: capturing a multi-channel input audio signal; for each channel of the multi-channel input audio signal, generating a noise-reduced target sound event of interest and ambient noise; determining an event type for rendering; selecting a rendering scheme based on the event type and a speaker layout; and rendering a multi-channel output audio signal using the selected rendering scheme.
[0006] In some embodiments, an event type is determined for each channel of the multi-channel input audio signal based on the contextual information and the target sound event.
[0007] In some embodiments, the context information is generated by context analysis of at least one of input audio, input video, or sensor input.
[0008] In some embodiments, a machine learning model is used to determine the context information as an indoor context or an outdoor context based on the context information and the target sound event.
[0009] In some embodiments, for each sound event, the sound event type indicates one of a center rendering event, a surround rendering event, or a height rendering event, wherein, for a center rendering event, the rendering is distributed across a speaker layout to create a solid center position in the sound field for the target sound event, and for a surround rendering event, the rendering is distributed across the speaker layout to provide a wide sound field, and for a height rendering event, the rendering is distributed across the speaker layout to highlight an enhanced height effect.
[0010] In some embodiments, the speaker layout includes three speakers, including a left speaker, a right speaker, and a top speaker, and wherein the rendering is distributed over the left speaker and the right speaker to provide a wide sound field, and distributed to the top speaker to highlight an enhanced height effect.
[0011] In some embodiments, the speaker layout includes four speakers, including upper left and upper right speakers and lower left and lower right speakers, wherein for a center rendering event, the rendering is distributed across all four speakers, for a surround rendering event, the rendering is distributed across the lower left and lower right speakers to provide a wide sound field, and for a height rendering event, the rendering is distributed across the upper left and upper right speakers to highlight an enhanced height effect.
[0012] In some embodiments, the event type is determined during capture of the multi-channel input audio signal, and the event type is stored as metadata for rendering scheme selection in subsequent rendering.
[0013] In some embodiments, the format of the metadata depends on whether the capturing and rendering of the multi-channel input audio signal are performed by the same device.
[0014] In some embodiments, the method further comprises applying at least one of equalization or dynamic range control to the rendered multi-channel output audio signal.
[0015] In some embodiments, rendering the multi-channel output audio signal includes applying a mixing ratio to the target sound event and the ambient noise based on the event type.
[0016] In some embodiments, a multi-channel output audio signal is rendered by a mobile device including a folding screen, and the method further includes: determining, by at least one processor, whether the screen is folded or unfolded; and based on the determination, selecting a first speaker layout for rendering if the screen is folded, and selecting a second speaker layout if the screen is unfolded, wherein the first speaker layout is different from the second speaker layout.
[0017] In some embodiments, an audio processing system includes: one or more processors; and a non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform any of the foregoing methods.
[0018] In some embodiments, a non-transitory computer-readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to perform any of the foregoing methods.
[0019] Certain embodiments disclosed herein provide one or more of the following advantages: The disclosed context-aware audio capture and rendering embodiments can be used for binaural recording to capture realistic binaural soundscapes while maintaining the creative goals of the content creator. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In the accompanying drawings, for ease of description, a specific arrangement or ordering of schematic elements (e.g., elements representing devices, units, instruction blocks, and data elements) is shown. However, it should be understood by those skilled in the art that the specific ordering or arrangement of the schematic elements in the drawings does not imply the need for a specific processing order or sequence, or separation of processes. In addition, the inclusion of a schematic element in a drawing does not mean that the element is required in all embodiments, nor does it mean that the features represented by the element may not be included in other elements or combined with other elements in some embodiments.
[0021] In addition, in the accompanying drawings, connecting elements (such as solid lines, dotted lines or arrows) are used to illustrate the connection, relationship or association between two or more schematic elements, and the absence of any such connecting element does not mean that there is no connection, relationship or association. In other words, some connections, relationships or associations between elements are not shown in the figures to avoid blurring the present disclosure. In addition, for ease of explanation, a single connecting element is used to represent multiple connections, relationships or associations between elements. For example, when a connecting element represents the communication of a signal, data or instruction, it should be understood by those skilled in the art that such an element represents one or more signal paths required for realizing the communication.
[0022] Figure 1 Binaural recording using earbuds and a mobile device is shown according to one embodiment.
[0023] Figure 2A Audio capture is shown when a user holds a mobile device in a forward-facing position according to one embodiment.
[0024] Figure 2B Audio capture is shown when a user holds a mobile device in a rear-facing or "selfie" position, according to one embodiment.
[0025] Figure 3A A three-speaker layout of a smartphone with a foldable screen according to one embodiment is shown.
[0026] Figure 3B shows that when the screen is unfolded according to one embodiment Figure 3A The quad-speaker layout of a smartphone.
[0027] Figure 4 A speaker layout is shown according to one embodiment, where the speakers fire both upwards and downwards.
[0028] Figure 5 A speaker layout is shown according to one embodiment, wherein the speakers are sideways emitting sound.
[0029] Figure 6 is a block diagram of a noise reduction unit that generates denoised target sound events of interest and ambient noise according to one embodiment.
[0030] Figure 7 is a block diagram of context-aware event type classification according to one embodiment.
[0031] Figure 8 Rendering across multiple speakers based on event type and speaker layout is shown according to one embodiment.
[0032] Fig. 9 is a flow chart of a context-aware audio capture and rendering process according to one embodiment.
[0033] Fig.10 is a block diagram of an example device architecture for implementing the features and processes described with reference to FIGS. 1-9 , according to one embodiment.
[0034] The same reference numbers used in the various drawings denote the same elements. DETAILED DESCRIPTION
[0035] In the detailed description below, many specific details are set forth to provide a thorough understanding of the various embodiments described. It will be apparent to one of ordinary skill in the art that the various embodiments described may be practiced without these specific details. In other cases, well-known methods, procedures, components, and circuits are not described in detail to avoid unnecessarily masking aspects of the embodiments. Several features are described below, each of which may be used independently or in any combination with other features.
[0036] The disclosed context-aware audio capture and rendering includes the following steps. First, a binaural capture device (e.g., a pair of earbuds) records a multi-channel input audio signal (e.g., binaural left (L) and right (R)), and a playback device (e.g., a smartphone, tablet, or other device) renders the multi-channel audio recording through multiple speakers. The recording device and the playback device can be the same device, two connected devices, or two separate devices. The number of speakers used for multi-speaker rendering is at least three. In some embodiments, the number of speakers is three. In other embodiments, the number of speakers is four.
[0037] The capture device includes a context detection unit for detecting the context of the audio capture, and guiding audio processing and rendering based on the detected context. In some embodiments, the context detection unit includes a machine learning model (e.g., an audio classifier) that classifies the captured environment into several event types. For each event type, a different audio processing profile is applied to create an appropriate rendering through multiple speakers. In some embodiments, the context detection unit is a scene classifier based on visual information, which classifies the environment into several event types. For each event type, a different audio processing profile is applied to create an appropriate rendering through multiple speakers. The context detection unit can also be based on a combination of visual information, audio information, and sensor information.
[0038] In some embodiments, the capture device or playback device includes at least a noise reduction system that generates a noise-reduced target sound event of interest and residual ambient noise. The target sound event of interest is further classified into different event types by an audio classifier. Some examples of target sound events include, but are not limited to, speech, noise, or other sound events. According to the situation detection unit, the source type is also different in different capture situations.
[0039] In some embodiments, the playback device renders the target sound event of interest on multiple speakers by applying different mixing ratios of sound sources and ambient noise, and applying different equalization (EQ) and dynamic range control (DRC) according to the classified event type.
[0040] Nomenclature
[0041] As used herein, the term "including" and its variations should be understood as open-ended terms, meaning "including, but not limited to". Unless the context clearly indicates otherwise, "or" should be understood as "and / or". The term "based on" should be understood as "based at least in part on". The terms "an example embodiment" and "example embodiment" should be understood as "at least one example embodiment". The term "another embodiment" should be understood as "at least one other embodiment". The term "determine" or "determined" should be understood as obtaining, receiving, calculating, estimating, estimating, predicting, or deducing. In addition, in the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs.
[0042] Example System
[0043] Figure 1 Binaural recording using earbuds 102 and mobile device 101 according to one embodiment is shown. System 100 includes a two-step process: recording video using a camera of mobile device 101 (e.g., a smartphone), and concurrently recording audio associated with the video recording. In one embodiment, audio recording can be performed by, for example, mobile device 101 recording an audio signal output by a microphone embedded in earbuds 102. The audio signal can include, but is not limited to, comments spoken by the user and / or ambient sounds, and binaural recording can be captured if both the left and right microphones are used. In some embodiments, a microphone embedded in or attached to mobile device 101 can also be used.
[0044] FIG. 2A illustrates capturing audio when a user holds the mobile device 101 in a forward-facing position and uses a rear-facing camera, according to one embodiment. In this example, the camera capture area 200a is located in front of the user. The user wears earbuds 102a, 102b, each of which includes microphones for capturing left / right (binaural) sounds, respectively, which are combined into a binaural recording stream. Microphones 103a-103c embedded in the mobile device 101 capture left, front, and right sounds, respectively, and generate audio recording streams that are synchronized with the binaural recording streams and rendered on speakers embedded in or coupled to the mobile device 101.
[0045] FIG. 2B illustrates capturing audio when a user holds the mobile device in a front-facing position (“selfie” mode) and uses a front-facing camera, according to one embodiment. In this example, the camera capture area 200b is located behind the user. The user wears earbuds 102a, 102b, each of which includes a microphone for capturing left / right (binaural) sound, respectively. Microphones 103a-103c embedded in the mobile device 101 capture left, front, and right sound, respectively, and generate an audio recording stream that is synchronized with the binaural recording stream and rendered on a speaker coupled to the mobile device 101.
[0046] Figure 3A A three-speaker layout of a mobile device 101 with a folding screen when the screen is in a folding mode according to one embodiment is shown. The mobile device 101 is shown in a folding mode with an upper speaker 300 and a bottom speaker 301.
[0047] Figure 3B shows that when the screen is unfolded according to one embodiment Figure 3A The mobile device 101 is shown in an unfolded mode, with the left speaker 303 located at the lower left, the right speaker 304 located at the lower right, and the center speaker 302 located at the top.
[0048] As shown in the example mobile device 101 shown in Figures 3A and 3B, there are many different types of speaker layouts and arrangements on the mobile device 101 (e.g., smartphone, tablet computer). For example, a smartphone may have two symmetrical speakers to represent the left and right sound fields in landscape mode, or may have three speakers, two of which represent the left and right sound fields, and one upward-firing or front-firing speaker to represent the middle and upper parts of the sound field.
[0049] Figure 4 Speaker layout showing both upward and downward firing speakers, Figure 5 Shown is a speaker layout where the speakers fire sideways.
[0050] As another example, a tablet may have four speakers to represent the left and right sound fields (e.g., two speakers for the left channel and two speakers for the right channel). A tablet may also have four independent speakers with a symmetrical layout to represent the left, right, left height, and right height portions of the sound field. Audio rendering is used to adaptively distribute audio signals to these different speaker layouts and arrangements with appropriate gain.
[0051] Figure 66 is a block diagram of a system 600 for context-aware audio processing according to one embodiment. The system 600 includes a window processor 602, a spectrum analyzer 603, a frequency band feature analyzer 604, a gain estimator 605, a machine learning model 606, a context analyzer 607, a gain analyzer / adjuster 609, a frequency band gain to interval gain converter 610, a spectrum modifier 611, a speech reconstructor 612, and a window overlap-add processor 613.
[0052] The window processor 602 generates a speech frame that includes overlapping windows of samples of input audio 601 (e.g., an audio recording captured by the mobile device 101) containing speech. The speech frame is input to the spectrum analyzer 603, which generates frequency bin features and a fundamental frequency (F0). The analyzed spectral information can be represented by a fast Fourier transform (FFT) spectrum, a quadrature mirror filter (QMF) feature, or any other audio analysis process. These bins are scaled by the spectrum modifier 611 and input to the speech reconstructor 612, which outputs the reconstructed speech frame. The reconstructed speech frame is input to the window overlap-add processor 613, which produces the output speech.
[0053] Referring back to step 603, the interval features and F0 are input to a band feature analyzer 604, which outputs the band features and F0. In one embodiment, the band features are extracted based on FFT parameters. The band features may include, but are not limited to, Mel frequency cepstral coefficients (MFCC) and Bark frequency cepstral coefficients (BFCC). In one embodiment, a band harmonicity feature may be calculated, which indicates how many periodic signals the current frequency band consists of. In one embodiment, the harmonicity feature may be calculated based on the FFT frequency interval of the current speech frame by correlating the current speech frame with the previous speech frame.
[0054] The frequency band characteristics and F0 are input to the gain estimator 605, which estimates the gain for noise reduction based on the model selected from the model pool 606 ( CGains). In one embodiment, the model is selected based on the model number or other data output by the context analyzer 607 in response to the input visual information and / or other sensor information. In one embodiment, the model is a deep neural network (DNN) that is trained to estimate the gain and voice activity detection (VAD) of each frequency band based on frequency band features and F0. The DNN model can be based on a fully connected neural network (FCNN), a recurrent neural network (RNN) or a convolutional neural network (CNN) or any combination of FCNN, RNN and CNN. In one embodiment, a Wiener filter or other suitable estimator can be combined with the DNN model to obtain the final estimated noise reduction gain.
[0055] The estimated gain CGains is input to the gain analyzer / adjuster 609, which generates an adjusted gain based on the audio processing profile AGains . AGains The adjusted interval gain is input to the frequency band gain to interval gain converter 610, which produces the adjusted interval gain. The adjusted interval gain is input to the spectrum modifier 611, which applies the adjusted interval gain to its corresponding frequency interval (e.g., the interval amplitude is scaled by their respective adjusted interval gain). The adjusted interval feature is then input to the speech reconstructor 612, which outputs the reconstructed speech frame. The reconstructed speech frame is input to the window overlap-add processor 612, which generates the reconstructed output speech using the overlap-add algorithm.
[0056] In some embodiments, the context analyzer 607 outputs a model number or other data for identifying a model in the model pool based on the input audio 601 and / or the input visual information and / or other sensor data 608. The context analyzer 607 may include one or more audio scene classifiers that are trained to classify audio content into one or more categories representing recording locations. In some embodiments, the recording location categories include indoor, outdoor, and transportation. For each category, a specific audio processing profile may be assigned. In some embodiments, the context analyzer 607 is trained to classify more specific recording locations (e.g., bays, forests, concerts, conference rooms, etc.).
[0057] In some embodiments, the context analyzer 607 is trained using visual information (e.g., digital pictures and video recordings, or a combination of audio recordings and visual information). In other embodiments, other sensor data may also be used to determine context, such as inertial sensors (e.g., accelerometers, gyroscopes) or positioning technologies, such as global navigation satellite systems (GNSS), cellular networks, or Wi-Fi fingerprints, alone or in combination with audio and visual information. For example, accelerometers and gyroscopes and / or global positioning system (GPS) data may be used to determine the speed of the mobile device 101. The speed may be combined with the audio recording and / or visual information to determine whether the mobile device 101 is being transported (e.g., in a vehicle, bus, airplane, etc.).
[0058] In one embodiment, different models can be trained for different scenarios to achieve better performance. The training data can be adjusted to achieve different model behaviors. When training a model, the training data can be divided into two parts: (1) a target audio database containing the signal portion of the input audio to be retained in the output speech; (2) a noise audio database containing the noise portion of the input audio that needs to be suppressed in the output speech. Different training data can be defined to train different models for different recording locations. For example, for a bay model, the sound of the tide can be added to the target audio database to ensure that the model retains the sound of the tide.
[0059] After defining a specific training database, a conventional training procedure can be used to train the model (e.g., back propagation). In one embodiment, context information can be mapped to a specific audio processing profile. The specific audio processing profile may include at least one specific mixing ratio for mixing input audio (e.g., an original audio recording) with a processed audio recording in which ambient noise is suppressed. The mixing ratio is controlled by a context analyzer 607. The mixing ratio can be applied in the time domain, or the CGains can be adjusted using the mixing ratio according to formula [1] by a gain adjuster 609, as described below.
[0060] Although the DNN-based noise reduction algorithm can significantly suppress noise, the noise reduction algorithm may also introduce artifacts in the output speech or delete some target sound events of interest. Therefore, in order to reduce artifacts and restore target sound events of interest, the processed audio recording is mixed with the original audio recording. In one embodiment, a fixed mixing ratio can be used. For example, the mixing ratio can be 0.25.
[0061] However, a fixed mixing ratio may not be suitable for different situations. Therefore, in one embodiment, the mixing ratio can be adjusted based on the recording situation output by the situation analyzer 607. To achieve this, it is necessary to estimate the situation based on the input audio information. For example, for indoor categories, a larger mixing ratio (e.g., 0.35) can be used. For outdoor situations, a lower mixing ratio (e.g., 0.25) can be used. For transportation classes, an even lower mixing ratio (e.g., 0.2) can be used. In embodiments in which a more specific recording location can be determined, different audio processing profiles can be used. For example, for a conference room, a small mixing ratio (e.g., 0.1) can be used to eliminate more noise. For concerts, a large mixing ratio (e.g., 0.5) can be used to avoid degradation of music quality.
[0062] In one embodiment, mixing the original audio recording with the processed audio recording can be achieved by mixing the denoised audio file with the original audio file in the time domain. In another embodiment, the mixing ratio can be calculated according to formula [1]. dMixRatio Adjustment CGains To achieve mixing:
[0063] AGains = CGains + dMixRatio , [1]
[0064] Among them, if CGains >1, then r = 1 .
[0065] In some embodiments, a particular audio processing profile also includes EQ curves and / or DRC data, which can be applied to post-processing steps, as described below with reference to FIG8. Sound event types and contextual information can be stored as metadata and shared between capture devices and playback devices. For example, if the recording location is determined to be a concert, a music-specific EQ curve can be applied to the output of system 600 to preserve the timbre of various instruments, and / or the DRC can be configured to perform less compression to ensure that the music level is within a certain loudness range suitable for music. In audio scenarios dominated by speech, the EQ curve can be configured to enhance speech quality and intelligibility (e.g., enhance at 1 KHz), and the DRC can be configured to perform more compression to ensure that the speech level is within a certain loudness range suitable for speech.
[0066] Figure 7 7 is a block diagram of a context-aware event classification module 700 according to one embodiment. The event classification unit 700 includes a context analysis unit 701, a noise reduction unit 702, and an event classifier 703. The noise reduction unit 702 can utilize the Figure 6The noise reduction system 600 is implemented as described above. The noise reduction unit 702 generates a noise-reduced target sound event of interest. The audio event classifier 703 further classifies the target sound event of interest into different event types to select an appropriate rendering scheme based on the context information (e.g., sound source type) determined and output by the context analysis unit 710. More specifically, the context analysis unit 701 obtains input audio, video, and sensor data and generates currently captured context information. The input audio is first fed to the noise reduction unit 702, which generates a noise-reduced target sound event.
[0067] The event classification unit 703 obtains the target sound event of noise reduction, and determines the event type for rendering based on the situation information output by the situation analysis unit 701. In some embodiments, the situation information can be indoor / outdoor classification, where different classifier models are used for different sound events. The event type can be used to select a rendering scheme from multiple rendering streams on multiple speakers. In some embodiments, the event type can be "center", "surround" and "height".
[0068] In some embodiments, the event type can be determined during the capture of audio. In some embodiments, the event type is transmitted in the metadata stream along with the audio data or separately. The metadata stream format can be different when the capture device and the playback device are the same device and when they are different devices.
[0069] Figure 8 Context-aware rendering on multiple speakers based on event type according to one embodiment is shown. To render content on multiple speakers, a multi-channel input audio signal is first processed by a context-aware noise reduction unit 801 (e.g., using the context-aware noise reduction system 600 shown in FIG. 6 ) to generate target sound events of interest and ambient noise (e.g., residual ambient noise) for channel L and channel R.
[0070] In some embodiments, the context-aware noise reduction unit 801 Figure 7 The context information output by the context analysis unit 701 is shown as input, which can be stored as metadata to be shared between the capture device and the playback device. To avoid repeated calculations, in some embodiments, the frequency band gain calculated in the noise reduction unit 702 (output from the gain estimator 605) can also be stored as metadata and applied directly by the context-aware noise reduction unit 801 through the gain adjuster 609.
[0071] To generate output for each speaker, the target sound event L, the ambient noise L, the target sound event R, and the ambient noise R are processed by corresponding post-processing and mixing modules 802a, ..., 802n. The post-processing mixing modules 802a, ..., 802n apply at least EQ and DRC to the inputs, and achieve mixing by applying a mixing ratio to each input. The post-processing and mixing ratio of each output channel is based on the event type, so different sound events apply different rendering schemes.
[0072] In this example, speaker layout and placement include: Figure 3B (Expand the screen) with three speakers shown, where the left and right speakers are located at the lower left and lower right, respectively, and the center speaker is located at the top. In some embodiments, the sound event types can be "center rendering event", "surround rendering event", and "height rendering event". For a center rendering event, the rendering is distributed on the left, right, and center speakers to create a stable sound source in the center channel of the sound field. For a surround rendering event, the rendering is emphasized on the left and right speakers to provide a wide sound field. For a height rendering event, the rendering on the center speaker is emphasized to enhance the height effect.
[0073] In another example, the speaker layout and arrangement of four speakers are shown in Figures 4 and 5. In some embodiments, the sound event types can be "center rendering event", "surround rendering event" and "height rendering event". For the center rendering event, the rendering is distributed on all four speakers to create a stable sound source in the center channel. For the surround rendering event, the rendering is prominent on the lower left and lower right speakers to provide a wide sound field. For the height rendering event, the rendering is prominent on the upper left and upper right speakers to enhance the height effect.
[0074] Example Process
[0075] Fig. 9 is a flow chart of a process 900 for context-aware capture and rendering according to one embodiment. Fig.10 The described device architecture 1000 implements the process 900 .
[0076] Process 900 includes the following steps: capturing a multi-channel input audio signal (901), generating a noise-reduced target sound event of interest and ambient noise for each channel of the multi-channel input audio signal (902); determining an event type for rendering (903); selecting a rendering scheme based on the event type and the speaker layout (904); and rendering a multi-channel output audio signal using the selected rendering scheme (905). Each step has been described in detail above with reference to FIGS. 1-8.
[0077] Exemplary System Architecture
[0078] FIG. 1000 shows a diagram suitable for implementing reference Figure 1-9 A block diagram of an example system 1000 of the described example embodiment. The system 1000 includes a central processing unit (CPU) 1001, which is capable of executing various processes according to a program stored in, for example, a read-only memory (ROM) 1002 or a program loaded from, for example, a storage unit 1008 to a random access memory (RAM) 1003. In the RAM 1003, data required when the CPU 1001 executes various processes is also stored as needed. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.
[0079] The following components are connected to the I / O interface 1005: an input unit 1006, which may include a keyboard, a mouse, etc.; an output unit 1007, which may include a display, such as a liquid crystal display (LCD) and one or more speakers; a storage unit 1008, including a hard disk or other suitable storage device; and a communication unit 1009, including a network interface card, such as a network card (e.g., wired or wireless).
[0080] In some embodiments, input unit 1006 includes one or more microphones located at different locations (depending on the host device) so that audio signals in various formats (eg, mono, stereo, spatial, immersive, and other suitable formats) can be captured.
[0081] In some embodiments, the output unit 1007 includes a system with various numbers of speakers. The output unit 1007 can render audio signals in various formats (eg, mono, stereo, immersive, binaural, and other suitable formats).
[0082] The communication unit 1009 is used to communicate with other devices (e.g., via a network). As required, a drive 1010 is also connected to the I / O interface 1005. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a flash drive, or other suitable removable media, is installed on the drive 1010 so that a computer program read therefrom can be installed into the storage unit 1008 as required. It will be appreciated by those skilled in the art that although the system 1000 is described as including the above components, in actual applications, some of the components may be added, removed, and / or replaced, and all such modifications or changes fall within the scope of the present disclosure.
[0083] According to an exemplary embodiment of the present disclosure, the above process may be implemented as a computer software program or on a computer-readable storage medium. For example, an embodiment of the present disclosure includes a computer program product that includes a computer program tangibly embodied on a machine-readable medium, the computer program including program code for executing the method. In such an embodiment, the computer program may be downloaded and installed from a network via the communication unit 709, and / or installed from the removable medium 1011, such as Fig.10 shown.
[0084] In general, various exemplary embodiments of the present disclosure may be implemented in hardware or dedicated circuits (e.g., control circuits), software, logic, or any combination thereof. For example, the above-mentioned units may be implemented by a control circuit (e.g., Fig.10 The CPU (in combination with other components of the computer) is executed, so the control circuit can perform the actions described in the present disclosure. Some aspects can be implemented in hardware, while other aspects can be implemented in firmware or software, which can be performed by a controller, microprocessor or other computing device (e.g., control circuit). Although various aspects of the example embodiments of the present disclosure are shown and described in block diagrams, flow charts, or using some other graphical representation, it should be understood that, as non-limiting examples, the boxes, devices, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.
[0085] In addition, the various blocks shown in the flowchart may be viewed as method steps, and / or operations resulting from the operation of computer program code, and / or multiple coupled logic circuit elements configured to perform related functions. For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine-readable medium, the computer program including program code configured to perform the method described above.
[0086] In the context of the present disclosure, a machine-readable medium may be any tangible medium that may contain or store a program used by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may be non-transitory and may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0087] The computer program code for executing the disclosed method can be written in any combination of one or more programming languages. These computer program codes can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device with a control circuit, so that when executed by the processor of the computer or other programmable data processing device, the program code enables the function / operation specified in the flow chart and / or block diagram to be realized. The program code can be executed completely on a computer, partially on a computer, executed as an independent software package, partially on a computer and partially on a remote computer, or completely on a remote computer or server, or distributed on one or more remote computers and / or servers.
[0088] While this document contains many details of particular embodiments, these should not be construed as limitations on the scope of the claims but rather as descriptions of features specific to particular embodiments.
[0089] Certain features described in the context of separate embodiments in this specification may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although the features may be described as acting in a certain combination above, and even initially claimed as such, in some cases, one or more features from the claimed combination may be removed from the combination, and the claimed combination may be for a variant of a sub-combination or a sub-combination. The logical flow depicted in the figure does not require the specific order or continuous order shown to achieve the desired result. In addition, other steps may be provided, or some steps may be deleted from the process, and other components may be added to or deleted from the system. Therefore, other embodiments are also within the scope of the following claims.
Claims
1. An audio processing method, comprising: capturing a multi-channel input audio signal via at least one processor; generating, by at least one processor, for each channel of a multi-channel input audio signal, a noise-reduced target sound event of interest and ambient noise; determining, by at least one processor, an event type for rendering; selecting, by at least one processor, a rendering scheme based on an event type and a speaker layout; as well as The multi-channel output audio signal is rendered by at least one processor using the selected rendering scheme.
2. The method according to claim 1, wherein: An event type is determined for each target sound event of the multi-channel input audio signal through event classification.
3. The method according to claim 2, wherein: Event categorization is governed by contextual information.
4. The method according to claim 3, wherein: The context information is generated by context analysis of at least one of input audio, input video, or sensor input.
5. A method according to any preceding claim, wherein the context information is determined as an indoor context or an outdoor context using a machine learning model.
6. The method according to any one of the preceding claims 1 to 5, wherein: For each target sound event, the sound event type indicates one of a center rendering event, a surround rendering event, or a height rendering event, wherein, for the center rendering event, the rendering is distributed across the speaker layout to create a center channel position in the sound field for the target sound event, and for the surround rendering event, the rendering is distributed across the speaker layout to provide a wide sound field, and for the height rendering event, the rendering is distributed across the speaker layout to highlight an enhanced height effect.
7. A method according to any one of the preceding claims 1-6, wherein the speaker layout includes three speakers, including a left speaker, a right speaker and a top speaker, and wherein the rendering is distributed over the left speaker and the right speaker to provide a wide sound field, and is distributed to the top speaker to highlight an enhanced height effect.
8. A method according to any one of the preceding claims 1-6, wherein the speaker layout comprises four speakers, including upper left and upper right speakers and lower left and lower right speakers, wherein for a center rendering event, the rendering is distributed across all four speakers, for a surround rendering event, the rendering is distributed across the lower left and lower right speakers to provide a wide sound field, and for a height rendering event, the rendering is distributed across the upper left and upper right speakers to highlight an enhanced height effect.
9. The method of claim 1, wherein the sound event type is determined during capture of the multi-channel input audio signal and stored as metadata for rendering scheme selection in subsequent rendering.
10. The method of claim 9, wherein the format of the metadata depends on whether capturing and rendering of the multi-channel input audio signal are performed by the same device.
11. The method according to any one of the preceding claims 1 to 10, further comprising: At least one of equalization or dynamic range control is applied to the rendered multi-channel output audio signal.
12. The method according to any of the preceding claims 1-11, wherein rendering the multi-channel output audio signal comprises applying a mixing ratio to the target sound event and the ambient noise based on the event type.
13. The method according to any one of the preceding claims 1 to 12, wherein: The multi-channel input audio signal is rendered by a mobile device including a folding screen, and the method further comprises: Determining, by at least one processor, whether the screen is folded or unfolded; and According to the determination result, if the screen is folded, a first speaker layout is selected for rendering, and if the screen is unfolded, a second speaker layout is selected, wherein the first speaker layout is different from the second speaker layout.
14. A system for processing audio, comprising: one or more processors; as well as A non-transitory computer-readable medium storing instructions, which, when executed by the one or more processors, cause the one or more processors to perform the operations according to any one of claims 1-13.
15. A non-transitory computer-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform the operations according to any one of claims 1-13.
