Apparatus and method for capturing and processing audio signals

Abstract

An audio processing apparatus (120; 110) is disclosed, comprising processing circuitry (121; 111), wherein the processing circuitry is configured to implement the following processing stages to process a plurality of spatial audio signals captured by a microphone array (116a-d): (a) pre-processing a mixed signal of the plurality of spatial audio signals captured by the microphone array (116a-d); (b) estimating a number of dominant sound sources in the mixed signal of the plurality of spatial audio signals captured by the microphone array (116a-d); (c) performing prior information aided localization for one or more dominant sound sources in the mixed signal of the plurality of spatial audio signals captured by the microphone array (116a-d); (d) performing blind separation for one or more dominant sound source signals in the mixed signal of the plurality of spatial audio signals captured by the microphone array (116a-d); (e) performing blind separation disambiguation based on the localization results of processing stage c); (f) encapsulating the one or more localized and separated dominant sound sources into a spatial multi-channel audio format; (g) performing ambient sound recognition to separate ambient sound from the mixed signal of the plurality of spatial audio signals; (h) adding the separated ambient sound to the spatial multi-channel audio format.

Description

Technical Field

[0001] This invention generally relates to audio technology. More specifically, this invention relates to apparatus and methods for acquiring and processing audio signals, particularly 3D spatial audio. Background Technology

[0002] Smartphones, tablets, and other types of consumer electronic devices have been mass-produced and are widely available globally. These devices are typically small, thin, portable, handheld, and lightweight. Furthermore, they are usually equipped with microphones and speakers, as well as processing resources (usually in the form of several microprocessors). Therefore, consumer electronic devices such as smartphones and tablets are portable and easy-to-use devices capable of performing data acquisition and processing of audio signals (e.g., representing 3D spatial audio scenes).

[0003] Once recording is completed using consumer electronic devices such as smartphones, spatial audio data can be transmitted to other audio processing devices to achieve 3D sound rendering. For example, a recording made by someone at a concert using a smartphone can be transmitted (offline) to a remote computer in someone else's room, and then rendered through headphones, giving people in the room an auditory experience of spatial sound effects, as if they were actually at the concert. Figure 2a (This scenario is illustrated.) If the original audio scene was recorded in an object-based audio format, the spatial audio can be reproduced in a manner described above, yielding superior results (both objectively and subjectively). This format can be used to transmit signals belonging to the individual elements that constitute the audio scene. Typically, these elements are called objects. In addition to the object signals, metadata can also be transmitted, which may include the position of static objects in 3D space, the trajectory of moving objects in 3D space, the sound radiation model of the objects, and / or classification labels identifying the object type.

[0004] Typically, objects in an audio scene are categorized into primary sound sources and ambient sounds (such as...). Figure 2b As shown in the scenario, for example, the main sound sources are guitar, drums, and singer. A main sound source is determined to be the most prominent sound source in the scene based on criteria such as the maximum produced sound level, the direct sound from the source (rather than ambient reflected sound), and / or priority (e.g., prioritizing vocals). Ambient sound refers to all sounds that are not part of the main sound source.

[0005] For a given 3D audio scene, by processing the data collected by the microphone array, the signals and metadata of each main sound source and ambient sound can be obtained. For example, some array processing algorithms can identify the direction of arrival (DoA) of the loudest sound source in the 3D audio scene, while others can separate the signals of these sound sources from the scene recording.

[0006] However, limitations remain in using consumer electronic devices such as smartphones to capture and process spatial audio. First, recording 3D sound requires multiple microphones positioned throughout space, but the small size of handheld consumer electronic devices like smartphones limits the number of microphones available and their spacing. Furthermore, sound scattering caused by the geometry and materials of smartphones can contaminate the data recorded by the microphones. Additionally, the processing power of spatial audio is determined by the processing capabilities of the chips in the consumer electronic device. Summary of the Invention

[0007] The object of this invention is to provide an improved device and method for acquiring and processing audio signals.

[0008] The foregoing and other objectives are achieved through the subject matter of the independent claims. Other implementations will be apparent from the dependent claims, the description, and the drawings.

[0009] According to a first aspect, an audio processing apparatus is provided, wherein the audio processing apparatus includes processing circuitry (such as one or more processors) for performing the following processing stages to process multiple spatial audio signals acquired by a microphone array:

[0010] (a) Preprocessing the mixed signal of the plurality of spatial audio signals acquired by the microphone array;

[0011] (b) Estimate the number of primary sound sources in the mixed signal of the plurality of spatial audio signals acquired by the microphone array;

[0012] (c) Perform prior information-assisted localization on one or more main sound sources in the mixed signal of the plurality of spatial audio signals acquired by the microphone array;

[0013] (d) Perform blind separation on one or more main sound source signals in the mixed signal of the plurality of spatial audio signals acquired by the microphone array;

[0014] (e) Based on the localization results of processing stage (c), perform blind separation and disambiguation;

[0015] (f) Encapsulate one or more located and isolated primary sound sources into a spatial multichannel audio format;

[0016] (g) Perform ambient sound recognition to separate the ambient sound from the mixed signal of the plurality of spatial audio signals;

[0017] (h) Add the separated ambient sound to the spatial multichannel audio format.

[0018] Therefore, the audio processing apparatus according to the first aspect can combine several audio signal processing algorithms that exhibit good performance in locating or separating sound sources in a 3D audio scene recorded by a microphone array. In this context, good performance refers to the computational cost of the algorithm, the algorithm's ability to parse multiple potential recorded audio signals, and the quality of the results produced by the algorithm (i.e., the accuracy of location or the fidelity of the separated signal).

[0019] In one possible implementation, the audio processing device is an application server, which is used to receive the plurality of spatial audio signals acquired by the microphone array via a data network.

[0020] In another possible implementation, the plurality of spatial audio signals are acquired by a microphone array comprising four microphones, wherein each microphone acquires a corresponding spatial audio signal component, the four microphones in the microphone array being arranged as follows: (a) at the vertices of an imaginary right-angled tetrahedron located on the front and rear walls of a generally cuboid-shaped housing, or (b) along a straight line extending along the first or second sidewall of the generally cuboid-shaped housing.

[0021] According to a second aspect, a handheld consumer electronic device for acquiring spatial audio is provided, wherein the handheld consumer electronic device includes a generally cuboid-shaped housing having a main front wall (also referred to herein as the front surface), a main rear wall opposite the front wall, a first side wall, a second side wall opposite the first side wall, a top wall, and a bottom wall opposite the top wall. Furthermore, the handheld consumer electronic device according to the second aspect includes a microphone (i.e., transducer) array comprising at least four microphones, each microphone for acquiring a corresponding spatial audio signal component, the four microphones in the microphone array being arranged such that: (a) they are arranged at the vertices of an imaginary right-angled tetrahedron located on the front and rear walls of the housing, or (b) they are arranged along a straight line extending along the first or second side wall of the housing. The handheld consumer electronic device according to the second aspect, with its specific microphone array geometry, is capable of acquiring spatial audio with high quality.

[0022] In one possible implementation, the housing includes a display screen and a casing, the display screen forming at least a portion of the front wall of the housing, and the casing defining the rear wall, the first side wall, the second side wall, the top wall, and / or the bottom wall of the housing.

[0023] In another possible implementation, the first, second, and third microphones of the right-angled tetrahedral microphone array are arranged on the rear wall of the housing, and the fourth microphone of the right-angled tetrahedral microphone array is arranged on the front wall of the housing.

[0024] In another possible implementation, the right-angled tetrahedral microphone array is located near one of the corners of the housing.

[0025] In another possible implementation, the length of the side of the right-angled tetrahedron ranges from 10 mm to 30 mm, especially from 20 mm to 30 mm.

[0026] In another possible implementation, the microphones in the microphone array are arranged in a straight line extending along the first or second sidewall of the housing, and the distance between adjacent microphones in the microphone array is substantially uniform or non-uniform.

[0027] In another possible implementation, the microphones in the microphone array are used for gain calibration and phase matching before acquiring the corresponding spatial audio signal components.

[0028] In another possible implementation, the handheld consumer electronic device includes an audio processing device according to the first aspect to process the plurality of spatial audio signals acquired by the microphone array.

[0029] In another possible implementation, the handheld consumer electronic device includes a communication interface for providing the plurality of spatial audio signals acquired by the microphone array to a remote audio processing device (such as an application server) for processing the plurality of spatial audio signals acquired by the microphone array.

[0030] In another possible implementation, the handheld consumer electronic device is a smartphone or tablet.

[0031] According to a third aspect, an audio processing method is provided, wherein the audio processing method includes the following processing steps to process multiple spatial audio signals acquired by a processing microphone array:

[0032] (a) Preprocessing the mixed signal of the plurality of spatial audio signals acquired by the microphone array;

[0033] (b) Estimate the number of primary sound sources in the mixed signal of the plurality of spatial audio signals acquired by the microphone array;

[0034] (c) Perform prior information-assisted localization on one or more main sound sources in the mixed signal of the plurality of spatial audio signals acquired by the microphone array;

[0035] (d) Perform blind separation on one or more main sound source signals in the mixed signal of the plurality of spatial audio signals acquired by the microphone array;

[0036] (e) Based on the localization results of processing step (c), perform blind separation and disambiguation;

[0037] (f) Encapsulate one or more located and isolated primary sound sources into a spatial multichannel audio format;

[0038] (g) Perform ambient sound recognition to separate the ambient sound from the mixed signal;

[0039] (h) Add the separated ambient sound to the spatial multichannel audio format.

[0040] The audio processing method according to the third aspect can be executed by the audio processing apparatus according to the first aspect. Therefore, other features of the audio processing method according to the third aspect are directly implemented by the functions of the audio processing apparatus according to the first aspect and its different implementations and embodiments described above and below.

[0041] According to a fourth aspect, a computer program product is provided, including a computer-readable storage medium for storing program code that, when executed by a computer or processor, causes the computer or processor to perform the method according to the third aspect.

[0042] Details of one or more embodiments are set forth in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from the specification, drawings, and claims. Attached Figure Description

[0043] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings, in which:

[0044] Figure 1 This is a schematic diagram illustrating a handheld consumer electronic device (presented in the form of a smartphone, including a microphone array for acquiring audio signals) according to an embodiment, an audio processing device for processing audio signals according to an embodiment, and headphones for rendering the processed audio signals.

[0045] Figure 2a and Figure 2b This is a schematic diagram illustrating exemplary application scenarios of a handheld consumer electronic device (presented in the form of a smartphone, including a microphone array for acquiring audio signals) and an audio processing device according to an embodiment;

[0046] Figure 3 This is a schematic diagram, illustrating in more detail the processing blocks and stages implemented by a handheld electronic device (including a microphone array for acquiring audio signals) and / or an audio processing device for processing audio signals according to an embodiment;

[0047] Figure 4 Different views of the housing of a handheld electronic device according to an embodiment are shown;

[0048] Figure 5a , Figure 5b and Figure 5c The front wall portion, rear wall portion, and different usage scenarios of a handheld electronic device (presented in the form of a smartphone) according to the first embodiment are shown.

[0049] Figure 6a , Figure 6b and Figure 6c The front wall portion, rear wall portion, and different usage scenarios of a handheld electronic device (presented in the form of a smartphone) according to the second embodiment are shown.

[0050] Figure 7 For three microphones located at different positions on an exemplary smartphone casing, the geometry (distances from the sound source center are not drawn to scale), impulse response diagrams for different DoA values, amplitude diagrams of the impulse response spectrum for different DoA values, and phase diagrams of the impulse response spectrum for different DoA values ​​are shown.

[0051] Figure 8a , Figure 8b and Figure 8c It shows Figure 7 A more detailed view of the middle plot, namely the amplitude plot of the impulse response spectrum corresponding to different DoA on the left, and the phase plot of the impulse response spectrum corresponding to different DoA on the right.

[0052] Figure 9a and Figure 9b This illustrates the amplitude difference in the impulse response spectrum between the reference electret pickup (referred to as "MicA") and 32 other electret pickups when measurements are performed using a two-port acoustic coupling device. Figure 9a ) and phase difference ( Figure 9b );

[0053] Figure 10a and Figure 10b A schematic diagram showing the placement of the speakers relative to a smartphone-like prototype is presented. All speakers were placed in an anechoic laboratory environment and used to measure the far-field impulse response, which describes the sound scattering caused by the smartphone's form factor.

[0054] Figure 11 A schematic diagram of a signal processing chain implemented by an audio processing device according to an embodiment is shown, which uses a microphone array mounted on a handheld consumer electronic device (presented in the form of a smartphone) according to an embodiment to identify the main sound source and ambient sound.

[0055] Figure 12 This is a flowchart of the steps of an audio processing method according to an embodiment.

[0056] In the following text, the same reference numerals refer to the same or at least functionally equivalent features. Detailed Implementation

[0057] In the following description, reference is made to the accompanying drawings, which form part of this invention, and which illustrate by way of description specific aspects of embodiments of the invention or aspects in which embodiments of the invention may be used. It should be understood that embodiments of the invention can be used in other aspects and include structural or logical variations not depicted in the drawings. Therefore, the following detailed description should not be construed in a limiting sense, and the scope of the invention is defined by the appended claims.

[0058] For example, it is understood that the disclosure relating to the described method is equally applicable to the corresponding device or system used to perform the method, and vice versa. For instance, if one or more specific method steps are described, the corresponding device may include one or more units (such as functional units) to perform the described one or more method steps (e.g., one unit performs one or more steps, or multiple units perform one or more of a plurality of steps respectively), even if such one or more units are not explicitly described or shown in the drawings. On the other hand, for example, if a specific apparatus is described based on one or more units (such as functional units), the corresponding method may include a step to perform the function of one or more units (e.g., one step performs the function of one or more units, or multiple steps perform the function of one or more of a plurality of units respectively), even if such one or more units are not explicitly described or shown in the drawings. Furthermore, it is understood that, unless otherwise expressly stated, features of the various exemplary embodiments and / or aspects described herein may be combined with each other.

[0059] Before describing the embodiments in detail below, some technical background and terminology will be introduced, and one or more of the following abbreviations and acronyms will be used:

[0060] DoA (Direction of arrival)

[0061] FFT (Fast Fourier Transform)

[0062] MSCoh Magnitude Squared Coherence

[0063] Singular Value Decomposition (SVD)

[0064] STFT (Short Term Fourier Transform)

[0065] LMS (Steepest Descent Least Mean Squares)

[0066] LCMV (Linearly Constrained Minimum Variance)

[0067] Second-order statistics of the eigenvalues ​​in SORTE

[0068] p-MUSIC Pseudospectrum Multiple Signal Classification

[0069] Auxiliary Independent Vector Analysis (aux-IVA)

[0070] HOA (High Order Ambisonics)

[0071] PCB (Printed Circuit Board)

[0072] MEMS (Micro-electromechanical systems)

[0073] As used in this article, "signal" can include both data and information.

[0074] As used in this article, "noise" can refer to both sound and non-audio signals. More specifically, "noise" refers to sound or data characteristics in a signal that are considered useless or unwanted.

[0075] As used in this article, a “spatial audio scene” refers to a scene consisting of multiple sound sources placed in a 3D environment and radiating sound.

[0076] As used in this article, “sound source identification” refers to the characterization of the physical characteristics of sound sources in a spatial audio scene (e.g., the location of the sound source in space and the sound source signal separated from the scene).

[0077] As used in this article, "main sound source" refers to a sound source that is significant in an audio scene based on selected criteria.

[0078] As used in this article, "ambient sound" refers to all sounds in an audio scene that are not the primary sound source, i.e., sounds that are not of primary interest.

[0079] As used in this article, an "audio object" is a single element in a recorded audio scene (such as a sound source, a combination of sound sources, or ambient reverberation), which typically consists of a signal (representing sound changes caused solely by the sound source) and some metadata.

[0080] As used in this article, “metadata” refers to information about the audio object rather than the audio signal itself, such as spatial location, primary sound source label, or ambient sound label.

[0081] As used in this article, "audio format" refers to a data structure or protocol used to arrange audio data and any associated metadata, enabling this data to be transmitted between devices. Audio data is typically divided into separate entities called channels.

[0082] As used in this article, an "audio channel" refers to a separate entity within an audio format that represents a single data instance / stream. A channel is typically associated with the signal from a single microphone, but it can also be associated with the signal of an audio object or metadata of an audio scene.

[0083] As used in this article, “object-based audio” refers to an audio format in which the signal of each object in an audio scene is transmitted on a single audio channel, while all metadata is typically transmitted on a single independent channel.

[0084] As used in this article, "audio mix" refers to a recording of an audio scene from one or more microphones. At a single microphone location, the recording contains a mixture of the main sound source and ambient sound.

[0085] As used in this article, "real-time" and "offline" refer to data being transmitted immediately when it is created in a given application (typically cyclically executed after short time intervals) or as a single data block after any period of time.

[0086] As used in this article, a "free field" refers to an idealized theoretical 3D environment in which sound waves do not reflect back from infinity. In other words, it is an infinitely large 3D open space filled with a medium in which sound can propagate.

[0087] As used in this article, "far field" refers to a situation where, under idealized conditions, the distance between the sound source and the sound receiver is large enough that the size of the object no longer affects the sound propagation between them across the entire wavelength range of interest.

[0088] As used in this article, a "filter" refers to a system used as a stage in a signal processing algorithm to alter the characteristics of a signal.

[0089] As used in this article, "spectrum" refers to the sum of multiple distinct sinusoidal components represented by a signal through Fourier analysis. Specifically, the term "frequency" is used when this representation is used to describe how a physical quantity changes over time.

[0090] As used in this article, "digital" refers to the characteristics of a system or device. Taking a computer as an example, this characteristic means the acquisition, storage, and processing of real-world data (signals) at a specific point in time, called a sample.

[0091] As used in this article, "audiofication" refers to the process of reproducing a recorded 3D audio scene through a speaker system or headphones in an environment different from the original recording environment. Audification typically utilizes the psychoacoustic characteristics of binaural hearing to enhance the rendered auditory realism.

[0092] As used in this paper, "prior information assistance" and "blind" in the context of sound source recognition algorithms refer to whether prior knowledge about the audio scene is used to generate results.

[0093] As used in this article, "disambiguation" refers to the process of associating the signal generated by the blind sound source separation algorithm with the corresponding metadata (such as the location or type of the sound source, such as speech or instrumental sound).

[0094] As used in this article, "sampling" refers to discretizing the values ​​associated with the functional domain representing the change of a physical quantity. Sampling performed by digital systems is typically uniform sampling, where the distance between any two consecutive samples remains the same.

[0095] As used in this paper, "impulse response" refers to a time-space function used to describe the linear physical mechanism that determines the change of physical quantities between an energy source and a sensor under a given physical environment (i.e., medium, boundary conditions, initial conditions, geometry).

[0096] As used in this article, "aliasing" refers to a characteristic that manifests as error in the spectrum of a sampled signal. Aliasing is closely related to the sampling method of the signal and the amount of information contained between samples in the original continuous signal. This information is lost after sampling, resulting in spectral errors.

[0097] As used in this article, "convolution" refers to the mathematical process of combining the audio signal of a sound source with a hypothetical physical mechanism between the sound source and a known receiver to obtain the sound produced at the receiver.

[0098] As used in this paper, "acoustic rigidity" characterizes a surface with extremely high acoustic impedance in the normal direction, meaning that the velocity of surface acoustic particles is close to zero, preventing sound waves from penetrating the surface. This term can also be used to describe acoustic propagation media that share a separate surface with another propagation medium.

[0099] As used in this article, "directivity" refers to the characteristics of acoustic sensors and sound sources that describe how they acquire and radiate sound based on spatial location or the surrounding DoA. This characteristic may also be related to the frequency of the sound involved.

[0100] Figure 1 This is a schematic diagram illustrating a handheld consumer electronic device 110 (presented as a smartphone 110, including microphone arrays 116a to 116d for acquiring audio signals) according to an embodiment, a remote audio processing device 120 (such as a cloud application server 120) for processing the audio signals acquired by the handheld consumer electronic device 110 according to an embodiment, and an audio rendering device 130 (presented as headphones or headsets 130) for rendering the processed audio signals. Although in Figure 1 In the illustrated embodiment, the handheld consumer electronic device 110 is implemented as a smartphone with conventional smartphone form factor, but in other embodiments, the handheld consumer electronic device 110 may be, for example, a tablet computer or other handheld consumer electronic device with similar form factor to a smartphone.

[0101] like Figure 1 As shown, in addition to microphone arrays 116a to 116d, smartphone 110 may also include processing circuitry 111, such as one or more processors 111, for processing data to provide the functionality further detailed below. The processing circuitry 111 may be implemented in hardware and / or software and may include digital circuitry or a combination of analog and digital circuitry. The digital circuitry may include components such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), or general-purpose processors. Figure 1As shown, the smartphone 110 may also include a communication interface 113 for communicating with the audio processing device 120 (e.g., a cellular and / or Wi-Fi interface 113 for communicating with the remote audio processing device 120 via a cellular or Wi-Fi communication network 115a). The smartphone 110 may also include a memory 115 for storing executable program code, which, when executed by the processing circuitry 111, causes the smartphone to perform the functions and methods described herein. Similar to the smartphone 110, the audio rendering device 130 (especially the headset 130) may also include one or more processors 131, a communication interface 133 for communicating with the audio processing device 120 via a cellular or Wi-Fi communication network 115b, and a memory 135.

[0102] like Figure 1 As further shown, the audio processing device 120 may be a cloud application server 120. The audio processing device (especially the cloud server 120) may include processing circuitry 121, such as one or more processors 121, for processing data to provide the functionality described in further detail below. The processing circuitry 121 may be implemented in hardware and / or software and may include digital circuitry, or a combination of analog and digital circuitry. The digital circuitry may include components such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), or general-purpose processors. Figure 1 As shown, the audio processing device 120 may further include a communication interface 123 for communicating with the smartphone 110 via Wi-Fi or a cellular communication network 115a, and / or with the audio rendering device 130 via Wi-Fi or a cellular communication network 115b, etc. The audio processing device (especially the cloud server 120) may also include a memory 125 for storing executable program code, which, when executed by the processing circuitry 121, causes the audio processing device (especially the cloud server 120) to perform the functions and methods described herein.

[0103] The following describes several detailed embodiments of a handheld consumer electronic device 110 having a smartphone form factor, and several detailed embodiments of an audio processing device 120. Generally, according to the embodiments disclosed herein, the microphone array 116a to 116d of the handheld consumer electronic device 110 having a smartphone form factor includes four microphones 116a to 116d, wherein each microphone 116a to 116d is used to acquire spatial audio signals, and the four microphones 116a to 116d in the microphone array 116a to 116d are arranged as follows: (i) arranged at the vertices of a right-angled tetrahedron located on the front wall 112a and rear wall 112b of the housing 112 of the handheld consumer electronic device 110 (e.g., ...). Figures 5a to 5c (as shown), or (ii) arranged in a straight line extending along the first sidewall 112c or the second sidewall 112d of the housing 112 (as shown). Figures 6a to 6c (As shown).

[0104] Multiple spatial audio signals acquired by the microphone arrays 116a to 116d of the smartphone 110 can be processed by the smartphone 110's own processing circuitry 111 or by a remote audio processing device 120 (such as an application server 120), which receives the multiple spatial audio signals acquired by the microphone arrays 116a to 116d from the smartphone via a communication network 115a. To process the multiple spatial audio signals acquired by the microphone arrays 116a to 116d, the smartphone 110 and / or the audio processing device 120 can perform the following processing stages, which will be described in detail below:

[0105] (a) Preprocessing the mixed signal of multiple spatial audio signals acquired by microphone arrays 116a to 116d;

[0106] (b) Estimate the number of primary sound sources in the mixed signal of multiple spatial audio signals acquired by microphone arrays 116a to 116d;

[0107] (c) Perform prior information-assisted localization on one or more main sound sources in the mixed signal of multiple spatial audio signals acquired by microphone arrays 116a to 116d;

[0108] (d) Perform blind separation on one or more main sound source signals in the mixed signal of multiple spatial audio signals acquired by microphone arrays 116a to 116d;

[0109] (e) Based on the localization results of processing stage (c), perform blind separation and disambiguation;

[0110] (f) Encapsulate one or more located and isolated primary sound sources into a spatial multichannel audio format;

[0111] (g) Perform ambient sound recognition to separate the ambient sound from the mixed signal of the plurality of spatial audio signals;

[0112] (h) Add the separated ambient sound to the spatial multichannel audio format.

[0113] Therefore, the embodiments disclosed herein can perform the following operations: offline recording of a 3D audio mixed signal using microphone arrays 116a to 116d mounted on a handheld consumer electronic device 110 with a smartphone form factor; offline processing of the 3D audio mixed signal to identify and describe the main sound source; offline processing of the 3D audio mixed signal to identify and describe ambient sound; and offline encapsulation of the separated main sound source and the separated ambient sound into a multi-channel audio data format, wherein the multi-channel audio data format consists of a signal and corresponding metadata, which can be rendered by the audio rendering device 130.

[0114] Figure 2a and Figure 2b This is a schematic diagram illustrating exemplary application scenarios of a handheld consumer electronic device 110 and an audio processing device 120 according to an embodiment. Figure 3 This is a schematic diagram illustrating in more detail the processing blocks and processing stages implemented by the handheld consumer electronic device 110 according to an embodiment and / or the audio processing device 120 according to an embodiment. Figure 3 As shown, the microphone arrays 116a to 116d of the smartphone 110 can collect audio signals in various acoustic environments, such as indoor auditoriums, outdoor stages, family rooms, vehicle cabins, industrial sites, public gathering places, and outdoor venues. Potential sound sources can be, for example, human voices, singing, instrument sounds, mechanical noise, and environmental noise. Figure 3 In all the application scenarios shown, sound waves generated by a sound source in the air can be recorded by a handheld consumer electronic device 110 (especially a smartphone 110 equipped with microphone arrays 116a to 116d). For example, when a user holds the smartphone 110 close to one ear in a call position, when holding the device with one or both hands in a video recording position, or when the smartphone 110 is placed on a surface, the smartphone 110 can record sound through its microphones 116a to 116d. With the help of the special microphone array geometry of the smartphone 110 according to the embodiments disclosed herein, the spatial characteristics of a sound source in a given environment can be intrinsically captured in the recorded audio signal. In this way, data describing a 3D audio scene can be acquired digitally by the processing circuitry 111 of the smartphone 110.

[0115] As described above, in one embodiment, the audio signals recorded in the manner described above are intended for subsequent transmission and processing by a remote audio processing device 120. The audio processing device 120 applies sound source identification algorithms to a 3D audio scene acquired in the form of multi-channel audio signals (i.e., one channel per microphone). The goal of these identification algorithms is to separate the main sound source signals from the 3D audio scene and determine their spatial locations, while simultaneously separating ambient sounds in a format that preserves spatial characteristics. For the identification process outlined above, a classification standard for dividing sound sources into main sound sources and ambient sounds can be selected. Subsequently, the audio processing device 120 can encapsulate all sound source signals separated from the 3D audio scene, along with relevant metadata (such as the location of the main sound source), into a multi-channel audio format, intended for further transmission to an audible device 130, such as headphones 130.

[0116] The audible device 130 (such as headphones 130) receives multi-channel audio formats from the audio processing unit 120 and feeds them into 3D audio scene rendering algorithms (such as high-order Ambisonics) for processing. These rendering algorithms aim to reproduce an auditory experience consistent with the original sound scene, either through speakers placed in a new environment different from the audio recording environment, or through headphones. Speakers are suitable for applications including living room audio, in-vehicle cabin audio, recording studios, cinemas, and virtual reality rooms. Headphones can be used for various indoor and outdoor activities, such as working at a desk, doing housework, exercising, or walking / commuting.

[0117] In order to process the multiple spatial audio signals acquired by the microphone arrays 116a to 116d in the manner described above, the smartphone 110 and / or the audio processing device 120 may employ (i.e. implement) one or more of the following algorithms (some of which will be detailed below):

[0118] • Magnitude Squared Coherence (MSCoh), see reference [MSCoh2023] (a list of references is provided at the end of the specification);

[0119] • Singular Value Decomposition (SVD), see reference [SVD2023];

[0120] • Short-Term Fourier Transform (STFT), see reference [HZ2019];

[0121] • Minimize the least mean squares (LMS) error of the steepest descent in multiple channels, see reference [SJE2001];

[0122] • Algorithms for estimating the number of main sound sources from multi-channel microphone recordings of 3D audio scenes, such as the Sound Source Counting (SORTE) algorithm based on second-order eigenvalue statistics;

[0123] • Algorithms for locating the main sound source based on multi-channel microphone recording in 3D audio scenes, such as pseudospectrum-based multiple signal classification (p-MUSIC) algorithm;

[0124] Algorithms for separating the main sound source signal from multi-channel microphone recordings in 3D audio scenes, such as linearly constrained minimum variance (LCMV) beamforming and / or auxiliary independent vector analysis (aux-IVA) algorithms.

[0125] The Source Counting Based on Eigenvalue Second-Order Statistics (SORTE) algorithm, as its name suggests, is a general signal processing strategy. This algorithm analyzes an ordered set of eigenvalues ​​associated with the signal's power spectral density, which can be obtained, for example, through SVD. The algorithm processes the amplitude difference (gap) between any two consecutive eigenvalues ​​in the ordered set obtained through SVD, and then... Cluster the gaps between every two adjacent eigenvalues ​​of the largest eigenvalues ​​into a new set, and calculate the variance of this new gap set. The ultimate goal is to obtain the variance as a function of... The variation pattern relative to previous values ​​when the variance increases, that is, the variation pattern of variance as the number of gaps between the first few largest eigenvalues ​​used to calculate the variance gradually increases.

[0126] For a sensor array that collects signals generated by sound sources, applying the above process to the auto-power spectral density matrix of the sensor data allows estimation of the number of uncorrelated sound sources that contribute the largest component to the sensor signal. The estimated number of sound sources is equal to the previous... The number of eigenvalues, the variance of the gap set corresponding to these eigenvalues, and the previous... Compared to the variance of the gap set generated by each eigenvalue, the variation is the smallest. For a detailed theoretical explanation of the SORTE algorithm, see reference [ZhetAl2010].

[0127] Let there be a coordinate system whose origin is located at the point where... The center of an array of microphones (such as microphone arrays 116a to 116d in smartphone 110) has Cartesian coordinates as follows: Let the signals collected by these sensors be... ,in, For discrete time-domain sample indexing, For inclusion A column vector of n elements, each element corresponding to a microphone channel. Furthermore, assume there exists in 3D space... There are several main sound sources, and these main sound sources are... Sound is generated at one microphone; [Set] To obtain an accurate estimate of the known number of sound sources, the SORTE algorithm can be implemented according to the following steps:

[0128] (1) Based on the selected window function, the overlap between windows, and the number of frequency points used to calculate the FFT in each window. And parameters such as sampling frequency, calculate STFT.

[0129] Return spectrum , in the form of Matrix, where, For frequency point index, This represents the number of time-domain samples collected by the microphone.

[0130] (2) Calculate each frequency point The corresponding self-power spectral density matrix ,in, This represents the expectation operator. The size is .

[0131] (3) Calculate each frequency point corresponding SVD, where:

[0132] · It is a diagonal square matrix, and its diagonal contains of 1 eigenvalue, Sort by magnitude from largest to smallest, among which, due to Therefore

[0133] · for Left eigenvector matrix,

[0134] · for Right eigenvector matrix,

[0135] For the above decomposition to hold, the matrix The full rank must be reached.

[0136] (4) Calculate each corresponding That is, the interval between every two consecutive eigenvalues ​​in an ordered set. Amplitude differences (gap).

[0137] (5) For For each value in the sequence, calculate the sequence. The variance of is calculated using the following formula:

[0138] .

[0139] (6) For For each value in the dataset, the computational cost is... The calculation formula is as follows:

[0140]

[0141] (7) For ,calculate The minimum value of is the estimated number of sound sources collected in the microphone recording.

[0142] Pseudospectrum-based multiple signal classification (p-MUSIC) is a priori information-assisted data processing technique for sensor arrays, capable of locating uncorrelated sound sources that contribute the largest component to sensor signals. p-MUSIC belongs to the subspace method category. It utilizes SVD analysis to analyze the auto-power spectral density matrix of sensor data and sorts the acquired eigenvalues ​​according to their amplitude. Eigenvalues ​​with the largest amplitudes and all greater than a preset threshold are considered as sound sources corresponding to their respective sensors, while eigenvalues ​​all less than the amplitude threshold constitute a subspace considered as noise.

[0143] Several well-established and recognized variants of the p-MUSIC algorithm are documented in the literature. The p-MUSIC variant analyzes the narrowband spectrum of a signal and requires prior information about the underlying physical mechanism (i.e., the impulse response) between each sensor and a finite set of DoA (Does) or a finite 3D location grid around the sensor array. Furthermore, based on the known physical mechanism, a quantity called the pseudospectrum is computed for each location on each DoA or 3D grid in the set. The pseudospectrum characterizes the orthogonality check result between the known impulse response and the eigenvectors of the noise subspace initially obtained via SVD. Under the assumptions of MUSIC, the pseudospectrum exhibits maximum amplitude at the DoA or 3D grid location corresponding to the sound source because the known impulse response vector is orthogonal to the eigenvectors of the noise subspace. The first MUSIC version proposed in the literature is [RS1986].

[0144] Let there be a coordinate system whose origin is located at the point where... The center of an array of microphones (such as microphone arrays 116a to 116d in smartphone 110) has Cartesian coordinates as follows: Let the signals collected by these sensors be... ,in, For discrete time-domain sample indexing, For inclusion A column vector of n elements, each element corresponding to a microphone channel. Furthermore, assume there exists in 3D space... There are several main sound sources, and these main sound sources are... Sound is generated at one microphone; [Set] This is a known and accurate estimate of the number of sound sources. Finally, assume there is a set of [missing information] surrounding the sensor array. Selected direction These directions correspond to the complementary elevation angle. and azimuth The p-MUSIC algorithm calculates pseudospectral based on a 2D mesh. ,in, This is the frequency domain sample index. The size is... matrix The known spectrum contains the impulse responses, which determine each selected direction. The physical mechanism between each microphone; each microphone corresponds to a row in the matrix, and each selected direction corresponds to a column in the matrix.

[0145] The p-MUSIC algorithm can be implemented by following these steps:

[0146] (1) Based on the selected window function, the overlap between windows, and the number of frequency points used to calculate the FFT in each window. And parameters such as sampling frequency, calculate STFT.

[0147] Return spectrum , in the form of Matrix, where, For frequency point index, This represents the number of time-domain samples collected by the microphone.

[0148] (2) Calculate each frequency point The corresponding self-power spectral density matrix ,in, This represents the expectation operator. The size is .

[0149] (3) Calculate each frequency point corresponding SVD, where:

[0150] · It is a diagonal square matrix, and its diagonal contains of 1 eigenvalue, Sort by magnitude from largest to smallest, among which, due to Therefore

[0151] · for Left eigenvector matrix,

[0152] · for Right eigenvector matrix,

[0153] For the above decomposition to hold, the matrix The full rank must be reached.

[0154] (4) The front Each feature value is categorized into a corresponding main sound source. The remaining... Each eigenvalue is considered a component of the noise subspace. Therefore, let... and for A matrix that contains only the eigenvectors of the noise subspace.

[0155] (5) Corresponding to each frequency point Calculate each DoA Corresponding pseudo-spectrum , The calculation formula is as follows:

[0156] ,

[0157] in, Represents the impulse response matrix The j-th column, which corresponds to The underlying physical mechanism between each of the microphones and the j-th DoA.

[0158] (6) For all frequency points The pseudospectrum is averaged to obtain a frequency-averaged pseudospectrum:

[0159] .

[0160] (7) Amplitude of frequency-averaged pseudospectrum It can be interpreted as a finite and discretized 3D surface, which is composed of Cartesian coordinates in a newly constructed virtual coordinate system. (This is confirmed.) The surface should precisely reflect... There are several peaks, and these peaks correspond to DoA of the main sound source.

[0161] Based on The described surface, using a peak detection algorithm, in Locate the main sound source on the grid.

[0162] (8) (Optional) To improve positioning accuracy, a constraint is imposed on the peak detection algorithm: different main sound sources must not be associated with each other. Two peaks on the grid that are less than or close to a threshold.

[0163] Linearly constrained minimum variance (LCMV) beamforming is a priori information-assisted data processing technique for sensor arrays. Its core principle is to combine the optimal filtered versions of each signal acquired by the sensors to give the array the desired directivity, typically manifested as the ability to distinguish signals acquired from all directions or locations except a selected orientation or a selected location in space. This technique relies on the underlying physical mechanisms between each sensor in the array and the selected orientation / location.

[0164] The linearly constrained minimum variance (LCMV) beamformer is a special case of the minimum variance distortionless response (MVDR) beamformer. MVDR forces its output to a unit value in selected directions or locations while minimizing its output variance. In other words, the components generated in non-selected directions or locations are minimized. For LCMV, multiple directions or locations are selected, some constrained by MVDR (forcing the output to a unit value), and others constrained to zero output (forcing the output to 0). The theory of LCMV and MVDR beamformers is detailed in reference [BR2015].

[0165] Let there be a coordinate system whose origin is located at the point where... The center of an array of microphones (such as microphone arrays 116a to 116d in smartphone 110) has Cartesian coordinates as follows: Let the signals collected by these sensors be... ,in, For discrete time-domain sample indexing, For inclusion A column vector of n elements, each element corresponding to a microphone channel. Furthermore, assume the LCMV beamformer is selected. one direction In these directions, the beamformer output is forced to be a unit value or 0. The dimensions are... matrix (in, (For frequency domain sample indexing) contains known spectra of impulse responses that determine each selected direction. The physical mechanism between each microphone; each microphone corresponds to a row in the matrix, and each selected direction corresponds to a column in the matrix.

[0166] An LCMV beamformer can be implemented by following these steps:

[0167] (1) Based on the selected window function, the overlap between windows, and the number of frequency points used to calculate the FFT in each window. And parameters such as sampling frequency, calculate STFT.

[0168] Return spectrum , in the form of Matrix, where, For frequency point index, This represents the number of time-domain samples collected by the microphone.

[0169] (2) Calculate each frequency point The corresponding self-power spectral density matrix ,in, This represents the expectation operator. The size is .

[0170] (3) Use SVD to calculate each frequency point corresponding The inverse matrix. For the above decomposition to hold, the matrix... The full rank must be reached.

[0171] (4) Calculate the optimal beamforming filter corresponding to frequency point k according to the following formula. Spectrum:

[0172]

[0173] in:

[0174] The superscript "H" indicates the Hermitian transpose of the matrix.

[0175] · For the reason A row vector consisting of 10 elements represents a frequency sample. The spectrum of each optimal beamforming filter.

[0176] · For inclusion A row vector of elements, where each element takes the value 1 (representing maximum beamformer response) or 0 (representing zero beamformer response), and each element corresponds to a selected direction. .

[0177] (5) Calculate each frequency point Corresponding beamformer output To obtain the contents A column vector of n elements.

[0178] (6) Based on the previously used analysis window function, synthesis window function, window overlap, and number of frequency points Including parameters such as sampling frequency, the beamformer output is calculated. The inverse STFT.

[0179] Return by A column vector consisting of 10 elements represents the time-domain form of the beamformer output.

[0180] (7) Normalize the output of the time-domain beamformer to limit its amplitude range to within a certain range. Within the range.

[0181] Auxiliary independent vector analysis (aux-IVA) is a blind data processing technique for sensor arrays. This algorithm separates the signals generated by the sound sources at the sensors by maximizing the independence between the extracted source signal estimates. This independence can be quantified in different ways, with a common method being maximizing non-Gaussianity (i.e., dissimilarity information). The aux-IVA algorithm belongs to the iterative algorithm family called independent component analysis (ICA). Unlike similar algorithms, it processes sensor signals in the frequency domain, taking into account delay differences between sensors, rather than focusing solely on amplitude differences.

[0182] The aux-IVA algorithm treats the spectrum of the signal acquired by the sensor as the product of the unmixing matrix and the spectrum of the original sound source signal vector. The algorithm aims to compute the unmixing matrix, which determines the underlying physical mechanism between each sound source and each sensor. Therefore, the unmixing matrix can be viewed as the spectrum of the impulse response between each sound source producing a component and each sensor. However, the original sound source signal vector needs to satisfy non-Gaussianity for the algorithm to achieve its goal. Furthermore, since the algorithm aims to maximize non-Gaussianity, it struggles to achieve its objective when the sensor signal contains increasingly more mixed signals from various sound sources. This is a direct result of the well-established central limit theorem, which states that "as the sample size increases, regardless of the population distribution, the distribution of the sample mean will approach a normal Gaussian distribution."

[0183] ICA-like algorithms can be implemented in different ways, specifically in non-Gaussian quantization methods, frequency permutation problem solving methods, and iterative step size optimization methods to achieve correct estimation convergence. Understandably, the aux-IVA algorithm uses negative entropy (a fundamental concept in information theory) as a measure of non-Gaussianity; it solves the frequency permutation problem by calculating the unmixing matrix of the complete spectrum of the mixed signal, rather than unmixing each frequency individually; and it avoids iterative step size optimization by minimizing the auxiliary function when calculating the unmixing matrix.

[0184] For the theoretical details of frequency domain aux-IVA applications, please refer to reference [PS1998], while for the solutions to the permutation problem, please refer to reference [AH2006], and for the solutions to avoid step size optimization, please refer to reference [NO2011].

[0185] As described above, according to the embodiments disclosed herein, the microphone array 116a to 116d of the handheld consumer electronic device 110 having a smartphone form factor includes four microphones 116a to 116d, which are arranged as follows: (a) arranged at the vertices of a right-angled tetrahedron located on the front wall 112a and rear wall 112b of the housing 112 of the handheld consumer electronic device 110 (e.g., Figures 5a to 5c (as shown), or (b) arranged in a straight line extending along the first sidewall 112c or the second sidewall 112d of the housing 112 (as shown). Figures 6a to 6c (As shown).

[0186] like Figure 4As shown, the external dimensions (i.e., the shape of a smartphone) can be described as a thickened flat panel with length and width, and rounded edges and corners. Therefore, a handheld consumer electronic device 110 (especially a smartphone 110) includes a rectangular parallelepiped-shaped housing 112 with rounded edges and corners, the housing having a front wall (or front surface) 112a, a rear wall (or back surface) 112b opposite to the front wall 112a, a first side wall 112c, a second side wall 112d opposite to the first side wall 112c, a top wall 112e, and a bottom wall 112f opposite to the top wall 112e. As... Figure 4 As shown, the exemplary dimensions of housing 112 are as follows: length 160 mm, width 80 mm, and thickness 10 mm.

[0187] Figure 5a , Figure 5b and Figure 5c The diagram illustrates the front wall 112a portion, the rear wall 112b portion, and various usage scenarios of a handheld consumer electronic device 110 (particularly a smartphone) according to a first main embodiment, wherein four microphones 116a to 116d are arranged at the vertices of a right-angled tetrahedron located on the front wall 112a and rear wall 112b of the housing 112 of the smartphone 110. Figure 5a and Figure 5b As can be seen, in one embodiment, three microphones 116b to 116c can be arranged flush with the surface of one side (i.e., the rear wall 112b) of the video cameras 118a and 118b, while the fourth microphone 116a is arranged flush with the surface of one side (i.e., the front wall 112a) of the display screen. It is understood that the positions of microphones 116a to 116d form a right-angled tetrahedron, wherein the length of the three right-angled sides (the sides forming the right angles) is 20 mm (e.g., ...). Figure 5a and Figure 5b (as shown). Among the three coplanar microphones 116b to 116d, the microphone 116b closest to the corner can be placed, for example, 20 mm from the two nearest edges. Therefore, the fourth microphone 116a on the screen side (i.e., the front wall 112a) can also be placed 20 mm from the two nearest edges.

[0188] Figure 6a , Figure 6b and Figure 6cThe diagram illustrates the front wall 112a portion, rear wall 112b portion, and various usage scenarios of a handheld consumer electronic device 110 (particularly a smartphone 110) according to a second primary embodiment, wherein four microphones 116a to 116d are arranged along a straight line extending along the first sidewall 112c or the opposite second sidewall 112d of the housing 112. In other words, in this second primary embodiment, the four microphones 116a to 116d can be equidistantly placed along the centerline of the longest sides 112c and 112d of the housing 112 of the smartphone 110. It is understood that the length of the straight line formed by the positions of the four microphones 116a to 116d and the spacing between two adjacent microphones 116a to 116d together determine the lower low-frequency limit and the upper high-frequency limit of the microphone array 116a to 116d, as described in reference [PAN2004]. Both of these values ​​are determined by the length selected for the smartphone 110. The longer the length, the wider the operating frequency range of the microphone arrays 116a to 116d, covering lower frequencies; however, this also increases the spacing and lowers the upper limit of high frequencies. Even so, Figure 6a and Figure 6b The geometry shown utilizes the available length as much as possible to achieve an equidistant arrangement of microphones 116a to 116d. A small gap needs to be reserved between the two microphone positions closest to the corner of the smartphone to accommodate the actual physical size of the acoustic sensor. This small gap is not limited to accommodate any available construction elements.

[0189] Understandable, Figure 5a and Figure 5b The tetrahedral microphone arrays 116a to 116d shown are as follows: Figure 6a and Figure 6b The linear microphone arrays 116a to 116d shown all offer the following advantages: easy integration into existing smartphones; ability to capture 3D audio scene spatial features while reducing the number of sensors used; placement on the smartphone 110 in a location that will not be obstructed by the smartphone user's hand in common application scenarios; and minimizing the impact of sound scattering caused by the shape of the smartphone 110 on the recorded signal. Furthermore, Figure 5a and Figure 5b The tetrahedral microphone arrays 116a to 116d shown are as follows: Figure 6a and Figure 6b The linear microphone arrays 116a to 116d shown do not occupy any area on the surface of the smartphone 110 that is typically reserved for other components.

[0190] It is understandable that any three points in space constitute a plane, and a fourth point located outside this plane has a mirror-symmetric point on the other side of the plane, the distance from which the mirror-symmetric point is exactly the same as that from the original three points. In other words, the propagation distance of sound produced at one sound source location relative to a microphone array at a planar location may be equal to the propagation distance at another sound source location. In a free field, this is equivalent to the sound signals produced by the two sound sources at every planar microphone being exactly the same. Therefore, using at least four microphones, not all of which are coplanar, ensures that the inter-microphone differences in the sound produced by a given sound source are recorded in most possible application scenarios. Such differences are a necessary condition for the effective operation of some sound source recognition algorithms used according to the embodiments disclosed herein.

[0191] exist Figure 5c and Figure 6c The diagram illustrates tetrahedral microphone arrays 116a to 116d and linear microphone arrays 116a to 116d in schematic form for several common usage scenarios of the smartphone 110. These usage scenarios include one-handed video recording, two-handed video recording, ear-to-ear calling, and the phone lying flat on a plane. Figure 5c and Figure 6c It can be inferred that, for Figure 5c The tetrahedral array shown indicates that microphones 116a to 116d are not always blocked by hands in all usage scenarios, but for... Figure 6c The linear array shown has microphones 116a to 116d partially blocked only in a few usage scenarios. When microphones 116a to 116d are blocked by a hand, the components of the sound sources in the surrounding 3D audio scene in the recorded signal may change as the blocking effect intensifies, as described in reference [EBB2020].

[0192] for Figure 5a and Figure 5b The tetrahedral microphone arrays 116a to 116d shown are as follows: Figure 6a and Figure 6bThe linear microphone arrays 116a to 116d shown have been measured in the laboratory using a prototype smartphone 110. The measurements show that, compared to other microphone arrangements (i.e., microphones arranged on four thin sides, or microphones arranged on the two largest surface areas closest to the sides), these two array layouts minimize the acoustic scattering effect of the shape on the audio signals recorded by the microphones 116a to 116d mounted on the smartphone under most DoA conditions. The prototype used contains 32 commercially available electret microphones with a diameter of 6 mm, placed in key positions on the shape and mounted flush. The prototype body is made of hardwood, approximating an acoustically rigid surface in the surrounding air, and has dimensions of 320 mm × 160 mm × 20 mm (L × W × T). In the far field, at a distance of 2 m from the center of the prototype, 10 speakers (Genelec model 8020C) are arranged on a semi-circular truss around the prototype, using a turntable (with... A prototype was rotated at intervals, constructing a total of 720 sound source positions on a virtual sphere. Microphone data was acquired using a sinusoidal sweep method [AF2000, AF2007]. Further details of this experiment will be described below. Figure 7 For three key microphone locations, the impulse response and its spectral amplitude and spectral phase, obtained from laboratory results, are shown. These three locations are labeled “Mic50”, “Mic49”, and “Mic44”, and are respectively placed on a maximum area surface of the prototype (as far away from all edges as possible), a longest side of the prototype, and a shortest side of the prototype.

[0193] exist Figure 7 In the diagram, the sound source positions on the virtual sphere are represented by different colored dot rings surrounding each corresponding microphone position. It is important to emphasize that the sound source positions are shown relative to each microphone and vary from microphone to microphone, although this difference is not obvious in the presented diagram. This is because the microphones are positioned off-center from the turntable's rotation axis. Furthermore, the red rings in the diagram correspond to an offset angle of [missing information]. The loudspeakers. For all 32 microphones, the ring contains 72 closely spaced sound source locations, also due to the microphones being positioned off-center from the turntable's rotation axis. Sound sources are indexed starting from 1, in the order of lowest residual elevation angle (top of each ring), traversing each ring sequentially. For a given microphone, the impulse response, spectral amplitude, and spectral phase corresponding to all sound source locations are plotted in a single color map, where the x-axis represents time or frequency, the y-axis is the sound source index arranged in the aforementioned order, and the color map represents amplitude, magnitude, or phase.

[0194] Understandable, Figure 7The directivity of the microphone and the effect of the diffuser shape on directivity are illustrated. Three color maps of the impulse response clearly show the location of the first maximum peak on the x-axis (bright yellow). If all sound sources around the microphone were equidistant from the free-field microphone, the bright yellow band on the color map would be a straight line parallel to the y-axis. However, because some microphones are not positioned on the turntable's rotation axis, the distances between the sound sources and any given microphone are not perfectly uniform; these distance differences are on the order of millimeters. Furthermore, the diffuser shape, due to its size, introduces an additional delay to the propagation path, the specific duration of which depends on the microphone's placement on the diffuser relative to the sound source. Therefore, the bright yellow bands corresponding to "Mic50" and "Mic49," located on larger features of the diffuser, are not as straight as the band for "Mic44," which introduces minimal delay to the propagation path of most sound sources' DoA, as "Mic44" is located on the thinnest feature of the diffuser shape. Furthermore, from Figure 7 The color mapping of the spectral amplitude shows that, for all three microphones, the amplitude variation with DoA at low frequencies is much smaller than that at high frequencies. For all three microphones, the amplitude variation with DoA at low frequencies ranges from 11 dB to 12 dB. However, at high frequencies up to 7500 Hz, the amplitude variation with DoA differs: "Mic50" varies by 71 dB, while "Mic49" and "Mic44" only vary by 22 dB to 23 dB. The region of significant amplitude drop is clearly discernible on the color mapping for "Mic50". The color mapping data of the unwound phase of the impulse response spectrum shows that, in the frequency range of 200 Hz to 7500 Hz, the phase variation with DoA for "Mic49" and "Mic44" is minimal. However, this is not the case for "Mic50," where the slope of its unwound phase drop is significantly increased and clearly discernible at high frequencies (up to 7500 Hz). These enlargements indicate that, for these regions, in addition to introducing a delay in the free field propagation path between the sound source and the microphone, the shape of the scatterer also introduces an additional delay in the color map.

[0195] Figures 8a to 8c The amplitude and phase maps of all three microphones have been magnified, and only the color mapping map corresponding to the magenta sound source ring is shown, which is located at the residual elevation angle. At that point, it is at the same height as the center of mass of the scatterer's shape, and also at the same height as the acoustic center of "Mic50". Figures 8a to 8cSince only a single ring is shown, the color map can be considered circular on the y-axis. The sound sources directly reaching the microphone and the regions that block them are marked in each figure. It can be seen from the figures that "Mic44" is blocked by all sound source locations on the ring, and its impulse response spectrum amplitude and phase change with DoA are minimal. "Mic49" shows a slight change with the sound source DoA at high frequencies, with the change caused by the blocking being the most significant. "Mic50" also shows a change with the sound source DoA at high frequencies, but its change is much greater than that of the other microphones. The spectral amplitude corresponding to the blocking location decreases significantly, and its starting frequency is much lower than the starting frequency of the slight change observed in "Mic49". Similarly, the unwound phase at the blocking location of "Mic50" shows a significantly steeper descent slope at high frequencies. This indicates that an additional delay is introduced into the free-field propagation path between the sound source and the microphone. Overall, "Mic44" on the 20mm × 160mm surface showed the smallest variation with DoA, followed by "Mic49" on the 20mm × 320mm surface, while "Mic50" at the center of the 160mm × 320mm surface showed the largest variation. This suggests that a thinner scatterer is more suitable for microphone mounting if the goal is to minimize the change in inter-microphone variability caused by scattering.

[0196] Figure 7 and Figures 8a to 8c The experimental results shown only represent sound scattering produced by a smartphone-like shape. In real-world scenarios, smartphone devices have a casing that encloses a cavity filled with dense electronic components, all of which can be made of materials with elastic properties. Sound scattering caused by sound waves incident on an elastic structure is a combination of rigid and elastic components. This principle is explained in detail in [MJ&DF1986]. However, when the acoustic impedance of the structure is significantly greater than that of the surrounding propagation medium—that is, when the propagation medium changes from one with minimal resistance to sound wave propagation to one with significant resistance—the elastic component becomes extremely small. This condition is met when the propagation medium is air and the structure is made of a dense solid material with no air or other lightweight fluid filling the cavity. Therefore, sound scattering produced by a rigid smartphone shape in air is sufficient to approximate the sound scattering produced by a real smartphone assembled from various common components.

[0197] According to an embodiment, microphones 116a to 116d in the microphone array are used for gain calibration and phase matching before acquiring spatial audio signals. It is understood that the calibration and phase matching of microphones 116a to 116d used for sound source identification are crucial because both prior information-assisted algorithms and blind algorithms typically rely on different sound sources placed in 3D space. These sound sources, at one or more microphone locations at a certain distance from the sound source, cause the signal to exhibit different characteristics. These different characteristics typically depend on the wavelength of the relevant sound wave and are visually reflected in the spectral amplitude and phase of the microphone signal.

[0198] According to an embodiment, a standard method for calibrating and phase-matching electret microphones, exemplified by the acoustic coupling cavity method, can be used. Reference [GW&TE1892] describes this method for amplitude calibration. The goal is to position two microphones so that their diaphragms are exposed inside a small, sealed cavity. A sufficiently high uniform sound pressure level is generated within the cavity for the target operating frequency range by compressing the driver unit, and the impulse response is measured using methods such as sinusoidal sweep [AF2000, AF2007]. In this way, it is ensured that the two diaphragms achieve the same frequency-dependent amplitude and phase within the operating frequency range, regardless of their positional differences. For an ideally constructed cavity, any differences between the signals measured at the microphones originate from the construction differences between the two units. These differences can be compensated for by inverse filtering designed based on the measurement results. If no significant phase difference is identified within the operating frequency range, microphone calibration can be completed simply by adjusting the amplitude of each microphone using real-valued gain.

[0199] Calibration and phase matching of two microphones 116a to 116d can be performed using either an absolute or relative method. In the absolute method, one of the two microphones must be calibrated and capable of outputting known values ​​in a specific unit of measurement. In the relative method, the goal is to make the spectral amplitude and phase of the two microphones as close as possible. However, in the relative method, the absolute values ​​of the recorded results expressed in units of measurement may deviate from the true values ​​due to scaling factors. When more than two microphones require calibration and phase matching, one microphone is selected as the reference microphone, and all other microphones are matched to that reference microphone individually. For absolute calibration, the reference microphone can be pre-calibrated.

[0200] Figure 9a and Figure 9bExemplary results of calibration and phase matching of 32 electret microphones using an exemplary coupling cavity are shown, presented as amplitude and phase differences in the impulse response spectra of the microphones, respectively. A reference electret microphone, “MicA,” was used to calculate the differences between it and the remaining 32 electret microphones used; therefore, the calibration performed was relative. Three electret pickups exhibited anomalous characteristics different from the others and were therefore discarded. After discarding these three microphones, the amplitude plot shows that, in the range of 200 Hz to 4000 Hz, 29 curves are relatively flat and have similar characteristics, shifting only vertically along the y-axis. Furthermore, within this range, the maximum gain offset between any two curves is approximately 6 dB. This value represents the maximum span of the amplitude difference relative to the reference microphone. Above 4000 Hz, the first resonance peak appears due to the design of the exemplary acoustic coupling cavity used. From the phase diagram, the 29 curves are relatively flat and parallel between 200 Hz and 1000 Hz, then some curves begin to rise slightly up to 4000 Hz. At 200 Hz, the maximum span of the phase difference relative to the reference microphone corresponds to an alignment offset distance of 3.1 cm, accounting for 1.8% of the wavelength. At 4200 Hz, the maximum span of the phase difference relative to the reference microphone corresponds to an alignment offset distance of 0.38 cm, accounting for 4.6% of the wavelength. Between 200 Hz and 4200 Hz, the alignment offset of the 29 curves is no less than 1.8% and no greater than 4.6%. All the above results indicate that the electret microphone can be calibrated simply by adjusting the real-valued gain of each unit. This is because the process of achieving phase matching through inverse filtering is cumbersome and difficult to achieve ideal results. Therefore, for the small phase offset present in the measured electret pickup head, this method is not worthwhile. For gain calibration, for Figure 9a and Figure 9b The amplitude curves in the dataset, spanning the 200 Hz to 4000 Hz range from 29 available curves, were fitted using a first-order polynomial (linear fitting). Then, the real-valued coefficients of the zero-order polynomial were used as scaling factors to shift each curve along the y-axis, making it as close as possible to the reference microphone (x-axis). This process preserved... Figure 9a and Figure 9b The phase characteristics shown also reduce the new maximum gain offset in the 200 Hz to 4000 Hz range to approximately 1.7 dB. In summary, the amplitude and phase differences presented after calibrating each microphone using real-valued gain are sufficient to apply the calibration data to the sound source identification algorithm.

[0201] The prior information-assisted sound source identification algorithm implemented by the audio processing apparatus 120 according to the embodiment can operate based on the premise that the underlying physical mechanisms determining the 3D audio scene recorded by the microphones are known. These physical mechanisms include: sound radiation from the presence of sound sources; sound propagation paths from each sound source to each microphone; sound scattering effects caused by obstacles present in the 3D audio scene; and / or the recording capability of each microphone 116a to 116d as a sound wave sensor.

[0202] Understandably, in order to commercially deploy the sound source recognition function implemented by microphones 116a to 116d installed on handheld consumer electronic devices 110 (especially smartphones 110), this function needs to be adapted to various possible 3D audio scenarios. The physical mechanisms that determine 3D audio scenarios depend on many parameters (e.g., the relative position of each sound source to the sound receiver, the spectrum produced by each sound source and its relationship to the spectrum of all other sound sources, the presence of sound scattering obstacles, and the position of these obstacles relative to the sound sources and receivers), and the number of possible variations is too numerous to measure in advance to support the algorithm. Therefore, a set of representative deterministic physical mechanisms can be used in the prior information-assisted sound source recognition algorithm to resolve various possible 3D audio scenarios. This set of representative physical mechanisms can be measured in a controlled environment (such as a laboratory) and then stored in the smartphone 110 and / or audio processing device 120 for subsequent application as metadata in the audio signal processing chain.

[0203] To obtain the necessary hypothetical physical mechanisms, the acoustic scattering characteristics of a rigid smartphone form factor placed in a free field were acquired by setting a finite number of sound sources on a far-field virtual surface and installing numerous microphones at key locations on the smartphone 110. As mentioned above, the rigid smartphone form factor is sufficient to approximate the acoustic scattering produced in air by a real smartphone 110 assembled from various common components. This method is effective when all underlying processes generating sound in a 3D audio scene are linear. This is because, under linear conditions, any acoustic scattering obstacles or room boundaries present in the scene can be considered as equivalent virtual sound sources placed in a free field around the scattering body form factor and generate sound along with the actual sound sources. In real-world scenarios, certain types of sound sources, such as noise from large moving machinery or engines, and high-intensity sounds from large speakers at concerts, may not possess linear characteristics. Furthermore, when the sound source is near the scattering body form factor, the acoustic scattering characteristics of the smartphone form factor measured only for far-field sound sources may not accurately characterize the underlying physical mechanisms. The degree of proximity causing this characterization failure depends on the relative relationship between the size of the smartphone form factor and the frequency of the sound waves generated by the sound source. To comprehensively measure the physical mechanism of sound scattering, this finite number of sound sources need to be distributed at multiple distances along the shape of the scatterer, i.e., sampling the volume of space, rather than just sampling the sphere.

[0204] The above measurement method was implemented in an anechoic environment at the Doak Laboratory of the Institute of Acoustics and Vibration, University of Southampton. A smartphone-like prototype was specifically constructed to house 32 commercially available electret microphones, each 6 mm in diameter, placed flush-mounted in key positions on the form factor. The prototype's body was made of hardwood, approximating an acoustically rigid surface in the surrounding air, and its dimensions were 320 mm × 160 mm × 20 mm (L × W × T). In the far field, at a distance of 2 m from the center of the prototype, 10 speakers (Genelec model 8020C) were arranged on a semi-circular truss around the prototype, using a turntable (with... A prototype was rotated at intervals, and a total of 720 sound source positions were constructed on a virtual sphere. A schematic diagram of the experimental geometry is shown below. Figure 10a and Figure 10bAs shown, a sinusoidal sweep frequency method [AF2000, AF2007] was used to acquire microphone data, which was then converted into impulse responses. All equipment used in the experiment was controlled via a Dante network using MATLAB on a Mac Mini computer. Dante is a protocol that allows certain multi-channel audio devices (such as professional sound cards, ADC / DAC units, and signal acquisition systems) to communicate over Ethernet cables. In this network, any signal in the link can be transmitted to any device while ensuring that all devices maintain a low-latency and constant state. In the above experiment, this latency was set to 5 milliseconds via software.

[0205] As mentioned above, in order to handle the Figure 5a and Figure 5b The tetrahedral microphone arrays 116a to 116d shown are... Figure 6a and Figure 6b The multiple spatial audio signals acquired by the linear microphone arrays 116a to 116d shown can be processed in the following stages by the smartphone 110 and / or audio processing device 120:

[0206] (a) Preprocessing the mixed signal of multiple spatial audio signals acquired by microphone arrays 116a to 116d;

[0207] (b) Estimate the number of primary sound sources in the mixed signal of multiple spatial audio signals acquired by microphone arrays 116a to 116d;

[0208] (c) Perform prior information-assisted localization on one or more main sound sources in the mixed signal of multiple spatial audio signals acquired by microphone arrays 116a to 116d;

[0209] (d) Perform blind separation on one or more main sound source signals in the mixed signal of multiple spatial audio signals acquired by microphone arrays 116a to 116d;

[0210] (e) Based on the localization results of processing stage (c), perform blind separation and disambiguation;

[0211] (f) Encapsulate one or more located and isolated primary sound sources into a spatial multichannel audio format;

[0212] (g) Perform ambient sound recognition to separate the ambient sound from the mixed signal of the plurality of spatial audio signals;

[0213] (h) Add the separated ambient sound to the spatial multichannel audio format.

[0214] Therefore, it is understandable that this offline processing chain of 3D audio mixed signals acquired by microphones 116a to 116d can be used to obtain: an estimate of the number of primary sound sources present in the audio scene; a DoA estimate of the primary sound sources present in the audio scene; a separate signal estimate of the primary sound sources present in the scene; and / or a separate signal estimate representing the ambient sound in the scene at each microphone. Figure 11 A detailed block diagram of the processing chain implemented by the smartphone 110 itself and / or the audio processing device 120 is shown, wherein: thin solid lines of monochrome represent single-channel audio signals (such as the exemplary single-channel audio signal 1101); thick solid lines of shaded shade represent multi-channel audio signals (such as the exemplary multi-channel audio signal 1103); dashed lines of monochrome represent metadata (such as the exemplary metadata 1105); and symbol pairs... These represent the elevation and azimuth angles relative to the center of smartphone 110, respectively. For example, these angles are collected by microphones 116a to 116d and processed... Figure 11 The audio signal processed by the processing chain shown is based on three main sound sources, represented by guitar, vocals, and drums, while also including ambient sounds consisting of crowd noise and some environmental noise.

[0215] Therefore, in Figure 11 In the application scenario shown, the 3D audio scene is composed of It consists of a main sound source and ambient sound sources, all of which are installed on a smartphone 110. Recording is done using microphones 116a to 116d, which employ... Figure 5a and Figure 5b The tetrahedral geometry shown or Figure 6a and Figure 6b The linear geometric structure shown.

[0216] Let there be a coordinate system with its origin located at the center of the smartphone 110, where the spherical coordinates represent the coelentral angle. and azimuth Let the angular positions of the three main sound sources—guitar, vocalist, and drums—be as follows: , and The positions of microphones 116a to 116d are ,in, The audio signals (sound pressure changes over time) of the guitar, vocalist, and drums are respectively... , and ,in, The sample index, representing the discrete-time dimension, is acquired by the data recording device. The ambient sound signal is represented as... With discrete time and 3D spatial position Change. This definition is used because ambient sound sources typically consist of multiple sources dispersed throughout space. The signal recorded by the four microphones 116a to 116d is... , , and Together they constitute the original multi-channel mixed signal. Each recorded microphone signal can be represented as the sum of the components of the three main sound source signals and the component of the ambient sound at the microphone location, i.e.:

[0217]

[0218]

[0219] Among them, the operator Represents convolution. Quantity. , and Expressed as an impulse response, it characterizes the decisive physical mechanism by which sound propagates through the air from its source to its receiver. Quantity , and This refers to the positions of the three main sound sources in 3D space. These underlying physical mechanisms strictly rely on the fixed geometry and acoustic properties of all physical objects in the recording environment.

[0220] Figure 11 Some sound source identification algorithms used in the signal processing chain (such as those implemented by smartphone 110 and / or audio processing device 120) may require prior information about the physical mechanisms involved in the construction of the 3D audio scene captured by microphones 116a and 116b. The aforementioned measurement methods can be used to obtain such information. However, in the current application scenario, users may not be able to repeat such measurements each time before audio recording. To circumvent this problem, smartphone 110 can use single measurements of its acoustic characteristics, which are stored in smartphone 110 as metadata, representing the hypothetical physical mechanisms that determine any recorded 3D audio scene. These hypothetical physical mechanisms can take the form of impulse response. In the form of these impulse responses, under laboratory conditions, centered on smartphone 110, including... Each angle position The measurements were taken on a far-field spherical grid. Metadata extracted from the phone and intended for use in the signal processing chain may also include the known positions of microphones 116a to 116d relative to a coordinate system with the center of device 110 as the origin. .

[0221] In the preprocessing stage (a), the signal processing chain implemented by the smartphone 110 and / or the audio processing device 120 processes the raw multi-channel mixed signal. The signal is processed and converted into a preprocessed mixed signal. The conversion consists of two operations: calibration (see processing block 1111) and pre-filtering (see processing block 1113). Calibration involves adjusting the microphone signal based on calibration data acquired in the manner described above. If the microphone sensors 116a to 116d used in device 110 have achieved phase matching, calibration 1111 simplifies to applying a real-valued amplitude gain to each channel. Otherwise, more complex inverse filtering may be required to achieve phase matching. A second aspect of the preprocessing stage (a) implemented by smartphone 110 and / or audio processing device 120 may include pre-filtering 1113, which typically involves an anti-spatial aliasing filter to address the lower and upper frequency limits, respectively, imposed by the maximum and minimum spacing distances between microphones 116a to 116d, as described in reference [PAN2004]. The calibration and processing filters should preserve the amplitude differences between microphones as much as possible, but more importantly, preserve the spectral phase of the original recorded audio signal. This is because the performance of some sound source identification algorithms depends on these differences.

[0222] In the signal processing chain implemented by the smartphone 110 and / or the audio processing device 120, in the preprocessing stage (b) following the preprocessing stage (a), an algorithm for estimating the number of main sound sources present in the recorded 3D audio scene can be used to process the mixed signal. The process is then performed (see processing block 1115). This yields an estimated number of main sound sources. This estimated number can be used in some subsequent stages of the signal processing chain, namely, prior information-assisted localization of the main sound source (see processing block 1117) and separation of ambient sound (see processing block 1125) in processing stage (c). Processing block 1115 can estimate the number of main sound sources based on the highest sound level standard using the SORTE algorithm described above.

[0223] In the processing stage (c) (i.e., the prior information-assisted main sound source localization implemented by processing block 1117), the preprocessed mixed signal needs to be processed. With the assumed physical mechanism and the estimated number of main sound sources Combined, to determine the angular position of the main sound source relative to the smartphone 110 in 3D space. When At this time, this processing step generates metadata, namely the estimated DoA for the guitar, vocals, and drums respectively. , and However, this step only returns the DoA and cannot determine the correlation between the estimated DoA of the main sound source and the estimated value of the separated main sound source signal. In other words, at this stage, it is not yet possible to determine... , and Which one corresponds to the guitar, vocalist, or drum signal? The DoA estimate of the main sound source based on the highest sound level criterion can be achieved using the MUSIC algorithm described above.

[0224] In the processing stage (d) (i.e., blind separation of the main sound source signal implemented by processing block 1119), the preprocessed mixed signal is used alone. To obtain the blind estimate of the main sound source separated from the 3D audio mixture signal. When At that time, this processing step will generate signals corresponding to the guitar, vocalist, and drums. , and However, this step cannot determine the correlation between the separated signal of the main sound source and the determined angular position of the main sound source. In other words, at this stage, it is not yet possible to determine... , and Which one corresponds to the position of the guitar, vocalist, or drums? The separation of the main sound source signal based on the maximum entropy criterion can be achieved using the aux-IVA algorithm described above.

[0225] In the processing stage (e) (i.e., the blind separation result disambiguation is achieved by processing blocks 1121 and 1123 using prior information to assist in the localization result), the hypothetical physical mechanism is used. As prior information, the DoA (Domain of Arrival) estimate of the main sound source obtained from the prior information-assisted localization stage is used. , and Preprocessed mixed signal The above-described LCMV beamforming is applied. This process, implemented by processing block 1121, generates beams corresponding to angular positions. , and Separating the main sound source signal , and However, it is still impossible to determine at this point which of these signals corresponds to the position of the guitar, vocalist, or drums. To perform disambiguation, in processing block 1123, the aforementioned MSCoh algorithm is used to disambiguate each estimated signal. The spectrum of each estimated signal is respectively compared with that of the estimated signal The frequencies of the signals are cross-coherent. The highest cross-coherence value is found between the blindly separated signal from the guitar and the signal obtained by beamforming on the guitar's DoA. The same applies to the other two main sound sources. Therefore, coherence can clearly establish the correspondence between the separated signals and the locations of the main sound sources.

[0226] The above method ultimately produces the estimated main sound source signal separated from the mixed signal. , and And the corresponding metadata of the estimated angle and position of the main sound source. , and The above subscripts indicate that disambiguation has been completed, and each estimated signal is associated with a correctly estimated location in space. However, it is important to emphasize that... Figure 11 The signal processing chain shown does not classify the main sound sources contained in the mixed signal based on timbre; that is, it does not determine whether the sound source is a specific instrument, human voice, mechanical noise, or other type.

[0227] In the processing stage (f) (i.e., the encapsulation of the identified main sound source into a spatial audio data format implemented by processing block 1127), the main sound source estimation signal separated from the mixed signal can be... , and And the corresponding metadata of the estimated angle and position of the main sound source , and The encapsulation is based on a spatial audio format, in which a separate data channel is allocated to each sound source signal, and an additional channel is allocated for metadata containing angular position. This spatial multichannel audio format can be transmitted to other devices, such as the audible system 130, and used to render 3D audio in new environments via speakers or headphones. The estimated main sound source signal is separated from the mixed signal. , and And the corresponding metadata of the estimated angle and position of the main sound source , and It can also be converted to more advanced spatial audio formats, such as high-level Ambisonics.

[0228] Generally, ambient sound in a 3D audio scene exhibits diffuse characteristics and can also consist of many sound sources that are not considered the primary sound source and radiate sound from different locations in space. Typically, it is not feasible to individually identify each sound source constituting the ambient sound from the mixed signal; instead, the ambient sound is identified as a convergence quantity relative to the primary sound source. This convergence quantity varies with location in space, and this variation must be separated from the mixed signal and integrated into a spatial audio format describing the complete 3D audio scene captured by the microphone. The processing chain implemented by the smartphone 110 and / or audio processing device 120 (such as...) Figure 11 (As shown) This problem is solved by separating ambient sound from the mixed signal from each microphone (see processing stage (g) implemented by processing block 1125). Microphones 116a to 116d are distributed at different locations on the smartphone 110, thus enabling the acquisition of ambient sound variations with space, thereby capturing sound differences. Once the positions located at the estimated angles are determined... , and The estimated signal of the main sound source at the location , and This allows us to rewrite the previous formula for mixed signals and express the mixed signal as:

[0229]

[0230]

[0231] In the above formula:

[0232] · , and Corresponding to , and .

[0233] · , and The estimated and separated signals of the main sound source and the unknown real physical mechanism are respectively. The convolution results, these physical mechanisms are respectively in , and An assessment will be conducted at the site.

[0234] · For the microphone position The environmental signal is extracted independently and must be acquired in this step of the signal processing chain.

[0235] Due to signal , and These are estimates derived solely based on assumed physical mechanisms within the signal processing chain, and therefore cannot guarantee the validity of every estimate. , , and They are respectively equivalent to , and The multi-channel steepest descent LMS algorithm mentioned above can be used to mix signals. For reference, with signal , and For multiple input channels, thus for each microphone Obtain estimates of the true physical mechanisms. , and Then, the microphone Ambient sound estimation value It can be derived from the following formula:

[0236]

[0237]

[0238] The ambient sound signal separated at each microphone location can be represented as... .

[0239] In the above-described method for separating ambient sound signals (i.e., processing stage (g) implemented by processing block 1125), the estimated number of main sound sources... It's not directly addressed here. However, when implementing this method at the code level, some operations require this quantity, such as pre-allocating an appropriate amount of memory or executing a loop with a fixed number of steps. Such operations help optimize code execution efficiency and reduce computation time.

[0240] Finally, in the processing stage (h) (which can also be implemented by processing block 1127), the separated ambient sound is... The separated main sound source is then integrated into the spatial audio format. For the aforementioned basic spatial audio format, four new channels need to be added to the existing three channels corresponding to the separated main sound source signals. Each channel corresponds to a microphone position. Separated ambient sound signals Furthermore, the positions of the four microphones 116a to 116d in space can be determined. The addition of an additional metadata channel enables the audio rendering device 130 to subsequently audibly process the spatial characteristics of ambient sound.

[0241] Figure 12This is a flowchart illustrating audio processing method 1200, which summarizes the... Figure 11 The different processing stages (a) to (h) are described in the context. Figure 12 The audio processing method 1200 shown includes the following processing steps to process multiple spatial audio signals acquired by microphone arrays 116a to 116d:

[0242] (a) Preprocessing 1201 of the mixed signal of multiple spatial audio signals acquired by microphone arrays 116a to 116d;

[0243] (b) Estimate the number of main sound sources in the mixed signal of multiple spatial audio signals acquired by microphone arrays 116a to 116d in 1203;

[0244] (c) Perform 1205 prior information-assisted localization on one or more main sound sources in the mixed signal of multiple spatial audio signals acquired by microphone arrays 116a to 116d;

[0245] (d) Perform 1207 blind separation on one or more main sound source signals in the mixed signal of multiple spatial audio signals acquired by microphone arrays 116a to 116d;

[0246] (e) Based on the location results of processing step (c), perform 1209 blind separation and disambiguation;

[0247] (f) Encapsulate one or more located and separated main sound sources into a spatial multichannel audio format;

[0248] (g) Perform 1213 ambient sound recognition to separate ambient sound from the mixed signal;

[0249] (h) Add the separated ambient sound to the spatial multichannel audio format.

[0250] Method 1200 can be performed by audio processing device 120 (such as application server 120 according to an embodiment) or electronic device 110 (especially smartphone 110 according to an embodiment). Therefore, other features of method 1200 are implemented by the functions of audio processing device 120 and electronic device 110 and their different embodiments described above and below.

[0251] Those skilled in the art will understand that the “blocks” (“units”) in the various figures (methods and apparatuses) represent or describe the functionality of embodiments of the invention (and are not necessarily independent “units” in hardware or software), thereby equally describing the functionality or features (unit = step) of apparatus embodiments and method embodiments.

[0252] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatus and methods can be implemented in other ways.

[0253] For example, the described embodiments of the device are merely exemplary. For example, the unit division is only a logical functional division, and other division methods may exist in actual implementation. For example, multiple units or components may be merged or integrated into another system, or some features may be ignored or not performed. Furthermore, the mutual coupling or direct coupling or communication connections shown or discussed may be implemented through some interface. Indirect coupling or communication connections between devices or units may be implemented electronically, mechanically, or otherwise.

[0254] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiment solution according to actual needs.

[0255] In addition, the functional units in the embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0256] List of references:

[0257] [AF2000] A. Farina. “Simultaneous Measurement of Impulse Response andDistortion with a Swept-Sine Technique”. In: Audio Engineering Society Convention 108, Paris, 2000.

[0258] [AF2007] A. Farina. “Advancements in Impulse Response Measurements bySine Sweeps”. In: Audio Engineering Society Convention 122, 2007.

[0259] [AH2006]A. Hiroe. “Solution of permutation problem in frequencydomain ICA, using multivariate probability density functions”. In:International Conference on Independent Component Analysis and SignalSeparation. Springer. 2006, pp. 601–608.

[0260] [AKetAl2019]A. Küçük et al. “Real-time convolutional neural network-based speech source localization on smartphone”. In: IEEE Access 7, 2019, pp.169969–169978.

[0261] [AmpMe2021]AmpMe. AmpMe,Play Music louder. 2021. URL:https: / / www.ampme.com / ?locale=en_US.

[0262] [APetAl2018]A. Politis, S. Tervo, and V. Pulkki. “Compass: Coding andmultidirectional parameterization of ambisonic sound scenes,” in Proc. IEEEInt. Conf. Acoust. Speech Signal Process. (ICASSP), (Calgary, Canada), pp.6802–6806, Apr. 2018.

[0263] [BR2015]B. Rafaely. “Chapter 7: Beamforming with noise minimization”.In: “Fundamentals of Spherical Array Processing”. 1st edition, Springer-Verlag Berlin Heidelberg, 2015.

[0264] [EBB2020]E. B. Brixen. “The effect of hand position on handheldmicrophones' frequency response and directivity”. In: Audio EngineeringSociety Convention 148, May 2020.

[0265] [GW&TE1892]G. S. K. Wong and T. F. W. Embleton. “Three-port two-microphone cavity for acoustical calibrations”. In: The Journal of theAcoustical Society of America 71, 5, May 1982, pp. 1276–1277.

[0266] [JLetAl2015]J. Liu et al. “Snooping keystrokes with mm-level audioranging on a single phone”. In: Proceedings of the 21st Annual InternationalConference on Mobile Computing and Networking. 2015, pp. 142–154.

[0267] [HZ2019]H. Zhivomirov. “On the Development of STFT-analysis andISTFT-synthesis Routines and their Practical Implementation”. In: TEMJournal. Volume 8, Issue 1, February 2019, pp. 56-64.

[0268] [LG&DE2001]L. Girod and D. Estrin. “Robust range estimation usingacoustic and multimodal sensing”. In: Proceedings 2001 IEEE / RSJ InternationalConference on Intelligent Robots and Systems. Expanding the Societal Role ofRobotics in the Next Millennium (Cat. No. 01CH37180). Vol. 3. IEEE. 2001, pp.1312–1320.

[0269] [LM&AP2022]L. McCormack and A. Politis, “Estimating and reproducingambience in ambisonic recordings,” in Proc. 30th Eur. Signal Process. Conf.(EUSIPCO), (Belgrade, Serbia), pp. 314–318, Aug. 2022.

[0270] [MH&GF2011]M. H. Hennecke and G. A. Fink. “Towards acoustic self-localization of ad hoc smartphone arrays”. In: 2011 Joint Workshop on Hands-free Speech Communication and Microphone Arrays. IEEE. 2011, pp. 127–132.

[0271] [MJ&DF1986]M. Junger and D. Feit. “Chapter 11: Elastic scatterers andwaveguides”. In: “Sound, Structures, and Their Interaction”, 2 nd edition.Cambridge, MA: MIT Press, 1986.

[0272] [MSCoh2023]The MathWorks, Inc. “mscohere – Magnitude squaredcoherence”. In: “Spectral Analysis Documentation”, mathworks.com. Accessed:December, 2023. [Online]. Available: https: / / uk.mathworks.com / help / signal / ref / mscohere.html.

[0273] [NO2011]N. Ono. “Stable and fast update rules for independent vectoranalysis based on auxiliary function technique”. In: 2011 IEEE Workshop onApplications of Signal Processing to Audio and Acoustics (WASPAA). IEEE.2011, pp. 189–192.

[0274] [NSetAl2018]N. Shankar et al. “Influence of MVDR beamformer on aspeech enhancement based smartphone application for hearing aids”. In: 201840th Annual International Conference of the IEEE Engineering in Medicine andBiology Society (EMBC). IEEE. 2018, pp. 417–420.

[0275] [NSetAl2019]N. Shankar, G. S. Bhat, and I. Panahi. “Comparison andreal-time implementation of fixed and adaptive beamformers for speechenhancement on smartphones for hearing study”. In: Proceedings of Meetings onAcoustics 178ASA. Vol. 39. 1. Acoustical Society of America. 2019, p. 055009.

[0276] [PAN2004]P.A. Nelson. “Chapter 3: Source identification andlocation”. In: “Advanced Applications in Acoustics, Noise and Vibration”, 1stedition. Edited by K. F. Fahy and J. Walker. CRC Press, 2004.

[0277] [PLetAl2015]P. Lazik et al. “Ultrasonic time synchronization andranging on smartphones”. In: 21st IEEE Real-Time and Embedded Technology andApplications Symposium. IEEE. 2015, pp. 108–118.

[0278] [PS1998]P. Smaragdis. “Blind separation of convolved mixtures in thefrequency domain”. In: Neurocomputing 22.1-3, 1998, pp. 21–34.

[0279] [RS1986]R. Schmidt. “Multiple emitter location and signal parameterestimation”. In: IEEE transactions on antennas and propagation 34, 3, 1986,pp. 276–280.

[0280] [SJE2001]S.J. Elliott. “Chapter 4: Multichannel Control of TonalDisturbances”, In: “Signal Processing and its Applications”. Academic Press,2001, pp.186-191.

[0281] [SLetAl2017]S. Li et al. “Auto++ detecting cars using embeddedmicrophones in realtime”. In: Proceedings of the ACM on Interactive, Mobile,Wearable and Ubiquitous Technologies 1.3, 2017, pp. 1–20.

[0282] [SSetAl2014]S. Sur, T. Wei, and X. Zhang. “Autodirective audiocapturing through a synchronized smartphone array”. In: Proceedings of the12th annual international conference on Mobile systems, applications, andservices. 2014, pp. 28–41.

[0283] [SSSetAl2019]S. S. Sandha et al. “Exploiting smartphone peripheralsfor precise time synchronization”. In: 2019 IEEE Global Conference on Signaland Information Processing (GlobalSIP). IEEE. 2019, pp. 1–6.

[0284] [STetAl2007]S. Takada et al. “Sound source separation using null-beamforming and spectral subtraction for mobile devices”. In: 2007 IEEEWorkshop on Applications of Signal Processing to Audio and Acoustics.IEEE.2007, pp. 30–33.

[0285] [SVD2023]The MathWorks, Inc. “svd – Singular value decomposition”.In: “Linear Algebra Documentation”, mathworks.com. Accessed: December, 2023.[Online]. Available: https: / / uk.mathworks.com / help / matlab / ref / double.svd.html.

[0286] [VP2007]V. Pulkki, “Spatial sound reproduction with directional audiocoding,” J. Audio Eng. Soc. (JAES), vol. 55, pp. 503–516, June 2007.

[0287] [VSPetAl2023]V.S. Paul, H. Nara and & J. Hollebon. “Extraction ofambience sound from microphone array recordings for spatialisation”. In:Immersive and 3D Audio: from Architecture to Automotive (I3DA). IEEE. 2023.

[0288] [ZHetAl2010]Z. He, A. Cichocki, S. Xie and K. Choi. “Detecting theNumber of Clusters in n-Way Probabilistic Clustering”. In: IEEE Transactionson Pattern Analysis and Machine Intelligence, vol. 32, no. 11, November 2010,pp. 2006-2021.

Claims

1. An audio processing device (120; 110), characterized in that, The system includes processing circuitry (121; 111), wherein the processing circuitry (121; 111) is used to implement the following processing stages to process multiple spatial audio signals acquired by the microphone arrays (116a to 116d): (a) Preprocessing the mixed signal of the plurality of spatial audio signals acquired by the microphone array (116a to 116d); (b) Estimate the number of primary sound sources in the mixed signal of the plurality of spatial audio signals acquired by the microphone arrays (116a to 116d); (c) Perform prior information-assisted localization on one or more main sound sources in the mixed signal of the plurality of spatial audio signals acquired by the microphone arrays (116a to 116d); (d) Perform blind separation on one or more main sound source signals in the mixed signal of the plurality of spatial audio signals acquired by the microphone arrays (116a to 116d); (e) Based on the localization results of processing stage (c), perform blind separation and disambiguation; (f) Encapsulate one or more located and isolated primary sound sources into a spatial multichannel audio format; (g) Perform ambient sound recognition to separate the ambient sound from the mixed signal of the plurality of spatial audio signals; (h) Add the separated ambient sound to the spatial multichannel audio format.

2. The audio processing apparatus (120) according to claim 1, characterized in that, The audio processing device (120) is an application server (120) for receiving the plurality of spatial audio signals collected by the microphone arrays (116a to 116d) via a data network.

3. The audio processing apparatus (120; 110) according to claim 1 or 2, characterized in that, The plurality of spatial audio signals are acquired by a microphone array (116a to 116d) comprising four microphones (116a to 116d), wherein each microphone (116a to 116d) acquires spatial audio signals, and the four microphones (116a to 116d) in the microphone array (116a to 116d) are arranged as follows: (a) at the vertices of a right-angled tetrahedron located on the front and rear walls of the housing of the electronic device (110), wherein the housing is cuboid in shape, or (b) along a straight line extending parallel to the first or second sidewall of the housing.

4. An electronic device (110), characterized in that, The electronic device (110) includes: The shell (112) is in the shape of a cuboid and has a front wall (112a), a rear wall (112b) opposite to the front wall (112a), a first side wall (112c), a second side wall (112d) opposite to the first side wall (112c), a top wall (112e), and a bottom wall (112f) opposite to the top wall (112e). A microphone array (116a to 116d) includes four microphones (116a to 116d), each of which is used to acquire spatial audio signals. The four microphones (116a to 116d) in the microphone array (116a to 116d) are arranged as follows: (a) at the vertices of the right-angled tetrahedrons located on the front wall (112a) and the rear wall (112b) of the housing, or (b) along a straight line extending parallel to the first side wall (112c) or the second side wall (112d) of the housing (112).

5. The electronic device (110) according to claim 4, characterized in that, The housing (112) includes a display screen (114) and a casing, the display screen forming at least a portion of the front wall (112a) of the housing (112), and the casing defining the rear wall (112b), the first side wall (112c), the second side wall (112d), the top wall (112e), and / or the bottom wall (112f) of the housing (112).

6. The electronic device (110) according to claim 4 or 5, characterized in that, The first, second, and third microphones of the right-angled tetrahedral microphone array (116a to 116d) are arranged on the rear wall (112b) of the housing (112), and the fourth microphone of the right-angled tetrahedral microphone array (116a to 116d) is arranged on the front wall (112a) of the housing (112).

7. The electronic device (110) according to claim 6, characterized in that, The right-angled tetrahedral microphone array (116a to 116d) is located near one of the corners of the housing (112).

8. The electronic device (110) according to claim 6 or 7, characterized in that, The side length of the right-angled tetrahedron is in the range of 10 mm to 30 mm.

9. The electronic device (110) according to claim 4 or 5, characterized in that, The microphones in the microphone array (116a to 116d) are arranged in a straight line extending parallel to the first sidewall (112c) or the second sidewall (112d) of the housing (112), and the distance between adjacent microphones in the microphone array (116a to 116d) is uniform or non-uniform.

10. The electronic device (110) according to any one of claims 4 to 9, characterized in that, The microphones in the microphone array (116a to 116d) are used for gain calibration and phase matching before acquiring the spatial audio signal.

11. The electronic device (110) according to any one of claims 4 to 10, characterized in that, The electronic device (110) includes an audio processing apparatus (110) according to claim 1 for processing the plurality of spatial audio signals acquired by the microphone array (116a to 116d).

12. The electronic device (110) according to any one of claims 4 to 10, characterized in that, The electronic device (110) includes a communication interface (113) for providing the plurality of spatial audio signals acquired by the microphone array (116a to 116d) to an audio processing device (120) for processing the plurality of spatial audio signals acquired by the microphone array (116a to 116d).

13. The electronic device (110) according to any one of claims 4 to 12, characterized in that, The electronic device (110) is a smartphone (110) or a tablet computer (110).

14. An audio processing method (1200), characterized in that, The following processing stages are included to process multiple spatial audio signals acquired by the microphone arrays (116a to 116d): (a) Preprocessing (1201) the mixed signal of the plurality of spatial audio signals acquired by the microphone array (116a to 116d). (b) Estimate (1203) the number of primary sound sources in the mixed signal of the plurality of spatial audio signals acquired by the microphone arrays (116a to 116d); (c) Perform (1205) prior information-assisted localization on one or more main sound sources in the mixed signal of the plurality of spatial audio signals acquired by the microphone arrays (116a to 116d); (d) Perform (1207) blind separation on one or more main sound source signals in the mixed signal of the plurality of spatial audio signals acquired by the microphone arrays (116a to 116d); (e) Based on the location results of processing stage (c), perform (1209) blind separation and disambiguation; (f) Encapsulate one or more located and isolated primary sound sources (1211) into a spatial multichannel audio format; (g) Perform (1213) ambient sound recognition to separate the ambient sound from the mixed signal; (h) Add the separated ambient sound (1215) to the spatial multichannel audio format.

15. A computer program product, characterized in that, Includes a computer-readable storage medium for storing program code, wherein, when executed by a computer or processor, the program code causes the computer or processor to perform the method (1200) according to claim 14.

Images