Method for multi-channel audio reconstruction and loudspeaker system using same

Abstract

The present disclosure provides a method for multichannel audio reconstruction of a speaker system and a speaker system using the same. The method may comprise: receiving stereo audio; carrying out human voice extraction on the stereo audio and outputting a stereo human voice signal and a stereo non-human voice signal; rendering the stereophonic human voice signal and the stereophonic non-human voice signal based on principal and ambient sound extraction (PAE), respectively, to obtain a rendered human voice output and a rendered non-human voice output; and mixing the rendered human voice output and the rendered non-human voice output to generate a multi-channel audio.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to audio processing, and in particular to an optimization method for multi-channel audio reconstruction and a loudspeaker system using the same. BACKGROUND

[0002] Home theaters are increasingly popular because multi-channel sound reproduction can create a more immersive sound field with realistic front sound stage and natural surround for users compared to mono or stereo reproduction. Now, many loudspeaker systems such as JBL speakers, such as a soundbar with multi-beam or portable speakers featuring a "party mode", have physical multi-channels to support multi-channel sound reproduction.

[0003] However, in a home scenario, users more often enjoy music of near-stereo material. The most common and simplest way to broadcast such stereo material is to copy and transmit the stereo / mono signal to each speaker, which is used in the "party mode" feature of JBL speakers. This produces a louder but unnatural sound field, which is also a waste of multi-channel systems. Therefore, it is naturally desirable to re-render stereo audio to multi-channel audio so that the benefits of multi-channel reproduction can be obtained, like JBL soundbars equipped with matrix decoders (JBL NSP, Dolby Atmos, DTS). Although matrix decoders have been common for more than forty years, there are still some challenges in improving the surround and creating a more realistic proximity.

[0004] In addition, some JBL speakers provide a "karaoke mode" feature to users by playing non-human music, but most of them are limited to mono or stereo reproduction. It can be a trend to expand the "karaoke mode" to multi-channel sound reproduction for better entertainment.

[0005] Early conventional matrix decoders were designed to adjust the gain of each channel by analyzing the amplitude and phase information of the input signal. There are some problems with this kind of matrix decoding technology, such as crosstalk caused by low channel separation and inaccurate positioning of the sound image. In order to solve the above problems, new matrix decoding methods have gradually been derived. The core idea of the matrix encoding method is to separate the direct sound and reverberation sound contained in the stereo signal based on time-frequency transform. The direct sound can be played directly through matrix decoding, so that users can locate the sound source, so the sound source positioning is superior to that of conventional matrix decoders. The reverberation sound is independent of the direct sound, and the reverberation sound can be distributed to the surround sound channels to provide high-quality spatial sense.

[0006] In practice scenarios, most of the stereo material is music signals with human voice as the main component. Therefore, when extracting direct sound from a stereo signal, directional musical instruments can be overwhelmed by human speech, leading to an unclear separation of the sound image of the instruments. For example, the JBL NSP decoder extracts a main sound component and an ambient sound component based on coherence analysis between the left and right channels, and then places the extracted main sound component into front channels (left channel, right channel and center channel with adjustable gain) and distributes the ambient sound component to surround sound channels. This strategy can create a focused center image with a good immersive sound field. However, there are still two disadvantages for the NSP decoder. First, the instruments are almost overwhelmed in the center channel, losing their spatial distribution, so the width of the front sound field is limited. Second, some main sound residuals remain in the ambient sound part, which may sound a little unnatural due to limited separation degree.

[0007] Due to the development of machine learning, different sound sources such as human voice, bass, piano, etc. can be extracted with a higher separation degree. Therefore, the separated audio objects can be processed and remixed into multi-channel audio. However, due to the need for a large database for model training (about human voice data and various musical instrument sound data), a limited number of sound sources can be separated. Therefore, there is still a long way to go before this technical solution can be widely used.

[0008] Therefore, there is a need for improved technology to overcome the above-mentioned deficiencies. SUMMARY

[0009] According to one aspect of the present disclosure, a method for multi-channel audio reconstruction for a loudspeaker system is provided. The method can include receiving a stereo audio, performing human voice extraction on the stereo audio and outputting a stereo human voice signal and a stereo non-human voice signal, performing a main sound and ambient sound extraction (PAE) based rendering on the stereo human voice signal and the stereo non-human voice signal, respectively, to obtain a rendered human voice output and a rendered non-human voice output, and mixing the rendered human voice output and the rendered non-human voice output to generate a multi-channel audio.

[0010] According to another aspect of the disclosure, a speaker system is provided. The speaker system can include a memory configured to store instructions and a processor coupled to the memory. The processor can be configured to execute the instructions to receive stereo audio, perform voice extraction on the stereo audio and output a stereo voice signal and a stereo non-voice signal, perform primary and ambient extraction (PAE) based rendering on the stereo voice signal and the stereo non-voice signal, respectively, to obtain a rendered voice output and a rendered non-voice output, and mix the rendered voice output and the rendered non-voice output to generate a multi-channel audio.

[0011] According to yet another aspect of the disclosure, a non-transitory computer readable storage medium including computer executable instructions that, when executed by a computer, cause the computer to perform the methods disclosed herein. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 A schematic diagram of multi-channel audio generation according to one or more embodiments of the disclosure is shown.

[0013] Figure 2 A schematic diagram of a PAE based rendering method performed according to one or more embodiments of the disclosure is shown.

[0014] Figure 3 An example of an N-channel reproduction system consisting of N loudspeakers is shown.

[0015] Figure 4 Rendering of ambient sound components according to one or more embodiments of the disclosure is shown.

[0016] To facilitate an understanding of the present disclosure, like reference characters are used throughout the various figures and like reference characters will be understood to refer to like elements in which: the same designation is used to refer to the same element in the figures as much as possible. It is contemplated that elements disclosed in one embodiment can be advantageously used in other embodiments without specific recitation. The figures referred to herein are not intended to be drawn to scale unless specifically indicated. Also, for clarity and explanation, the figures are often simplified and details or components are omitted. The figures and the discussion herein serve to explain the principles discussed below, in which like numbers refer to like elements. DETAILED DESCRIPTION

[0017] Examples will be provided below for illustration. The description of various examples will be presented for purposes of illustration, but is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art, without departing from the scope and spirit of the described embodiments.

[0018] The present disclosure provides a novel method for multi-channel audio reconstruction that employs a novel approach to expand stereo audio signals into multi-channel audio signals such that a speaker system with physical multi-channels can be utilized to the maximum extent and can provide a wider front sound stage and a more immersive experience. This is achieved by using an optimized method of multi-channel audio reconstruction using a combination of stereo audio based pre-processing (i.e. vocal extraction) and PAE (primary and ambient extraction) based rendering. The pre-processing part helps to eliminate the influence between vocals and other primary instrument components, which guarantees the positioning stability of vocals and other instruments. The PAE based rendering part can provide a wider front sound stage and a more immersive experience. Reference will be made below to Figures 1 to 4 The method is explained in detail.

[0019] Figure 1 A schematic diagram of multi-channel audio generation according to one or more embodiments of the present disclosure is shown. The proposed method of multi-channel audio generation can be implemented based on a combination of stereo audio pre-processing and PAE based rendering, which can be implemented by the pre-processing module 102 and the rendering module 104 shown, respectively. Figure 1 Each of the pre-processing module 102 and the rendering module 104 can be a software or firmware module included in a speaker system.

[0020] In some embodiments, the pre-processing module 102 performs vocal extraction on a received stereo audio signal in real time to separate the received stereo audio into vocal segments and non-vocal segments (also referred to as stereo vocal signal and stereo non-vocal signal hereinafter). In some examples, the pre-processing module 102 performs vocal extraction using some existing techniques, for example, blind source separation (such as ICA (Independent Component Analysis), NMF (Non-negative Matrix Factorization), etc.). In some examples, the pre-processing module 102 performs vocal extraction using machine learning pre-processing, for example, using the machine learning pre-processing method disclosed in WO2022082607A1 (“VOCAL TRACK REMOVAL BY CONVOLUTIONAL NEURAL NETWORK EMBEDDED VOICE FINGER PRINTING ON STANDARD ARM EMBEDDED PLATFORM”). Since the pre-processing performed by the pre-processing module 102 only separates stereo audio into two kinds of signals, i.e. vocal signal and non-vocal signal, without further separating the sound signals of various instruments from the stereo audio, a large database including various types of sound data is not required, thereby greatly improving the practicability of the multi-channel audio reconstruction method proposed by the present disclosure.

[0021] In some embodiments, rendering module 104 may receive a signal output from preprocessing module 102 and render the received signal based on PAE in the time-frequency domain. In some embodiments, rendering module 104 may include two submodules 1042 and 1044 to perform PAE-based rendering on stereo vocal signals and stereo non-vocal signals, respectively. For example, submodule 1042 may receive the extracted stereo vocal signal and perform PAE on the stereo vocal signal to obtain the main tone component and ambient sound component of the stereo vocal signal. Submodule 1042 may also render the main tone component and ambient sound component of the stereo vocal signal separately. Submodule 1042 may then output the rendered vocal output, which may include a vocal signal assigned to at least one speaker or channel of a speaker system. Similarly, submodule 1044 may receive the extracted stereo non-vocal signal and perform PAE on the stereo non-vocal signal to obtain the main tone component and ambient sound component of the stereo non-vocal signal. Submodule 1044 can also render the main tone component and ambient sound component of the stereo non-human voice signal separately. Then, submodule 1044 can output the rendered non-human voice output, which may include the non-human voice signal distributed to at least one speaker or channel.

[0022] Then, the rendered human voice output from submodule 1042 and the rendered non-human voice output from submodule 1044 can be mixed to generate multi-channel audio.

[0023] In some implementations, a judgment module can be set between the preprocessing module 102 and the rendering module 104. Figure 1 (Not shown in the image) to determine whether the speaker system operates in party mode or karaoke mode. For example, if the speaker system is operated to switch from party mode to karaoke mode, the speaker system can be controlled so that only module 1044 performs PAE-based rendering, since non-vocal music will play in karaoke mode. On the other hand, if the speaker system is operated to switch from karaoke mode to party mode, the speaker system can be controlled so that both submodule 1042 and submodule 1044 each perform PAE-based rendering. Figure 1 The above framework of the multi-channel audio generation method shown has an additional advantage: the speaker system can more easily achieve multi-channel audio reconstruction in both "party mode" and "karaoke mode".

[0024] Figure 2 A schematic diagram of a PAE-based rendering method performed according to one or more embodiments of the present disclosure is shown. Figure 2The illustrated rendering process can be applicable to both stereo vocal signals and stereo non-vocal signals. In other words, the rendering process including the processes at blocks 202 to 210 can be performed by Figure 1 sub-module 1042 and sub-module 1044 in FIG. 1B, respectively.

[0025] In some embodiments, at block 202, for example, the stereo vocal / non-vocal signals output from the pre-processing module 102 can be received and transformed with a short-time Fourier transform (STFT). At block 204, PAE can be performed based on the transformed stereo vocal / non-vocal signals. Details of the PAE will be shown as follows.

[0026] Generally, stereo audio can be roughly divided into two categories according to its recording form, i.e., studio recording and live recording. In the case of studio recording, different sound sources are recorded separately and mixed, and finally a synthesized environmental sound component is added to form a two-channel signal. While live recording requires capturing all sound source signals naturally including environmental sound information through a distributed arrangement of multiple microphones on site. Regardless of which recording method is used, stereo audio can be seen as a combination of a main sound component (direct sound) and an environmental sound component (reverberant sound). In other words, either of the extracted vocal and non-vocal audio can be seen as a combination of a main sound component and an environmental sound component.

[0027] Therefore, for a stereo audio input (either of stereo vocal audio and stereo non-vocal audio) x(t) = [x L (t), x R (t)], it can be represented in time-frequency domain as follows.

[0028]

[0029] p represents a main sound component, G L / R represents a gain value of the main sound component in the left / right channel, a L / R represents an environmental sound component in the left / right channel. The expression represents a product with p. Similarly, represents a product with p. The variables b and k represent time and waveband index, respectively. To simplify the expression form, (b, k) is ignored in the subsequent analysis in the present disclosure. In addition, for the sake of simplicity, the expression x(t) of the stereo audio input can represent either of the stereo vocal signal or the stereo non-vocal signal, which is only for illustrating the principle of PAE.

[0030] The core of the matrix decoder lies in separating the main sound component and the environmental sound component, i.e., p, aL / R To solve this problem, a separation matrix W e R 2×2 is defined to extract the dominant sound component, and can be solved by minimizing the error between the extracted component and the target dominant sound component, which can be formed as an optimization problem as follows.

[0031]

[0032] The solution of W above can be optimized by different convex optimization methods such as least square method. Thus, the dominant sound component and the ambient sound component can be represented as follows.

[0033]

[0034] As shown in Figure 2 , the dominant sound component and the ambient sound component obtained at block 204 can be rendered at blocks 206 and 208 respectively based on the spatial perception principle.

[0035] For the rendering of the dominant sound component at block 206, it is first needed to calculate the direction of the sound source based on the spatial orientation information contained in the dominant sound component. Based on the calculated direction of the sound source, the loudspeaker for reproducing the sound source can be determined. Then, the signal assigned to each of the determined loudspeakers should be calculated. The basic idea of such rendering is to use two adjacent loudspeakers (or channels) to reproduce the sound source between the two adjacent loudspeakers. For example, there is an N-channel reproduction system consisting of N loudspeakers, where the azimuth angle is As shown in Figure 3 , if the direction of the sound source is , it can be obtained by gain estimation on the dominant sound component. Then the sound source in this direction, denoted as p i and p i+1 , can be reproduced using loudspeaker i and loudspeaker i+1. i The assigned signals for the two loudspeakers (or channels) p i+1 should satisfy the energy conservation principle. Thus the reconstruction matrix G is defined as the following equation (4).

[0036]

[0037] where and . is a function of , which can be used to modify the values of G1 and G2 to control the amount of dominant sound component fed to each loudspeaker / channel. Thus, can affect the perception of the direction of the sound source.

[0038] Thus, and can be calculated as equation (5).

[0039]

[0040] For the rendering of the ambient sound components at block 208, the goal is to achieve a diffuse sound field, in particular to generate ambient sound signals for the surround sound channels. It is required to ensure that the ambient sound components in the left and right channels of the N channels, i.e. a L and a R , are uncorrelated. This can be achieved by convolving the extracted ambient sound components a L and a R with respective recorded or simulated impulse responses, i.e. respective predetermined impulse responses. Figure 4 A schematic diagram showing the further decorrelation of the ambient sound components a L and a R is shown in Fig. 4. For example, M decorrelators are provided for the M surround sound channels. Each of the decorrelators can be implemented by two filters with respective recorded or simulated impulse responses, such as filters 401 and 402 shown in Fig. 4 as an example. The filters 401 and 402 can have different predetermined impulse responses. For example, the extracted ambient sound component a L is input to the filter 401 and the extracted ambient sound component a R is input to the filter 402. Then, the output of the filter 401 and the output of the filter 402 can be mixed to generate a rendered ambient sound signal for the surround sound channel 1. It can be understood that each of the decorrelators 2 to M can be implemented as above. Figure 4

[0041] Returning to Figure 2 , at block 210, an inverse short-time Fourier transform (ISTFT) can be performed on the rendered primary sound signals and the rendered ambient sound signals.

[0042] ​Through the processing at blocks 202-210, a rendered vocal output can be obtained, where the rendered vocal output can include a rendered primary sound signal and a rendered ambient sound signal obtained based on the extracted stereo vocal signal. Further, through the processing at blocks 202-210, a rendered non-vocal output can be obtained, where the rendered non-vocal output can include a rendered primary sound signal and a rendered ambient sound signal obtained based on the extracted stereo non-vocal signal. It can be appreciated that the rendered vocal output and the rendered non-vocal output are themselves multi-channel outputs to some extent. For example, in a party mode, the rendered vocal output and the rendered non-vocal output can be mixed to generate a multi-channel audio. In a karaoke mode, the rendered non-vocal output can be output as a multi-channel audio. In some embodiments, the speaker system can include at least one control element that can be operated to control the speaker system to switch between the party mode and the karaoke mode.

[0043] It can be appreciated that the methods discussed above can be implemented by a processor included in the speaker system. The speaker system can include a memory and a processor. The memory can be configured to store computer-readable instructions or code to cause the processor to perform the above-described aspects of the present disclosure. The processor can be any technically feasible hardware unit configured to process data and execute software applications, including but not limited to a central processing unit (CPU), a microcontroller unit (MCU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP) chip, etc.

[0044] The descriptions of various embodiments have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. The terminology used is intended to describe the principles described and their applications or modifications as

[0045] In the foregoing, reference has been made to embodiments presented in the disclosure. However, the scope of the disclosure is not limited to the particular described embodiments. Rather, any combination of the aforementioned features and elements, whether related to different embodiments or not, is contemplated to implement and practice contemplated embodiments. Additionally, although embodiments disclosed herein can achieve advantages over other possible solutions or over the prior art, whether or not a given embodiment achieves advantages over other possible solutions is not a limitation of the scope of the disclosure. Thus, the foregoing aspects, features, embodiments and advantages are merely illustrative and are not considered elements or limitations of the scope of the claims except where explicitly recited in a claim.

[0046] Aspects of the disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that can all generally be referred to herein as a "circuit," "module," "unit," or "system."

[0047] The disclosure can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0048] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanism that is encoded with instructions (such as a punch card or a hole in a card that is encoded with instructions) and any suitable combination of the foregoing. As used herein, the computer readable storage medium is not to be construed as being transitory signals per se, such as electric, electromagnetic or optical signals propagating through a transmission medium or space, but is to be construed to be a non-transitory computer readable storage medium that is a tangible device having a non-transitory computer readable medium therein.

[0049] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers.

[0050] Aspects of the present disclosure are described herein with reference to the drawings, which are as follows.

[0051] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0052] The flow and block diagrams in the drawings show the architectural, functional, and operational aspects of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow and block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical functions (or action). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession can in fact be executed substantially concurrently or can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block and / or flow diagrams and combinations of blocks in the block and / or flow diagrams can be implemented by dedicated hardware-based systems that perform the specified functions or acts, or combinations of dedicated hardware and computer instructions.

[0053] While the foregoing is directed to implementations of the present disclosure, other and further implementations of the disclosure can be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.

[0054] Clause 1. In some embodiments, a method for multi-channel audio reconstruction for a speaker system, the method comprising: receiving stereo audio; performing voice extraction on the stereo audio and outputting a stereo voice signal and a stereo non-voice signal; performing primary and ambient extraction (PAE) based rendering on the stereo voice signal and the stereo non-voice signal, respectively, to obtain a rendered voice output and a rendered non-voice output; and mixing the rendered voice output and the rendered non-voice output to generate multi-channel audio.

[0055] Clause 2. The method of clause 1, wherein performing the PAE-based rendering on the stereo vocal signal and the stereo non-vocal signal comprises: performing a short-time Fourier transform (STFT) on the stereo vocal signal and the stereo non-vocal signal to obtain transformed stereo vocal signals and transformed stereo non-vocal signals; performing PAE on the transformed stereo vocal signals and the transformed stereo non-vocal signals to obtain a vocal component and an ambient component for each of the transformed stereo vocal signals and the transformed stereo non-vocal signals; rendering the vocal component and the ambient component for each of the transformed stereo vocal signals and the transformed stereo non-vocal signals to obtain a rendered vocal signal and a rendered ambient signal corresponding to each of the stereo vocal signal and the stereo non-vocal signal; and performing an inverse short-time Fourier transform (ISTFT) on the rendered vocal signal and the rendered ambient signal corresponding to each of the stereo vocal signal and the stereo non-vocal signal to obtain the rendered vocal output and the rendered non-vocal output.

[0056] Clause 3. The method of any of clauses 1-2, wherein rendering the vocal component comprises: calculating a direction of a sound source based on spatial orientation information included in the vocal component; determining two adjacent loudspeakers for reproducing the sound source based on the spatial orientation information; and calculating a signal to be assigned to each of the two adjacent loudspeakers.

[0057] Clause 4. The method of any of clauses 1-3, wherein calculating the direction of the sound source comprises performing a gain estimation on the vocal component.

[0058] Clause 5. The method of any of clauses 1-4, wherein calculating the signal to be assigned to each of the two adjacent loudspeakers comprises calculating the signal to be assigned to each loudspeaker based on a reconstruction matrix and a separation matrix.

[0059] Clause 6. The method of any of clauses 1-5, wherein rendering the ambient component comprises convolving the ambient component in left and right channels, respectively, with different predetermined impulse responses to generate surround signals for surround sound channels.

[0060] Clause 7. The method of any of clauses 1-6, further comprising determining whether the loudspeaker system is operating in a party mode or a karaoke mode.

[0061] Clause 8. In some embodiments, a speaker system, the speaker system comprising: a memory configured to store instructions; and a processor configured to execute the instructions to: receive stereo audio; perform vocal extraction on the stereo audio and output a stereo vocal signal and a stereo non-vocal signal; perform primary and ambient sound extraction (PAE) based rendering on the stereo vocal signal and the stereo non-vocal signal, respectively, to obtain a rendered vocal output and a rendered non-vocal output; and mix the rendered vocal output and the rendered non-vocal output to generate a multi-channel audio.

[0062] Clause 9. The speaker system of clause 8, wherein the processor is further configured to: perform a short-time Fourier transform (STFT) on the stereo vocal signal and the stereo non-vocal signal to obtain a transformed stereo vocal signal and a transformed stereo non-vocal signal; perform PAE on the transformed stereo vocal signal and the transformed stereo non-vocal signal to obtain a primary sound component and an ambient sound component for each of the transformed stereo vocal signal and the transformed stereo non-vocal signal; render the primary sound component and the ambient sound component for each of the transformed stereo vocal signal and the transformed stereo non-vocal signal to obtain a rendered primary sound signal and a rendered ambient sound signal corresponding to each of the stereo vocal signal and the stereo non-vocal signal; and perform an inverse short-time Fourier transform (ISTFT) on the rendered primary sound signal and the rendered ambient sound signal corresponding to each of the stereo vocal signal and the stereo non-vocal signal to obtain the rendered vocal output and the rendered non-vocal output.

[0063] Clause 10. The speaker system of any one of clauses 8-9, wherein the processor is further configured to: compute a direction of a sound source based on spatial orientation information included in the primary sound component; determine two adjacent speakers for reproducing the sound source based on the spatial orientation information; and compute a signal assigned to each of the two adjacent speakers.

[0064] Clause 11. The speaker system of any one of clauses 8-10, wherein the processor is further configured to compute the direction of the sound source by performing a gain estimation on the primary sound component.

[0065] Clause 12. The speaker system of any one of clauses 8-11, wherein the processor is further configured to compute the signal assigned to each speaker by: computing the signal assigned to each of the two adjacent speakers based on a reconstruction matrix and a separation matrix.

[0066] Clause 13. The loudspeaker system of any one of clauses 8-12, wherein the processor is further configured to render the ambient sound component by convolving the ambient sound component in left and right channels with different predetermined impulse responses, respectively, to generate surround sound signals for surround sound channels.

[0067] Clause 14. The loudspeaker system of any one of clauses 8-13, wherein the loudspeaker system includes at least one control element that is operable to control the loudspeaker system to switch between a party mode and a karaoke mode.

[0068] Clause 15. In some embodiments, a computer-readable storage medium comprising computer-executable instructions that, when executed by a computer, cause the computer to perform the method of any one of clauses 1-7.

Claims

1. A method for multi-channel audio reconstruction for a speaker system, the method comprising: receiving stereo audio; performing voice extraction on the stereo audio and outputting a stereo voice signal and a stereo non-voice signal; performing primary and ambient sound extraction (PAE) based rendering on the stereo voice signal and the stereo non-voice signal, respectively, to obtain a rendered voice output and a rendered non-voice output; and mixing the rendered voice output and the rendered non-voice output to generate multi-channel audio.

2. The method of claim 1, wherein performing the PAE based rendering on the stereo voice signal and the stereo non-voice signal comprises: performing a short-time Fourier transform (STFT) on the stereo voice signal and the stereo non-voice signal to obtain a transformed stereo voice signal and a transformed stereo non-voice signal; performing PAE on the transformed stereo voice signal and the transformed stereo non-voice signal to obtain a primary sound component and an ambient sound component for each of the transformed stereo voice signal and the transformed stereo non-voice signal; rendering the primary sound component and the ambient sound component for each of the transformed stereo voice signal and the transformed stereo non-voice signal to obtain a rendered primary sound signal and a rendered ambient sound signal corresponding to each of the stereo voice signal and the stereo non-voice signal; and performing an inverse short-time Fourier transform (ISTFT) on the rendered primary sound signal and the rendered ambient sound signal corresponding to each of the stereo voice signal and the stereo non-voice signal to obtain the rendered voice output and the rendered non-voice output.

3. The method of claim 2, wherein rendering the primary sound component comprises: calculating a direction of a sound source based on spatial orientation information included in the primary sound component; determining two adjacent speakers for reproducing the sound source based on the spatial orientation information; and calculating a signal to be assigned to each of the two adjacent speakers.

4. The method of claim 3, wherein calculating the direction of the sound source comprises performing a gain estimation on the primary sound component.

5. The method of claim 3, wherein calculating the signal to be assigned to each of the two adjacent speakers comprises calculating the signal to be assigned to each speaker based on a reconstruction matrix and a separation matrix.

6. The method of claim 2, wherein rendering the ambient sound component comprises convolving the ambient sound component in left and right channels, respectively, with different predetermined impulse responses to generate surround sound signals for surround sound channels.

7. The method of claim 1, further comprising determining whether the speaker system is operating in a party mode or a karaoke mode.

8. A speaker system, the speaker system comprising: a memory configured to store instructions; and a processor configured to execute the instructions to: ​ receive stereo audio; perform human voice extraction on the stereo audio and output a stereo human voice signal and a stereo non-human voice signal; perform primary and ambient sound extraction (PAE) based rendering on the stereo human voice signal and the stereo non-human voice signal to obtain a rendered human voice output and a rendered non-human voice output, respectively; and mix the rendered human voice output and the rendered non-human voice output to generate a multi-channel audio.

9. The loudspeaker system of claim 8, wherein the processor is further configured to: perform a short-time Fourier transform (STFT) on the stereo human voice signal and the stereo non-human voice signal to obtain a transformed stereo human voice signal and a transformed stereo non-human voice signal; perform PAE on the transformed stereo human voice signal and the transformed stereo non-human voice signal to obtain a primary sound component and an ambient sound component for each of the transformed stereo human voice signal and the transformed stereo non-human voice signal; render the primary sound component and the ambient sound component for each of the transformed stereo human voice signal and the transformed stereo non-human voice signal to obtain a rendered primary sound signal and a rendered ambient sound signal corresponding to each of the stereo human voice signal and the stereo non-human voice signal; and perform an inverse short-time Fourier transform (ISTFT) on the rendered primary sound signal and the rendered ambient sound signal corresponding to each of the stereo human voice signal and the stereo non-human voice signal to obtain the rendered human voice output and the rendered non-human voice output.

10. The loudspeaker system of claim 9, wherein the processor is further configured to: compute a direction of a sound source based on spatial orientation information included in the primary sound component; determine two adjacent loudspeakers for reproducing the sound source based on the spatial orientation information; and compute a signal to be assigned to each of the two adjacent loudspeakers.

11. The loudspeaker system of claim 10, wherein the processor is further configured to compute the direction of the sound source by performing a gain estimation on the primary sound component.

12. The loudspeaker system of claim 10, wherein the processor is further configured to compute the signal to be assigned to each loudspeaker by computing the signal to be assigned to each of the two adjacent loudspeakers based on a reconstruction matrix and a separation matrix.

13. The loudspeaker system of claim 9, wherein the processor is further configured to render the ambient sound component by convolving the ambient sound component in left and right channels with different predetermined impulse responses, respectively, to generate surround sound signals for surround sound channels.

14. The loudspeaker system of claim 8, wherein the loudspeaker system includes at least one control element operable to control the loudspeaker system to switch between a party mode and a karaoke mode. ​ ​ 15. A computer-readable storage medium comprising computer-executable instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1-7.

Images