Adaptive downmixing of audio signals with improved continuity

Abstract

The present invention provides a method, system, and storage medium for forming an output multi-channel audio signal having two desirable attributes for efficient encoding from an input multi-channel audio signal. [Solution] The method receives an input multi-channel audio signal including a primary input audio channel and L non-primary input audio channels, determines L sets of input gains, forms a scaled non-primary input audio channel for each channel and gain, forms a primary output audio channel from the sum of the primary input audio channel and the scaled non-primary input audio channels, determines L sets of prediction gains, forms a prediction channel from the primary output audio channel, and forms L non-primary output audio channels.

Description

[Technical Field]

[0001]

[0001] Cross-reference of related applications This application claims priority to U.S. Provisional Patent Application No. 63 / 037,635, filed on 11 June 2020, and U.S. Provisional Patent Application No. 63 / 193,926, filed on 27 May 2021, both of which are incorporated herein by reference in their entirety.

[0002]

[0002] Technical field This disclosure generally relates to audio coding, and more particularly to the coding of multi-channel audio signals. [Background technology]

[0003]

[0003] When an input audio signal is to be stored or transmitted (for example, to a listener for playback) for later use, it is often desirable to encode the audio signal to reduce the amount of data. The process of data reduction applied to an input audio signal is generally called “audio encoding” (or “encoding”), and the device used for encoding is generally called an “audio encoder” (or “encoder”). The process of reproducing an output audio signal from the reduced data is generally called “audio decoding” (or “decoding”), and the device used for decoding is generally called an “audio decoder” (or “decoder”). Audio encoders and decoders can be configured to operate on an input signal consisting of a single audio channel or multiple audio channels. When the input signal consists of multiple audio channels, the audio encoder and audio decoder are called a multi-channel audio encoder and a multi-channel audio decoder, respectively. [Overview of the Initiative]

[0004]

[0004] An embodiment relating to adaptive downmixing of audio signals with improved continuity is disclosed.

[0005]

[0005] In some embodiments, the audio encoding method includes: a step of at least one processor receiving an input multi-channel audio signal comprising a primary input audio channel and L non-primary input audio channels; a step of at least one processor determining a set of L input gains, where L is a positive integer greater than 1; a step of forming individual scaled non-primary audio channels for each of the L non-primary input audio channels and L input gains from individual non-primary input audio channels scaled according to the input gain; a step of forming a primary output audio channel from the sum of the primary input audio channel and the scaled non-primary input audio channels; and a step of at least one processor The process includes the steps of: a processor determining a set of L predictive gains; for each of the L predictive gains, at least one processor forming a predictive channel from a primary output audio channel scaled according to the predictive gain; at least one processor forming L non-primary output audio channels from the difference between individual non-primary input audio channels and individual predictive signals; at least one processor forming an output multi-channel audio signal from the primary output audio channel and the L non-primary output audio channels; an audio encoder encoding the output multi-channel audio signal; and at least one processor transmitting or storing the encoded output multi-channel audio signal.

[0006]

[0006] In some embodiments, the step of determining a set of L input gains includes: determining a set of L mixing coefficients; determining input mixing intensity coefficients; and determining L input gains by scaling the L mixing coefficients by the input mixing intensity coefficients.

[0007]

[0007] In some embodiments, the step of determining L sets of predicted gains includes: determining L sets of mixing coefficients; determining predicted mixing intensity coefficients; and determining L predicted gains by scaling the L mixing coefficients by the predicted mixing intensity coefficients.

[0008]

[0008] In some embodiments, the input mixing intensity coefficient h is determined by a prior prediction constraint equation h = fg, where f is a predetermined constant value greater than 0 and less than or equal to 1, and g is the predicted mixing intensity coefficient.

[0009]

[0009] In some embodiments, the predicted mixing intensity coefficient g is:

[0010]

number

[0011]

number

[0012]

[0010] In some embodiments, the covariance matrix of the intermediate signal is calculated from the covariance matrix of the multi-channel input audio signal.

[0013]

[0011] In some embodiments, two or more input multi-channel audio channels are processed according to a mixing matrix to generate a primary input audio channel and L non-primary input audio channels.

[0014]

[0012] In some embodiments, the primary input audio channel is determined by the dominant eigenvector of the expected covariance matrix of a typical input multi-channel audio signal.

[0015]

[0013] In some embodiments, each of the L mixing coefficients is determined based on the correlation between each of the non-primary input audio channels and the primary input audio channel.

[0016]

[0014] In some embodiments, the encoding step includes the step of allocating more bits to the primary output audio channel than to the L non-primary output audio channels, or the step of discarding one or more of the L non-primary output audio channels.

[0017]

[0015] Other embodiments disclosed herein relate to systems, devices, and computer-readable media. Details of the disclosed embodiments are described in the accompanying drawings and the specification. Other features, objectives, and advantages are apparent from the specification, the drawings, and the claims.

[0018]

[0016] The specific embodiments disclosed herein provide one or more of the following advantages. An input multi-channel audio signal is processed by an audio pre-mixer to form an output multi-channel audio signal having two desirable attributes for efficient encoding. The first characteristic property is that at least one dominant audio channel of the output multi-channel audio signal includes most or all of the sonic elements of the input multi-channel audio signal. The second characteristic property is that each of the audio channels of the output multi-channel audio signal is predominantly uncorrelated with each of the other audio channels. A simple encoder can be provided with certain data to assist in the reproduction of audio channels discarded by the simple encoder.

[0019]

[0017] The above two characteristics enable the output multi-channel audio signal to be efficiently encoded by a simple encoder by either allocating fewer bits to the encoding of the less dominant channels or by choosing to completely discard the less dominant audio channels.

Brief Description of the Drawings

[0020]

[0018] In the drawings, for ease of explanation, a particular arrangement or order of schematic elements, such as those representing devices, units, instruction blocks, and data elements, is shown. However, it should be understood by those skilled in the art that a particular ordering or arrangement of schematic elements in the drawings is not intended to imply that a particular order or sequence of processing, or that a separation of processing, is required. Further, including schematic elements in the drawings is not intended to mean that such elements are required in all embodiments, or that the features represented by such elements may not be included in or combined with other elements in some embodiments.

[0021]

[0019] Furthermore, in the drawings, connecting elements such as solid lines, dashed lines, or arrows are used to indicate connections, relationships, or associations between or among two or more other schematic elements, and the absence of such connecting elements is not intended to mean that a connection, relationship, or association cannot exist. In other words, some connections, relationships, or associations between elements are not shown in the drawings so as not to obscure the disclosure. Furthermore, for the sake of clarity, a single connecting element is used to represent multiple connections, relationships, or associations between elements. For example, if a connecting element represents the transmission of signals, data, or commands, it will be understood by those skilled in the art that such an element represents one or more signal paths that may be required to affect the communication. [Figure 1]

[0020] Figure 1 is a block diagram of a simple audio encoder and a simple audio decoder configuration intended to form an output multi-channel audio signal according to one embodiment, and is a representation of an input multi-channel audio signal. [Figure 2]

[0021] Figure 2 is a block diagram of an audio codec system including an audio encoder, an audio decoder, an encoder pre-mixer, and a decoder post-mixer, according to one embodiment. [Figure 3]

[0022] Figure 3 shows the arrangement of processing elements in one embodiment, where the input multi-channel audio signal is split into sub-band signals by a filter bank, and each sub-band is processed by a mixing matrix to generate a remixed sub-band signal. [Figure 4]

[0023] Figure 4 is a block diagram of the arrangement of two mixing operations intended to perform the functions of the encoder pre-mixer of Figure 2 or the encoder pre-mixer of Figure 3, according to some embodiments. [Figure 5]

[0024] Figure 5 is a block diagram of a predictive mixer according to one embodiment. [Figure 6]

[0025] Figure 6 shows the arrangement of processing elements that implement the decoder-post-mixer of Figure 2 according to some embodiments. [Figure 7]

[0026] Figure 7 is a flowchart of the process for adaptive downmixing of an audio signal with improved continuity, according to one embodiment. [Figure 8]

[0027] Figure 8 is a block diagram of a system according to some embodiments for implementing the features and processes described with reference to Figures 1-7.

[0028] The same reference symbols used in various drawings represent similar elements. [Modes for carrying out the invention]

[0022]

[0029] The following detailed description includes many specific details to provide a complete understanding of the various embodiments described. It will be apparent to those skilled in the art that the various embodiments described can be carried out without these specific details. In other examples, well-known methods, procedures, components, and circuits are not described in detail so as not to unnecessarily obscure the aspects of the embodiments. Several features that can be used independently of each other or in any combination of other features are described below.

[0023]

[0030] term Where used herein, the term "including" and its variations should be read as an open-ended term meaning "including, but not limited to." The term "or" should be read as "and / or" unless the context explicitly indicates otherwise. The term "based on" should be read as "based at least partially on." The terms "one implementation" and "implementation" should be read as "at least one implementation." The term "another implementation" should be read as "at least one other implementation." The terms "determined," "determine," or "determining" should be read as "obtaining," "receiving," "calculating," "estimating," "predicting," or "deriving." Furthermore, in the following description and claims, unless otherwise specified, all technical and scientific terms used herein have the same meanings as those generally understood by those skilled in the art in which this disclosure belongs.

[0024]

[0031] Figure 1 is a block diagram of a configuration 10 of a simple audio encoder and a simple audio decoder intended to form a multi-channel audio signal 17(Z') which is a facsimile of a multi-channel audio signal 13(Z). The multi-channel audio signal 13 is processed by the simple audio encoder 14 to generate an encoded representation 15, which can be transmitted to and / or stored in the simple audio decoder 16 that generates the multi-channel audio signal 17 (20). Preferably, the data size of the encoded representation 15 is minimized while minimizing the difference between the multi-channel audio signal 13 and the multi-channel audio signal 17. Furthermore, the difference between the multi-channel audio signal 13 and the multi-channel audio signal 17 can be measured according to a similarity that would be perceived by a human listener. The measure of human-perceived similarity between audio signal 13 and audio signal 17 is based on a reference playback method (i.e., a hypothetical default method by which the audio channels of multi-channel audio signals 13 and 17 are presented to the listener as an auditory experience).

[0025]

[0032] The efficiency of the simple audio encoder 14 and decoder 16 may also be defined in terms of the data rate (measured in bits per second) of the encoded representation 15 required to provide the multi-channel audio signal 17, which will be determined by the listener to match the multi-channel audio signal 13 to a particular perceived quality level. The simple audio encoder 14 and decoder 16 can achieve higher efficiency (i.e., a lower data rate) when it is known that the multi-channel audio signal 13 has certain attributes. In particular, higher efficiency may be achieved when it is known that the multi-channel audio signal 13 has the following attributes (DD1 and DD2):

[0033] DD1: In a multi-channel audio signal, one or more channels are generally dominant over the others, where the more dominant audio channel is the channel that contains the substantial elements of most (or all) of the sonic elements in the scene. That is, when presented to the listener as a single audio channel, the dominant audio signal will contain most (or all) of the sonic elements of the multi-channel signal when the multi-channel audio signal is presented to the listener by the reference playback method.

[0026]

[0034] DD2: Each audio channel in a multi-channel audio signal is primarily uncorrelated with each of the other audio channels.

[0027]

[0035] Given the knowledge that the multi-channel audio signal 13 has attributes DD1 and DD2, the simple audio encoder 14 can achieve improved efficiency by using several techniques, including, but not limited to, choosing to: allocate fewer bits to encode the less dominant channels, or discard the less dominant channels entirely. The simple audio encoder 14 can provide data to the simple audio decoder 16 to assist in the playback of channels discarded by the simple audio encoder 14. Preferably, a multi-channel audio signal without attributes DD1 and DD2 can be processed by an encoder pre-mixer to form a multi-channel audio signal with attributes DD1 and DD2, for example, by calculation, determination, construction, or generation, which will be further explained in relation to Figure 2. The corresponding decoder post-mixer is applied to the output of the simple decoder to form the output multi-channel audio signal, and as a result, the decoder post-mixer performs an operation approximately inverse to that of the encoder pre-mixer.

[0028]

[0036] Figure 2 is a block diagram of an audio codec system 100, including an audio encoder 104 and an audio decoder 106, an encoder pre-mixer 102, and a decoder post-mixer 108. The audio encoder 104 and the audio decoder 106 form a multi-channel audio signal 109(X'), which is a copy of the multi-channel audio signal 101(X). Preferably, the data size of the encoded representation 105 is minimized while minimizing the difference between the multi-channel audio signals 101 and 109. Furthermore, the difference between the multi-channel audio signals 101 and 109 can be measured according to the similarity perceived by a human listener.

[0029]

[0037] The measure of human-perceived similarity between multi-channel audio signal 101 and multi-channel audio signal 109 is based on a reference playback method (i.e., a assumed default means by which the audio channels of audio signals 101 and 109 are presented to the listener as an auditory experience). The efficiency of the multi-audio encoder 104 and multi-channel audio decoder 106 may be defined in terms of the data rate (measured in bits per second) of the encoded representation 105 that provides the multi-channel audio signal 109, which will be determined by the listener to match the multi-channel audio signal 101 to a particular perceived quality level.

[0030]

[0038] Referring to Figure 2, the input multi-channel audio signal 101 is mixed according to the encoder pre-mixer 102(R) to generate the output multi-channel audio signal 103(Z), the output multi-channel audio signal 103(Z) is processed by the simple audio encoder 104 to generate the encoded representation 105, which can be transmitted to and / or stored in the simple audio decoder 106 to generate the multi-channel audio signal 107(Z'). The multi-channel audio signal 107 is processed by the decoder post-mixer 108(R') to generate the decoded multi-channel audio signal 109. The encoder pre-mixer 102 provides metadata 112(Q) containing information necessary to determine the behavior of the decoder post-mixer 108. The metadata 112 can be stored and / or transmitted together with the encoded representation 105. The efficiency measurements of the multi-channel audio encoder 104 and the multi-channel audio decoder 106 may include the size of the metadata 112 (typically measured in bits per second), as will be understood by those skilled in the art.

[0031]

[0039] The multi-channel audio signal 101 may consist of N audio channels, in which case there may be significant correlations between some pairs of channels, in which case no single channel may be considered dominant. That is, the multi-channel audio signal 101 may not have attributes DD1 and DD2, and therefore the multi-channel audio signal 101 may not be suitable for encoding and decoding using the simple audio encoder 104 and decoder 106, respectively.

[0032]

[0040] Preferably, the encoder pre-mixer 102 is configured to process an input multi-channel audio signal 101 to generate an output multi-channel audio signal 103, where the output multi-channel audio signal 103 has attributes DD1 and DD2. Consider that the input multi-channel audio signal X consists of N channels:

[0033]

number

[0034]

number

[0041] The coefficients of the encoder-premixer matrix R may change over time, and therefore, R can be considered a function of time. The values ​​of the elements of R may be calculated at regular intervals (for example, the interval may be 20 ms, or a value between 1 ms and 100 ms) or at irregular intervals. If the values ​​of the elements of R change, the change may be smoothly interpolated. In the following discussion, references to R should be treated as references to the time-varying encoder-premixer R(t), and references to R' should be treated as references to the time-varying decoder-premixer R'(t).

[0035]

[0042] In this embodiment, the encoder pre-mixer 102 processes the components of the audio signal within band b using a mixing coefficient R b (t) can be used, where 1 ≤ b ≤ B. Figure 4 shows the configuration of the processing element 150, which processes the multi-channel audio signal 151(X) into B subband signals X by the filter bank 152. [1] (t),X [2] (t),... X [B] (t) is divided into each subband signal (e.g., 153(X) [1](t)) is processed by a mixing matrix (e.g., 154(R1)) to generate a remixed sub - band signal (e.g., 155(Z [1] (t))). The remixed sub - band signals Z [1] (t), Z [2] (t),..., Z [B] (t) are recombined by a combiner 156 to form a multi - channel audio signal 157.

[0036]

[0043] For the purposes of the following discussion, references to the matrix R(t) can be interpreted as references to R b (t), where b indicates a sub - band. It will be understood that the following description may apply to signals processed in sub - bands or signals processed without sub - band processing. It will be understood by those skilled in the art that many methods may be used to process audio signals according to sub - bands and that the discussion of the matrix R applies to these methods.

[0037]

[0044] Referring to FIG. 2, R mixes the channels of the multi - channel audio signal 101 to generate a multi - channel audio signal 103 having the attributes DD1 and DD2 as described above. Thus, the encoder 106 can achieve improved data efficiency. The decoder post - mixer 108(R’) provides a mixing operation that is the inverse of the mixer R as follows:

[0038]

Number

[0045] FIG. 3 shows the encoder pre - mixer 102(R) of FIG. 2 or the encoder pre - mixer R of FIG. 4 bThis is a block diagram of configuration 200 of two mixing processes intended to achieve the function of [the function shown]. An N-channel multi-channel input signal 201(X) is mixed by matrix 202(M) to generate an N-channel intermediate signal 203(Y), which is then processed by mixer 204(P) to generate an N-channel signal 205(Z). Signals 201(X) and 205(Z) in Figure 3 are used to combine with input signals 101(X) and 103(Z) in Figure 2, respectively, or with sub-band signal 153(X) in Figure 4. b (t)) and 155(Z b It is intended to correspond to (t)).

[0039]

[0046] The analysis block 210(A) receives input from signal 201 and calculates coefficients 212 used to adapt the operation of mixer 204. The analysis block 210 also generates metadata 211(Q) corresponding to metadata 112 in Figure 2, which is provided to the decoder as 113(Q) which will be used by the decoder-post-mixer 108.

[0040]

[0047] From the arrangement of mixers 202 and 204 in Figure 3, it can be understood that the matrix R is as follows:

[0041]

number

[0042]

[0048] Therefore, it follows:

[0043]

number

[0049] Matrix M is adapted to ensure that the intermediate signal 203(Y) has attribute DD1. That is, the N-channel signal 203(Y) includes one channel which may be considered the dominant channel. Without loss of generality, matrix M is adapted to ensure that the first channel Y1(t) is the dominant channel. Hereafter, if the first channel of a multi-channel signal is the dominant channel, this first channel will be referred to as the primary channel. The primary channel may also be referred to as the "eigen channel" in some contexts.

[0044]

[0050] The [N×N] matrix M can be determined from the [N×N] expected covariance matrix Cov of the N-channel input signal X(t):

[0045]

number

[0046]

[0051] As used in formula

[10] , the expected value may be estimated based on assumed characteristics of a typical input multi-channel audio signal, or it may be estimated by statistical analysis of a typical set of input multi-channel audio signals.

[0047]

[0052] The covariance matrix Cov can be factorized according to eigenvalue analysis, as is well known to those skilled in the art:

[0048]

number

[0049]

[0053] The matrix M can be chosen as follows:

[0050]

number

[0054] Those skilled in the art will understand that the covariance matrix Cov depends on the panning method used to construct the original input signal X(t), and the typical use of the panning method used by the author of a typical signal.

[0051]

[0055] For example, if the original input signal is a two-channel stereo signal intended for playback on stereo speakers, a typical panning rule used by the content creator would result in some audio objects being panned to the first channel (in this context, often referred to as the left channel), some audio objects being panned to the second channel (in this context, often referred to as the right channel), and some objects being panned to both channels simultaneously. In this case, the covariance matrix may be as follows:

[0052]

number

[12] and

[13] , the following holds:

[0053]

number

[0056] The matrix M in equation

[15] is familiar to those skilled in the art as a mixing matrix suitable for converting an input audio signal X in L / R stereo format to an intermediate signal Z that would be in Mid / Side format. The first channel of Z (often referred to in this case as the Mid signal) is the dominant audio signal (primary channel), which has the property that most of the audio elements in the stereo mix reside in the Mid signal.

[0054]

[0057] As another example, if the original input signal is a 5-channel surround signal intended for playback in a typical 5-speaker setup, a typical panning rule used by the content creator would result in some audio objects being panned to one of the 5 channels, while others are panned to two or more channels simultaneously. In this case, the covariance matrix may be as follows:

[0055]

number

[12] and

[13] , the following holds:

[0056]

number

[0058] It will be understood that the top row of matrix M in equation

[17] is formed of similar (or identical) positive values. This means that, according to equation [6], the first channel of the intermediate signal Y is formed by the sum of the five channels of the original input audio signal X(t), which ensures that all sonic elements panned to the original input audio signal are present in Y1(t) (the first channel of the N-channel signal Y(t)). Thus, this choice of matrix M ensures that the intermediate signal Y has attribute DD1 (that Y1(t) is the primary channel).

[0057]

[0059] In yet another example, if the input multi-channel audio signal X(t) already contains a dominant channel (without loss of generality, it is assumed that the first channel X1(t) is dominant), the matrix M may be an [N×N] identity matrix. In a more specific example of an input multi-channel audio signal having a dominant / primary first channel, the input multi-channel audio signal can represent an acoustic scene encoded in ambisonic format (a means of encoding an acoustic scene well known to those skilled in the art).

[0058]

[0060] The matrix 212(P(t)) is calculated at time t by the analysis block 210(A) in Figure 3 according to the following process: 1. Determine the covariance of the intermediate signal Y(t) at time t. An example of a method for calculating the covariance is as follows:

[0059]

number

[0061] Alternatively, the covariance of the intermediate signal Y(t) may be calculated from the covariance of the input multi-channel audio signal X(t) as follows:

[0060]

number

[0061]

number

[0062]

Mathematics

[0062] 4. Solve equation

[25] under the quantities w, α, β to determine the input mixing strength coefficient h and the predicted mixing strength coefficient g:

[0063]

Mathematics

[0064]

Mathematics

[0065]

[0063] When the prior prediction constraint PPC1 is used, equation

[25] can be transformed as follows:

[0066]

Mathematics

[27] can be solved with respect to the maximum real value of g, and thus the value of h can be determined using equation

[26] .

[0067] 5. Form the [L×L] matrix Q as follows:

[0068]

Mathematics

[0069]

Mathematics

[0070]

[0064] The metadata 211(Q) in Figure 3 can transmit information that enables the unit vector u and coefficients g,h to be determined by the decoder-post-mixer 113 in Figure 2.

[0071]

[0065] The solution for g in equation

[27] can be approximated by selecting an initial estimate g1=1 and performing a number of iterations (according to Newton's method as known in the art):

[0072]

number

[27] .

[0073]

[0066] According to an alternative embodiment, it is possible to determine the [L×L] matrix P(t) at time t by determining an [N×1] vector u that shows the correlation between the primary channel of the intermediate signal Y(t) and the remaining N non-primary channels, determining the input mixing intensity coefficient h and the predicted mixing intensity coefficient g, and forming P(t) according to equation

[28] , so that the signal Z(t)=P(t)×Y(t) has attributes DD1 and DD2.

[0074]

[0067] The determination of the coefficients g and h may be governed by the prior prediction constraint equations. An example of the prior prediction constraint equations is given in equation

[26] (PPC1). A preferred choice for the coefficient f is that f = 0.5, but values ​​of f in the range 0.2 ≤ f ≤ 1 may be suitable for use.

[0075]

[0068] In an alternative embodiment, the following prior prediction constraints may be used:

[0076]

number

[0077]

[0069] According to the constraint PPC2 in equation

[31] , the solution to equation

[25] is as follows:

[0078]

number

[0070] Figure 5 is a block diagram of the predictive mixer 300 according to one embodiment. The matrix term (I) of equation

[29] L -gQ) and (I L +hQ H The signal may also be implemented by a prediction mixer 300, in which case the signal Y(t) consists of 4 channels (L=4), the first channel 301 (Y1) being the primary channel, and the remaining three non-primary channels 302 (e.g., Y2, Y3, Y4) being scaled according to three input gains 312 (H2, H3, H4) to form a scaled input signal component (e.g., 304). The scaled input signal component is added with the primary input channel 301 (Y1) to form the primary output 306 (Z1). The primary output 306 is scaled by three prediction gains 313 (G2, G3, G4) to form three prediction signals (e.g., 311). Each prediction signal is subtracted from its respective input (e.g., Y2302) (e.g., 308 and 309) to form its respective non-dominant output 310 (Z2).

[0079]

[0071] The three input gains 312 (H2, H3, H4) can be determined from the mixing coefficient u (determined according to equation

[23] ) and the input mixing intensity coefficient h (for each solution to equation

[25] ) as follows:

[0080]

number

[0072] The three predicted gains 313 (G2, G3, G4) can be determined from the mixing coefficient u (determined according to equation

[23] ) and the predicted mixing intensity coefficient g (for each solution to equation

[25] ) as follows:

[0081]

number

[0073] Those skilled in the art will understand that the linear matrix operations M 202 and P 204 in Figure 4 may be performed using a single matrix R = P × M.

[0082]

[0074] Those skilled in the art will understand that the decoder matrix R' in Figure 2 may be formed from matrices M' (inverse of M) and P' (inverse of P):

[0083]

number

[0084]

number

[0075] Figure 6 shows the arrangement of processing elements 400 that implement the decoder-post-mixer 108 of Figure 2. Metadata 402(Q) provides information to the inverse predictive decision block 403(B) which calculates coefficients necessary to determine the operation of the inverse predictive decision block 405(P'). Signal 401(Z') is processed by the inverse predictor 405(P') to generate an intermediate signal 406(Y'), which is then processed by the matrix 407(M') to generate an output signal 408.

[0076] Exemplary process Figure 7 is a flowchart of process 700 for adaptive downmixing of an audio signal with improved continuity, according to one embodiment. Process 700 can be implemented, for example, by system 800 shown in Figure 8.

[0085]

[0077] Process 700 is: to receive an input multi-channel audio signal including a primary input audio channel and L non-primary input audio channels (701); to determine a set of L input gains (where L is a positive integer greater than 1) (702); to form individual scaled non-primary audio channels for each of the L non-primary input audio channels and L input gains from individual non-primary input audio channels scaled according to the input gain (703); to form a primary output audio channel from the sum of the primary input audio channel and the scaled non-primary input audio channels (70 4) The steps include determining a set of L predictive gains (705); forming a predictive channel from a primary output audio channel scaled according to each of the L predictive gains (706); forming L non-primary output audio channels from the difference between each non-primary input audio channel and each predictive signal (706); forming an output multi-channel audio signal from the primary output audio channel and the L non-primary output audio channels (707); encoding the output multi-channel audio signal (708); and transmitting or storing the encoded output multi-channel audio signal (709). Each of these steps is described more fully in relation to Figure 1-6.

[0086]

[0078] Exemplary System Architecture Figure 8 shows a block diagram of an exemplary system 800 according to an embodiment that implements the features and processes described in relation to Figures 1-7. System 800 includes any device capable of playing audio, including but not limited to smartphones, tablet computers, wearable computers, vehicle computers, game consoles, surround sound systems, and kiosks.

[0087]

[0079] As shown in the diagram, the system 800 is, for example, read-only. The system includes a central processing unit (CPU) 801 capable of executing various processes according to programs stored in memory 802, or programs loaded, for example, from memory unit 808 into random access memory (RAM) 803. RAM 803 stores data required by the CPU 801 when executing various processes, as needed. The CPU 801, ROM 802, and RAM 803 are connected to each other via bus 804. An input / output (I / O) interface 805 is also connected to bus 804.

[0088]

[0080] The following components are connected to the I / O interface 805: an input unit 806 which may include a keyboard, mouse, etc.; an output unit 807 which may include a display such as a liquid crystal display (LCD) and one or more speakers; a storage unit 808 which includes a hard disk or another suitable storage device; and a communication unit 809 which includes a network interface card such as a network card (e.g., wireless or wired).

[0089]

[0081] In some implementations, the input unit 806 includes one or more microphones at various locations (depending on the host device) so that it can capture audio signals in various formats (e.g., mono, stereo, spatial, immersive, and other appropriate formats).

[0090]

[0082] In some implementations, the output unit 807 includes a system that uses a variety of speaker configurations. As shown in Figure 8, the output unit 807 can render audio signals in various formats (e.g., mono, stereo, immersive, binaural, and other appropriate formats) (depending on the capabilities of the host device).

[0091]

[0083] The communication unit 809 is configured to communicate with other devices (for example, via a network). The drive 810 is also connected to the I / O interface 805, if necessary. A removable medium 811, such as a magnetic disk, optical disk, magneto-optical disk, flash drive, or other suitable removable medium, is mounted on the drive 810, and a computer program to be read from there is installed in the storage unit 808, if necessary. Those skilled in the art will understand that although the system 800 is described as including the components described above, in actual applications it is possible to add, remove, and / or replace some of these components, and all such modifications or changes are within the scope of the present disclosure.

[0092]

[0084] The embodiments of the systems described herein can be implemented in a suitable computer-based audio processing network environment for processing digital or digitized audio files. Part of the adaptive audio system may include one or more networks, each containing any desired number of individual machines, each containing one or more routers (not shown) that function to buffer and route data transmitted between computers. Such networks may be built on a variety of different network protocols, and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0093]

[0085] According to exemplary embodiments of the present disclosure, the processes described above can be carried out as a computer software program or on a computer-readable storage medium. For example, embodiments of the present disclosure include a computer program product which includes a computer program substantially embedded on a machine-readable medium and which includes program code for performing the method. In such embodiments, the computer program may be downloaded and implemented from a network via a communication unit 809 as shown in Figure 8 and / or installed from a removable medium 811.

[0094]

[0086] In general, various exemplary embodiments of the present disclosure can be implemented in hardware or special-purpose circuits (e.g., control circuits), software, logic, or any combination thereof. For example, the above-described unit can be implemented by a control circuit (e.g., a CPU combined with other components in Figure 8), and thus the control circuit can perform the operations described in the present disclosure. Some embodiments can be implemented in hardware, while others can be implemented in firmware or software that can be implemented by a controller, microprocessor, or other computing device (e.g., a control circuit). Various embodiments of the exemplary embodiments of the present disclosure are illustrated and described as block diagrams, flowcharts, or any other graphic representations, but it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, special-purpose circuits or logic, general-purpose hardware or controllers, or other computing devices, or any combination thereof, as non-limiting examples.

[0095]

[0087] Furthermore, the various blocks shown in the flowchart can be viewed as a plurality of coupled logic circuit elements configured to perform method steps and / or operations resulting from the operation of computer program code and / or related functions. For example, embodiments of the present disclosure include a computer program product which includes a computer program materialized on a machine-readable medium, the computer program which includes program code configured to perform the above method.

[0096]

[0088] In the context of this disclosure, a machine-readable medium may be any tangible medium that contains or can store a program used by or associated with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may be non-transient and may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a suitable combination thereof. More specific examples of machine-readable storage media include electrical connections containing one or more wires, portable computer diskettes, hard disks, random-access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0097]

[0089] Computer program code for performing the methods of the present disclosure can be written in any combination of one or more programming languages. These computer program codes can be provided to the processor of a general-purpose computer, a dedicated computer, or other programmable data processing device having a control circuit, so that when the program code is executed by the computer's processor or other programmable data processing device, it causes the functions / operations shown in the flowcharts and / or block diagrams to be performed. The program code can be executed entirely on a computer, partially on a computer, as a standalone software package, partially on a computer and partially on a remote computer, entirely on a remote computer or server, or distributed across one or more remote computers and / or servers.

[0098]

[0090] This specification includes many specific implementation details, but these should not be construed as limitations on the scope that may be claimed, but rather as descriptions of features that may be specific to a particular embodiment. Certain features described herein in the context of separate embodiments may be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may be implemented separately or in some appropriate sub-combination in multiple embodiments. Furthermore, features are described above as acting in a particular combination, and may even be initially claimed as such, but one or more features of the claimed combination may, in some cases, be extracted from that combination, and the claimed combination may relate to a sub-combination or a variation of a sub-combination. The logical flow shown in the figures does not require a specific order or sequence shown in the figures to achieve the desired result. Furthermore, other steps may be provided, or steps may be removed from the described flow, and other components may be added to or removed from the described system. Thus, other implementations are also within the scope of the following claims.

[0099] The following is an illustrative list of the problem-solving methods included in this matter. (Note 1) Audio encoding method: A step in which at least one processor receives an input multi-channel audio signal comprising a primary input audio channel and L non-primary input audio channels; The step of at least one processor determining a set of L input gains, where L is a positive integer greater than 1; For each of the L non-primary input audio channels and L input gains, the step of forming individual scaled non-primary audio channels from individual non-primary input audio channels scaled according to the input gains; A step of forming a primary output audio channel from the sum of the primary input audio channel and the scaled non-primary input audio channels; The step of the at least one processor determining a set of L predicted gains; For each of the L predicted gains, the at least one processor forms a predicted channel from the primary output audio channel scaled according to the predicted gain; The step of the at least one processor forming L non-primary output audio channels from the difference between each non-primary input audio channel and each predicted signal; The step of the at least one processor forming an output multi-channel audio signal from the primary output audio channel and the L non-primary output audio channels; The steps include: an audio encoder encoding the output multi-channel audio signal; and The step of the at least one processor transmitting or storing its encoded output multi-channel audio signal; A method that includes this. (Note 2) In the method described in Appendix 1, the step of determining the set of L input gains is: Steps to determine L sets of mixing coefficients; A step of determining the input mixture strength coefficient; and A step of determining the L input gains by scaling the L mixing coefficients by the input mixing intensity coefficient; A method that includes this. (Note 3) In the method described in Appendix 2, the step of determining the set of L predicted gains is: Steps to determine L sets of mixing coefficients; Steps to determine the predicted mixed intensity coefficient; and A step of determining the L predicted gains by scaling the L mixing coefficients by the predicted mixing intensity coefficient; A method that includes this. (Note 4) The method described in Appendix 3, wherein the input mixing intensity coefficient h is determined by the prior prediction constraint equation h=fg, where f is a predetermined constant value greater than 0 and less than or equal to 1, and g is the predicted mixing intensity coefficient. (Note 5) In the method described in Appendix 4, the predicted mixing intensity coefficient g is:

number

number

Claims

1. Audio encoding method: A step in which at least one processor receives an input multi-channel audio signal including a primary input audio channel and L non-primary input audio channels; The step of at least one processor determining a set of L input gains, where L is a positive integer greater than 1, and the set of L input gains is determined by scaling a set of L mixing coefficients by an input mixing intensity coefficient; For each of the L input gains and the L non-primary input audio channels, the at least one processor forms individual scaled non-primary input audio channels from individual non-primary input audio channels scaled according to the input gains; The step of at least one processor forming a primary output audio channel from the sum of the primary input audio channel and the scaled non-primary input audio channels; The step of the at least one processor determining a set of L predictive gains, wherein the set of L predictive gains is determined by scaling the set of L mixing coefficients by a predictive mixing intensity coefficient; For each of the L predicted gains, the at least one processor forms a predicted channel from the primary output audio channel scaled according to the predicted gain; The step of the at least one processor forming L non-primary output audio channels from the difference between each non-primary input audio channel and each predicted channel; The steps include: the at least one processor forming an output multi-channel audio signal from the primary output audio channel and the L non-primary output audio channels; The steps of the at least one processor generating metadata corresponding to the input multi-channel audio signal; and The audio encoder encodes the output multi-channel audio signal and the metadata; A method that includes this.

2. One or more computer processors; and A non-temporary, computer-readable storage medium for storing instructions; A system including, wherein, when the instruction is executed by the one or more computer processors, it causes the one or more computer processors to perform an operation, the operation being: A step in which at least one processor receives an input multi-channel audio signal including a primary input audio channel and L non-primary input audio channels; The step of at least one processor determining a set of L input gains, where L is a positive integer greater than 1, and the set of L input gains is determined by scaling a set of L mixing coefficients by an input mixing intensity coefficient; For each of the L input gains and the L non-primary input audio channels, the at least one processor forms individual scaled non-primary input audio channels from individual non-primary input audio channels scaled according to the input gains; The step of at least one processor forming a primary output audio channel from the sum of the primary input audio channel and the scaled non-primary input audio channels; The step of the at least one processor determining a set of L predictive gains, wherein the set of L predictive gains is determined by scaling the set of L mixing coefficients by a predictive mixing intensity coefficient; For each of the L predicted gains, the at least one processor forms a predicted channel from the primary output audio channel scaled according to the predicted gain; The step of the at least one processor forming L non-primary output audio channels from the difference between each non-primary input audio channel and each predicted channel; The steps include: the at least one processor forming an output multi-channel audio signal from the primary output audio channel and the L non-primary output audio channels; The steps of the at least one processor generating metadata corresponding to the input multi-channel audio signal; and The audio encoder encodes the output multi-channel audio signal and the metadata; A system that includes this.

3. A non-temporary, computer-readable storage medium for storing instructions, wherein, when executed by one or more computer processors, the instructions cause the one or more computer processors to perform an action, the action being: A step in which at least one processor receives an input multi-channel audio signal including a primary input audio channel and L non-primary input audio channels; The step of at least one processor determining a set of L input gains, where L is a positive integer greater than 1, and the set of L input gains is determined by scaling a set of L mixing coefficients by an input mixing intensity coefficient; For each of the L input gains and the L non-primary input audio channels, the at least one processor forms individual scaled non-primary input audio channels from individual non-primary input audio channels scaled according to the input gains; The step of at least one processor forming a primary output audio channel from the sum of the primary input audio channel and the scaled non-primary input audio channels; The step of the at least one processor determining a set of L predictive gains, wherein the set of L predictive gains is determined by scaling the set of L mixing coefficients by a predictive mixing intensity coefficient; For each of the L predicted gains, the at least one processor forms a predicted channel from the primary output audio channel scaled according to the predicted gain; The step of the at least one processor forming L non-primary output audio channels from the difference between each non-primary input audio channel and each predicted channel; The steps include: the at least one processor forming an output multi-channel audio signal from the primary output audio channel and the L non-primary output audio channels; The steps of the at least one processor generating metadata corresponding to the input multi-channel audio signal; and The audio encoder encodes the output multi-channel audio signal and the metadata; A storage medium that includes this.

Images