Encoding of a down-mixed multi-channel audio signal comprising a primary input channel and two or more scaled non-primary input channels

Abstract

The invention relates to encoding of downmixed multi-channel audio signals comprising a primary input channel and two or more scaled non-primary input channels. Systems, methods, and computer program products for adaptive downmixing of audio signals with improved continuity are disclosed. An audio encoding system receives an input multi-channel audio signal comprising a primary input audio channel and L non-primary input audio channels. The system determines a set of L input gains. For each channel and gain, the system forms a corresponding scaled non-primary input audio channel. The system forms a primary output audio channel from a sum of the primary input audio channel and the scaled non-primary input audio channels. The system determines a set of L prediction gains. The system forms a prediction channel from the primary output audio channel. The system forms L non-primary output audio channels. The system forms an output multi-channel audio signal from the primary output audio channel and the L non-primary output audio channels.

Description

[0001] This application is a divisional application of patent application No. 202180055244.8, filed on June 10, 2021, entitled "Encoding of multichannel audio signals including a primary input channel and two or more scaled non-primary input channels in a downmixed manner".

[0002] Cross-references to related applications

[0003] This application claims priority to U.S. Provisional Patent Application No. 63 / 037,635, filed June 11, 2020, and U.S. Provisional Patent Application No. 63 / 193,926, filed May 27, 2021, each of which is incorporated herein by reference in its entirety. Technical Field

[0004] This disclosure relates generally to audio coding, and particularly to the coding of multichannel audio signals. Background Technology

[0005] When storing or transmitting input audio signals for later use (e.g., playback to listeners), it is generally desirable to encode the audio signals with a reduced amount of data. The data reduction process applied to the input audio signal is commonly referred to as "audio encoding" (or "encoding"), and the device used for encoding is commonly referred to as an "audio encoder" (or "encoder"). The process of regenerating the output audio signal from the reduced data is commonly referred to as "audio decoding" (or "decoding"), and the device used for decoding is commonly referred to as an "audio decoder" (or "decoder"). Audio encoders and decoders can be adapted to operate on input signals consisting of a single audio channel or multiple audio channels. When the input signal consists of multiple audio channels, the audio encoder and audio decoder are respectively called a multichannel audio encoder and a multichannel audio decoder. Summary of the Invention

[0006] An implementation of adaptive downmixing for audio signals with improved continuity is disclosed.

[0007] In some embodiments, an audio encoding method includes: receiving an input multichannel audio signal comprising a primary input audio channel and L non-primary input audio channels using at least one processor; determining a set of L input gains using the at least one processor, where L is a positive integer greater than 1; for each of the L non-primary input audio channels and the L input gains, forming a corresponding scaled non-primary input audio channel from the corresponding non-primary input audio channel scaled according to the input gains; forming a primary output audio channel from the sum of the primary input audio channel and the scaled non-primary input audio channels; and using the at least one processor... The processor determines a set of L prediction gains; for each of the L prediction gains, the at least one processor forms a prediction channel from a primary output audio channel scaled according to the prediction gain; the at least one processor forms L non-primary output audio channels from the difference between the corresponding non-primary input audio channels and the corresponding prediction signals; the at least one processor forms an output multichannel audio signal from the primary output audio channels and the L non-primary output audio channels; the at least one processor encodes the output multichannel audio signal; and the at least one processor transmits or stores the encoded output multichannel audio signal.

[0008] In some embodiments, determining the set of L input gains includes: determining a set of L mixing coefficients; determining input mixing intensity coefficients; and determining the L input gains by scaling the L mixing coefficients according to the input mixing intensity coefficients.

[0009] In some embodiments, determining the set of L prediction gains includes: determining a set of L mixing coefficients; determining prediction mixing intensity coefficients; and determining the L prediction gains by scaling the L mixing coefficients according to the prediction mixing intensity coefficients.

[0010] In some embodiments, the input mixing intensity coefficient h is determined by the pre-prediction constraint equation h=fg, where f is a predetermined constant value greater than zero and less than or equal to 1, and g is the predicted mixing intensity coefficient.

[0011] In some embodiments, the predicted mixing intensity coefficient g is the maximum real-valued solution of the following equation: ,in , , Furthermore, the quantity w, column vector v, and matrix E are components of the covariance matrix of the intermediate signal with the dominant channel.

[0012] In some embodiments, the covariance matrix of the intermediate signal is calculated from the covariance matrix of the multi-channel input audio signal.

[0013] In some embodiments, two or more input multichannel audio channels are processed according to a mixing matrix to produce the primary input audio channel and the L non-primary input audio channels.

[0014] In some embodiments, the primary input audio channel is determined by the dominant eigenvector of the expected covariance of a typical input multichannel audio signal.

[0015] In some embodiments, each of the L mixing coefficients is determined based on the correlation between a corresponding one of the non-primary input audio channels and the primary input audio channel.

[0016] In some embodiments, the encoding includes allocating more bits to the primary output audio channel than to the L non-primary output audio channels, or discarding one or more of the L non-primary output audio channels.

[0017] Other implementations disclosed herein relate to a system, apparatus, and computer-readable medium. Details of the disclosed implementations are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will become apparent from the specification, drawings, and claims.

[0018] The specific implementation disclosed herein offers one or more of the following advantages. The input multichannel audio signal is processed by an audio encoder premixer to form an output multichannel audio signal with two desirable properties for efficient encoding. The first property is that at least one dominant audio channel of the output multichannel audio signal contains most or all of the sound elements of the input multichannel audio signal. The second property is that each audio channel of the output multichannel audio signal is largely uncorrelated with every other audio channel. A simple encoder can provide data to a simple decoder to help regenerate audio channels discarded by the simple encoder.

[0019] The two properties mentioned above allow a simple encoder to efficiently encode the output multichannel audio signal by allocating fewer bits to the encoding of the less dominant channel or by choosing to discard the less dominant audio channel entirely. Attached Figure Description

[0020] In the accompanying drawings, for ease of description, a specific arrangement or order of schematic elements is shown, such as those representing devices, units, instruction blocks, and data elements. However, those skilled in the art should understand that the specific order or arrangement of schematic elements in the drawings does not imply a need for a particular processing order or sequence, or separation of processes. Furthermore, the inclusion of schematic elements in the drawings does not imply that such elements are required in all embodiments, or that the features represented by such elements may not be included in or combined with other elements in some implementations.

[0021] Furthermore, in drawings that use connecting elements such as solid or dashed lines or arrows to illustrate connections, relationships, or associations between or among two or more other schematic elements, the absence of any such connecting element does not imply the absence of any connection, relationship, or association. In other words, some connections, relationships, or associations between elements are not shown in the drawings so as not to obscure this disclosure. Additionally, for ease of illustration, a single connecting element is used to represent multiple connections, relationships, or associations between elements. For example, in the case where a connecting element represents communication of signals, data, or instructions, those skilled in the art will understand that such an element represents one or more signal paths as needed to influence communication.

[0022] Figure 1 This is a block diagram of an arrangement of a simple audio encoder and a simple audio decoder, according to some embodiments, intended to form an output multi-channel audio signal that is a facsimile of the input multi-channel audio signal.

[0023] Figure 2 This is a block diagram of an audio codec system according to some embodiments, the audio codec system including an audio encoder, an audio decoder 106, an encoder premixer, and a decoder postmixer.

[0024] Figure 3 An arrangement of processing elements according to some embodiments is shown, wherein an input multichannel audio signal is split into sub-band signals by a filter bank, wherein each sub-band is processed by a mixing matrix to produce a remixed sub-band signal.

[0025] Figure 4 It is intended to achieve according to some embodiments Figure 2 encoder premixer or Figure 3 A block diagram showing the arrangement of the two mixing operations of the encoder premixer.

[0026] Figure 5 This is a block diagram of a predictive mixer according to some embodiments.

[0027] Figure 6 Implementations according to some embodiments are shown. Figure 2 The arrangement of processing elements in the decoder and mixer.

[0028] Figure 7 This is a flowchart of an adaptive downmixing process for audio signals with improved continuity, according to some embodiments.

[0029] Figure 8 Reference is provided for implementation based on some embodiments. Figure 1-7 A block diagram of the system describing the features and processes.

[0030] The same reference numerals used in various figures denote the same elements. Detailed Implementation

[0031] In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the various described embodiments. It will be appreciated by those skilled in the art that various described implementations can be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail to avoid unnecessarily obscuring aspects of the embodiments. Several features are described below, each of which can be used independently of each other or in any combination with other features.

[0032] Nomenclature

[0033] As used herein, the term “comprising” and its variations shall be understood as open-ended terms meaning “including but not limited to”. Unless the context clearly indicates otherwise, the term “or” shall be understood as “and / or”. The term “based on” shall be understood as “at least partially based on”. The terms “one example implementation” and “example implementation” shall be understood as “at least one example implementation”. The term “another implementation” shall be understood as “at least one other implementation”. The term “determine” shall be understood as obtaining, receiving, calculating, operating, estimating, predicting, or deriving. Furthermore, in the following description and claims, unless otherwise defined, all technical and scientific terms used herein shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0034] Figure 1 This is a block diagram of an arrangement 10 of a simple audio encoder and a simple audio decoder, designed to form a multi-channel audio signal 17 (Z'), which is a copy of the multi-channel audio signal 13 (Z). The multi-channel audio signal 13 is processed by the simple audio encoder 14 to produce an encoded representation 15, which can be stored and / or transmitted 20 to the simple audio decoder 16 that produces the multi-channel audio signal 17. Preferably, the data size of the encoded representation 15 is minimized while minimizing the difference between the multi-channel audio signal 13 and the multi-channel audio signal 17. Furthermore, the difference between the multi-channel audio signal 13 and the multi-channel audio signal 17 can be measured based on the similarity perceived by a human listener. The measurement of the human-perceived similarity between the audio signal 13 and the audio signal 17 is based on a reference playback method (i.e., the assumed default means by which the audio channels of the multi-channel audio signals 13, 17 are presented to the listener as an auditory experience).

[0035] The efficiency of the simple audio encoder 14 and decoder 16 can be defined by the data rate (measured in bits per second) of the encoded representation 15 required to provide the multi-channel audio signal 17, which the listener will judge to match the multi-channel audio signal 13 with a specific perceived quality level. The simple audio encoder 14 and decoder 16 can achieve higher efficiency (i.e., lower data rate) when the multi-channel audio signal 13 is known to have specific properties. Specifically, higher efficiency can be achieved when the multi-channel audio signal 13 is known to have the following properties (DD1 and DD2):

[0036] DD1: One or more channels of a multi-channel audio signal are typically more dominant than the other channels. The more dominant audio channel is the one that contains the substantial elements of the sound elements in the scene. In other words, when a multi-channel audio signal is presented to a listener using a reference playback method, the dominant audio signal, when presented to the listener as a single audio channel, will contain most (or all) of the sound elements of the multi-channel signal.

[0037] DD2: Each audio channel of a multi-channel audio signal is largely uncorrelated with each of the other audio channels.

[0038] Given that the multichannel audio signal 13 has attributes DD1 and DD2, the simple audio encoder 14 can use several techniques to achieve improved efficiency, including but not limited to: allocating fewer bits to the encoding of less dominant channels or selectively discarding less dominant channels entirely. The simple audio encoder 14 can provide data to the simple audio decoder 16 to help regenerate channels discarded by the simple audio encoder 14. Preferably, the multichannel audio signal without attributes DD1 and DD2 can be processed by an encoder premixer to form (e.g., calculate, determine, construct, or generate) a multichannel audio signal with attributes DD1 and DD2, as referenced. Figure 2 Further described. A corresponding decoder post-mixer can be applied to the output of a simple decoder to form an output multichannel audio signal, such that the decoder post-mixer performs an approximately inverse operation relative to the operation of the encoder pre-mixer.

[0039] Figure 2This is a block diagram of an audio codec system 100, which includes an audio encoder 104 and an audio decoder 106, an encoder premixer 102, and a decoder postmixer 108. The audio encoder 104 and the audio decoder 106 form a multi-channel audio signal 109 (X'), which is a copy of the multi-channel audio signal 101 (X). Preferably, the data size of the encoded representation 105 is minimized while minimizing the difference between the multi-channel audio signal 101 and the multi-channel audio signal 109. Furthermore, the difference between the multi-channel audio signal 101 and the multi-channel audio signal 109 can be measured based on the similarity perceived by a human listener.

[0040] The measurement of the human-perceived similarity between multi-channel audio signals 101 and 109 is based on a reference playback method (i.e., the assumed default means of presenting the audio channels of audio signals 101 and 109 to the listener as an auditory experience). The efficiency of the multi-channel audio encoder 104 and the multi-channel audio decoder 106 can be defined by the data rate (measured in bits per second) of the coded representation 105 providing the multi-channel audio signal 109, which the listener will judge to match the multi-channel audio signal 101 with a specific perceptual quality level.

[0041] refer to Figure 2 The input multichannel audio signal 101 is mixed by the encoder premixer 102 (R) to produce an output multichannel audio signal 103 (Z). This output multichannel audio signal 103 (Z) is processed by a simple audio encoder 104 to produce an encoded representation 105. This encoded representation 105 can be stored and / or transmitted 110 to a simple audio decoder 106, which produces a multichannel audio signal 107 (Z'). The multichannel audio signal 107 is processed by a decoder postmixer 108 (R') to produce a decoded multichannel audio signal 109. The encoder premixer 102 provides metadata 112 (Q), which includes information needed to determine the behavior of the decoder postmixer 108. The metadata 112 can be stored and / or transmitted 110 along with the encoded representation 105. As those skilled in the art will understand, measurements of the efficiency of the multi-channel audio encoder 104 and the multi-channel audio decoder 106 may include the size of the metadata 112 (typically measured in bits per second).

[0042] The multi-channel audio signal 101 may consist of N audio channels, where there may be significant correlations between some channel pairs, and no single channel can be considered the dominant channel. That is, the multi-channel audio signal 101 may not have attributes DD1 and DD2, and therefore the multi-channel audio signal 101 may not be a suitable signal for encoding and decoding using the simple audio encoder 104 and decoder 106 respectively.

[0043] Preferably, the encoder premixer 102 is adapted to process the input multichannel audio signal 101 to generate an output multichannel audio signal 103, wherein the output multichannel audio signal 103 has attributes DD1 and DD2. Given an input multichannel audio signal X consisting of N channels:

[0044] [1]

[0045] The output multi-channel audio signal Z is calculated as follows:

[0046] [2]

[0047] = R(t) x X(t). [3]

[0048] The coefficients of the encoder premixer matrix R can vary with time, so R can be considered a function of time. The values ​​of the elements of R can be calculated at regular intervals (e.g., where the interval can be 20 ms, or values ​​between 1 ms and 100 ms) or at irregular intervals. When the values ​​of the elements of R change, the change can be smoothly interpolated. In the following discussion, references to R should be considered as references to the time-varying encoder premixer R(t), and references to R' should be considered as references to the time-varying decoder premixer R'(t).

[0049] In this embodiment, the encoder premixer 102 may utilize the mixing coefficient. To process the audio signal components in frequency band b, where . Figure 4 The arrangement of processing elements 150 is shown, whereby the multi-channel audio signal 151(X) is divided into B sub-band signals by filter bank 152. Each sub-band signal (e.g., 153) The subband signal is processed by a mixing matrix (e.g., 154 (R1)) to produce a remixed subband signal (e.g., 155 (R1)). The remixed subband signal. The signals are recombined by combiner 156 to form multi-channel audio signals 157 (Z).

[0050] For the purposes of the discussion below, the reference to matrix R(t) can be interpreted as a reference to... The reference is used where b refers to the sub-band. It will be understood that the following discussion can be applied to signals processed in a sub-band, or signals processed without sub-band processing. Those skilled in the art will understand that many methods can be used to process audio signals according to sub-bands, and the discussion of matrix R will apply to these methods.

[0051] refer to Figure 2 R mixes the channels of the multi-channel audio signal 101 to generate a multi-channel audio signal 103 with attributes DD1 and DD2, as described above, thereby enabling the encoder 104 to achieve improved data efficiency. The decoder post-mixer 108 (R') provides a mixing operation as the inverse of mixer R, such that:

[0052] [4]

[0053] Figure 3 It is intended to achieve Figure 2 The encoder premixer 102(R) or Figure 4 encoder premixer The block diagram of the arrangement 200 for the two mixing operations of the function. The N-channel multi-channel input signal 201 (X) is mixed by the mixing matrix 202 (M) to produce the N-channel intermediate signal 203 (Y), which is then processed by the mixer 204 (P) to produce the N-channel signal 205 (Z). Figure 3 Signals 201 (X) and 205 (Z) are intended to correspond to respectively Figure 2 The input signals 101 (X) and 103 (Z) in the data, or corresponding to... Figure 4 Subband signal 153 ( ) and 155 ( ).

[0054] Analysis block 210(A) receives input from signal 201 and calculates coefficient 212, which will be used to adjust the operation of mixer 204. Analysis block 210 also generates corresponding... Figure 2 Metadata 211(Q) of metadata 112 will be provided to decoder 113(Q) for use by decoder post-mixer 108.

[0055] from Figure 4 The arrangement of mixers 202 and 204 will be understood, and matrix R will be:

[0056] [5]

[0057] Where the matrix It can change over time.

[0058] therefore:

[0059] [6]-[9]

[0060]

[0061] Matrix M is adapted to ensure that the intermediate signal 203(Y) has attribute DD1. That is, the N-channel signal 203(Y) contains one channel that can be considered the dominant channel. Without loss of generality, matrix M is adapted to ensure that the first channel... It is the dominant channel. In the following text, when the first channel of a multichannel signal is the dominant channel, that first channel will be referred to as the primary channel. In some contexts, the primary channel may also be referred to as the "intrinsic channel".

[0062] Based on N-channel input signal The [N × N] expected covariance matrix Cov is used to determine the [N × N] matrix M:

[0063]

[0064] in The operation represents a column vector of length N. The Hermitian transpose of , and the E() operation represents the expected value of a variable.

[0065] The expected value used in Equation

[10] can be estimated based on the assumed characteristics of a typical input multichannel audio signal, or it can be estimated by statistical analysis of a set of typical input multichannel audio signals.

[0066] The covariance matrix Cov can be factored using eigenvalue analysis, as is familiar to those skilled in the art:

[0067]

[12]

[0068] Here, matrix V is a unitary matrix, and matrix D is a diagonal matrix, where the diagonal elements are non-negative real values ​​sorted in descending order.

[0069] Matrix M can be chosen as:

[0070]

[13]

[0071] Those skilled in the art will understand that the covariance matrix Cov will depend on the input signal used to form the original input signal. The panning method, and the typical use of panning methods by creators of typical signals.

[0072] For example, when the original input signal is a 2-channel stereo signal intended for playback on stereo speakers, the typical translation rules used by content creators will result in some audio objects being translated to the first channel (often referred to as the left channel in this context), some audio objects being translated to the second channel (often referred to as the right channel in this context), and some objects being translated to both channels simultaneously. In this case, the covariance matrix can be similar to:

[0073] For L / R stereo:

[14]

[0074] And according to equations

[12] and

[13] :

[0075] For L / R stereo:

[15]

[0076] The matrix M in Equation

[15] will be familiar to those skilled in the art as a mixing matrix suitable for converting the original input audio signal X in L / R stereo format into an intermediate signal Z in Mid / Side format. Those skilled in the art will also understand that the first channel of Z (often referred to as the Mid signal in this case) is the dominant audio signal (main channel) and has the property that most of the audio elements in the stereo mix will appear in the Mid signal.

[0077] As an alternative example, when the original input signal is a 5-channel surround signal intended for playback on a common five-speaker setup, the typical translation rules used by content creators will result in some audio objects being translated to one of the five channels, while others are translated to two or more channels simultaneously. In this case, the covariance matrix can resemble:

[0078] For 5 channels: Cov =

[16]

[0079] And according to equations

[12] and

[13] :

[0080] For 5 channels: M =

[17]

[0081] It will be understood that the top row of matrix M in equation

[17] consists of similar (or identical) positive values. This means that, according to equation [6], the intermediate signal The first channel will be the original input audio signal The sum of the five channels forms the signal, and this ensures that all sound elements translated in the original input audio signal will appear in the signal. (N channel signal) In the first channel). Therefore, this choice of matrix M ensures that the intermediate signal Y has property DD1 ( It is the main vocal tract.

[0082] In another alternative example, when a multi-channel audio signal is input... The dominant tract is already included (and without loss of generality, it is assumed that the first tract is included). When the dominant channel is the primary channel, matrix M can be an [N × N] identity matrix. In a more specific example of an input multichannel audio signal with a dominant / primary first channel, the input multichannel audio signal can represent an acoustic scene encoded in Ambisonic format (means for encoding acoustic scenes that will be familiar to those skilled in the art).

[0083] At time t, by Figure 4 Analysis block 210(A) in the middle calculates matrix 212 according to the following procedure. ):

[0084] 1. Determine the intermediate signal The covariance at time t. An example of a method for calculating covariance is:

[0085]

[18]

[0086] Alternatively, it can be obtained from the input multi-channel audio signal. Covariance calculation of intermediate signals The covariance is as follows:

[0087]

[19]

[0088] in

[0089]

[20]

[0090] 2. From the [L × L] covariance matrix Extracting scalars , column vector and matrix , where N = L -1, and:

[0091] [twenty one]

[0092] 3. Determine the mixing coefficient The quantities α, β and vector:

[0093] [twenty two]

[0094] [twenty three]

[0095] [twenty four]

[0096] 4. Given w, α and β, solve equation

[25] to determine the input mixing intensity coefficient h and the predicted mixing intensity coefficient. :

[0097]

[25]

[0098] The solution to this equation will also satisfy the pre-predicted constraint equations. An example of a pre-predicted constraint equation is:

[0099]

[26]

[0100] Where f is satisfied The predetermined constant value.

[0101] When using the predictive constraint PPC1, equation

[25] can be modified as follows:

[0102]

[27]

[0103] And for The maximum real value of h can be solved by equation

[27] , and therefore the value of h can be determined by equation

[26] .

[0104] 5. The [L × L] matrix Q is formed as follows:

[0105]

[28]

[0106] 6. The [L × L] matrix P(t) is formed as follows:

[0107]

[29]

[0108] in It is an [L × L] identity matrix.

[0109] Figure 4 Metadata 211(Q) in the document can convey information that allows [the following to be communicated]: Figure 2 The decoder post-mixer 113 determines the unit vector u and the coefficients. Information about h.

[0110] Equation

[27] The solution can be obtained by choosing an initial estimate. And iterate (according to Newton's method, as is known in the art) multiple times to approximate:

[0111]

[30]

[0112] This allows it to be available from Find a reasonable approximation of the solution. It will be understood that other methods known in the art for finding approximate solutions to cubic equations

[27] will be employed.

[0113] According to an alternative embodiment, the intermediate signal can be determined at time t. The correlation between the primary channel and the remaining N non-primary channels Vector u, and determine the input mixing intensity coefficient h and the predicted mixing intensity coefficient according to equation

[28] . To determine P(t) Matrix P(t) such that the signal It will have attributes DD1 and DD2.

[0114] coefficient The determination of h can be governed by the predictive constraint equations. An example of the predictive constraint equations (PPC1) is given in equation

[26] . A preferred choice for the coefficient f is f = 0.5, but the range of values ​​for f is... It may be suitable for use.

[0115] In an alternative embodiment, the following pre-prediction constraints may be used:

[0116]

[31]

[0117] Where c is a predetermined constant. A typical value could be c = 1, but the value of c can be anything. Choose from the range.

[0118] According to the constraint PPC2 in equation

[31] , the solution to equation

[25] is:

[0119]

[32] -

[33]

[34] -

[35]

[0120] Figure 5 This is a block diagram of the predictive mixer 300 according to some embodiments. Matrix terms of equation

[29] and This can be implemented by a predictive mixer 300, where, in this example, the signal It consists of 4 channels (L = 4), with the first channel being 301 ( ) is the primary channel, and the remaining 3 non-primary channels 302 (e.g., Y3, Y4) are based on the three input gains 312 ( , and The input signal is scaled to form a scaled input signal component (e.g., 304). The scaled input signal component is compared with the main input channel 301 ( The sum of these three values ​​is 305 to form the main output 306 (Z1). The main output 306 (Z1) is then multiplied by three prediction gains 313 (...). , and ) Scaling to form three prediction signals (e.g., 311). From the corresponding inputs (e.g., 302) Subtract each predicted signal (e.g., 308 and 309) to form the corresponding non-dominant output 310. ).

[0121] Three input gains 312 ( , and The mixing coefficient u (determined according to equation

[23] ) and the input mixing intensity coefficient h (determined according to the solution of equation

[25] ) can be determined, where:

[0122]

[36]

[0123] Three prediction gains 313 ( , and The mixing coefficient u (determined according to equation

[23] ) and the predicted mixing intensity coefficient can be used to determine the mixing coefficient u. (Based on the solution to equation

[25] ) where:

[0124]

[37]

[0125] Those skilled in the art will understand that Figure 4 The arrangement of linear matrix operations M 202 and P 204 can be achieved using a single matrix R = P × M.

[0126] Those skilled in the art will understand that Figure 2 decoder matrix From the matrix (the inverse of M) and (The reverse of P) formation:

[0127]

[38]

[0128] and It can be pre-calculated (not changing over time), and This can be achieved using the following method:

[0129]

[39]

[0130] Figure 6 Implementation shown Figure 2 The processing elements of the decoder post-mixer 108 are arranged in an arrangement 400. Metadata 402 (Q) provides information to the inverse prediction determination block 403 (B), which calculates and determines the inverse predictor 405. The required coefficient for operation is 404. Signal 401 ( ) by inverse predictor 405 ( ) Processing to generate intermediate signal 406 ( ), intermediate signal 406 ( Then from matrix 407 ( Processed to generate output signal 408 .

[0131] Example process

[0132] Figure 7 This is a flowchart of an adaptive downmixing process 700 for audio signals, which improves continuity according to some embodiments. Process 700 can be, for example... Figure 8 The system shown is implemented using system 800.

[0133] Process 700 includes the following steps: receiving an input multichannel audio signal including a main input audio channel and L non-main input audio channels (701); determining a set of L input gains, where L is a positive integer greater than 1 (702); for each of the L non-main input audio channels and L input gains, forming a corresponding scaled non-main input audio channel from the corresponding non-main input audio channel scaled according to the input gain (703); forming a main output audio channel from the sum of the main input audio channel and the scaled non-main input audio channels (704). 04); For each of the L prediction gains, determine a set of L prediction gains (705); form prediction channels from the main output audio channels scaled according to the prediction gains (706); form L non-main output audio channels from the difference between the corresponding non-main input audio channels and the corresponding prediction signals (706); form an output multi-channel audio signal from the main output audio channels and the L non-main output audio channels (707); encode the output multi-channel audio signal (708); and transmit or store the encoded output multi-channel audio signal (709). Reference Figure 1-6 To describe each of these steps more comprehensively.

[0134] Example System Architecture

[0135] Figure 8 A reference for implementation according to an embodiment is shown. Figure 1-7A block diagram of an example system 800 describing the features and processes. System 800 includes any device capable of playing audio, including but not limited to: smartphones, tablet computers, wearable computers, in-vehicle computers, game consoles, surround sound systems, and self-service terminals.

[0136] As shown in the figure, system 800 includes a central processing unit (CPU) 801, which is capable of executing various processes based on a program stored, for example, in a read-only memory (ROM) 802 or a program loaded into a random access memory (RAM) 803 from a storage unit, for example, 808. The RAM 803 also stores data required as needed when the CPU 801 executes various processes. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0137] The following components are connected to I / O interface 805: input unit 806, which may include a keyboard, mouse, etc.; output unit 807, which may include a display such as a liquid crystal display (LCD) and one or more speakers; storage unit 808, including a hard disk or another suitable storage device; and communication unit 809, including a network interface card such as a network card (e.g., wired or wireless).

[0138] In some implementations, the input unit 806 includes one or more microphones at different locations (depending on the host device), enabling the capture of audio signals in various formats (e.g., mono, stereo, spatial, immersive, and other suitable formats).

[0139] In some implementations, the output unit 807 includes a system with a variety of numbers of speakers. For example... Figure 8 As shown, the output unit 807 (depending on the capabilities of the host device) can present audio signals in various formats (e.g., mono, stereo, immersive, binaural, and other suitable formats).

[0140] Communication unit 809 is configured to communicate with other devices (e.g., via a network). Drive 810 is also connected to I / O interface 805 as needed. Removable media 811, such as a disk, optical disk, magneto-optical disk, flash drive, or other suitable removable media, is mounted on drive 810 such that computer programs read from it can be installed into storage unit 808 as needed. Those skilled in the art will understand that although system 800 is described as including the components described above, in practice, it is possible to add, remove, and / or replace some of these components, and all such modifications or alterations fall within the scope of this disclosure.

[0141] The aspects of the system described herein can be implemented in a suitable computer-based sound processing network environment for processing digital or digitized audio files. Parts of the adaptive audio system may include one or more networks comprising any desired number of individual machines, including one or more routers (not shown) for buffering and routing data transmitted between computers. Such networks can be built on a variety of different network protocols and can be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0142] According to exemplary embodiments of this disclosure, the above processes can be implemented as computer software programs or on computer-readable storage media. For example, embodiments of this disclosure include computer program products comprising computer programs tangibly implemented on machine-readable media, the computer programs including program code for performing methods. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 809, and / or installed from removable media 811, such as... Figure 8 As shown.

[0143] Generally, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry (e.g., control circuitry), software, logic, or any combination thereof. For example, the units discussed above can be implemented by control circuitry (e.g., with...) Figure 8 The control circuitry, in conjunction with other components of the CPU, executes the actions described in this disclosure. Some aspects may be implemented in hardware, while others may be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device (e.g., control circuitry). Although various aspects of exemplary embodiments of this disclosure are shown and described as block diagrams, flowcharts, or other illustrated representations, it is understood that, as non-limiting examples, the blocks, apparatuses, systems, techniques, or methods described herein may be implemented as hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0144] Furthermore, the various blocks shown in the flowchart can be considered as method steps, and / or operations generated by the operation of computer program code, and / or multiple coupled logic circuit elements configured to perform (multiple) related functions. For example, embodiments of this disclosure include a computer program product comprising a computer program tangibly implemented on a machine-readable medium, the computer program containing program code configured to perform the methods described above.

[0145] In the context of this disclosure, a machine-readable medium can be any tangible medium that can contain or store a program used by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be non-transient and can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. More specific examples of machine-readable storage media will include electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0146] Computer program code used to perform the methods of this disclosure may be written in any combination of one or more programming languages. This computer program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device with control circuitry, such that when executed by the processor of a computer or other programmable data processing device, the program code enables the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a computer, partially on a computer, as a standalone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server, or distributed across one or more remote computers and / or servers.

[0147] While this document contains numerous specific implementation details, these details should not be construed as limiting the scope of possible claims, but rather as descriptions of features that may be specific to particular embodiments. Certain features described in this specification within the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Furthermore, although features may be described above as functioning in certain combinations, and even initially claimed to be so, in some cases, one or more features from the claimed combination may be removed from the combination, and the claimed combination may involve sub-combinations or variations thereof. The logical flow described in the accompanying drawings does not require the specific order or sequential order shown to achieve the desired result. Furthermore, additional steps or steps may be provided or removed from the described flow, and additional components may be added to or removed from the described system. Therefore, other implementations are within the scope of the appended claims.

Claims

1. An audio encoding method, comprising: The system uses at least one processor to receive an input multichannel audio signal, which includes a main input audio channel and L non-main input audio channels. The at least one processor is used to determine a set of L input gains, where L is a positive integer greater than 1, and wherein the set of L input gains is determined by scaling a set of L mixing coefficients according to the input mixing intensity coefficient; For each of the L non-primary input audio channels and L input gains, the at least one processor is used to form a corresponding scaled non-primary input audio channel from the corresponding non-primary input audio channel scaled according to the input gain; The primary output audio channel is formed from the sum of the primary input audio channel and the scaled non-primary input audio channels using the at least one processor; The at least one processor is used to determine a set of L prediction gains, wherein the set of L prediction gains is determined by scaling the set of L mixing coefficients according to the prediction mixing intensity coefficient; For each of the L prediction gains, a prediction channel is formed from the main output audio channel scaled according to the prediction gain using the at least one processor; L non-primary output audio channels are formed using the at least one processor from the difference between the corresponding non-primary input audio channels and the corresponding prediction channels; The at least one processor is used to form an output multi-channel audio signal from the main output audio channel and the L non-main output audio channels; The at least one processor generates metadata in response to the input multi-channel audio signal; as well as The output multi-channel audio signal and the metadata are encoded using an audio encoder. The input mixing intensity coefficient is determined based on the predicted mixing intensity coefficient.

2. A system comprising: One or more computer processors; as well as A non-transitory computer-readable medium storing instructions that, when executed by the one or more computer processors, cause the one or more computer processors to perform the following operations: The system uses at least one processor to receive an input multichannel audio signal, which includes a main input audio channel and L non-main input audio channels. The at least one processor is used to determine a set of L input gains, where L is a positive integer greater than 1, and wherein the set of L input gains is determined by scaling a set of L mixing coefficients according to the input mixing intensity coefficient; For each of the L non-primary input audio channels and L input gains, the at least one processor is used to form a corresponding scaled non-primary input audio channel from the corresponding non-primary input audio channel scaled according to the input gain; The primary output audio channel is formed from the sum of the primary input audio channel and the scaled non-primary input audio channels using the at least one processor; The at least one processor is used to determine a set of L prediction gains, wherein the set of L prediction gains is determined by scaling the set of L mixing coefficients according to the prediction mixing intensity coefficient; For each of the L prediction gains, a prediction channel is formed from the main output audio channel scaled according to the prediction gain using the at least one processor; L non-primary output audio channels are formed using the at least one processor from the difference between the corresponding non-primary input audio channels and the corresponding prediction channels; The at least one processor is used to form an output multi-channel audio signal from the main output audio channel and the L non-main output audio channels; The at least one processor generates metadata in response to the input multi-channel audio signal; as well as The output multi-channel audio signal and the metadata are encoded using an audio encoder. The input mixing intensity coefficient is determined based on the predicted mixing intensity coefficient.

3. A non-transitory computer-readable medium storing instructions that, when executed by one or more computer processors, cause the one or more computer processors to perform the following operations: The system uses at least one processor to receive an input multichannel audio signal, which includes a main input audio channel and L non-main input audio channels. The at least one processor is used to determine a set of L input gains, where L is a positive integer greater than 1, and wherein the set of L input gains is determined by scaling a set of L mixing coefficients according to the input mixing intensity coefficient; For each of the L non-primary input audio channels and L input gains, the at least one processor is used to form a corresponding scaled non-primary input audio channel from the corresponding non-primary input audio channel scaled according to the input gain; The primary output audio channel is formed from the sum of the primary input audio channel and the scaled non-primary input audio channels using the at least one processor; The at least one processor is used to determine a set of L prediction gains, wherein the set of L prediction gains is determined by scaling the set of L mixing coefficients according to the prediction mixing intensity coefficient; For each of the L prediction gains, a prediction channel is formed from the main output audio channel scaled according to the prediction gain using the at least one processor; L non-primary output audio channels are formed using the at least one processor from the difference between the corresponding non-primary input audio channels and the corresponding prediction channels; The at least one processor is used to form an output multi-channel audio signal from the main output audio channel and the L non-main output audio channels; The at least one processor generates metadata in response to the input multi-channel audio signal; as well as The output multi-channel audio signal and the metadata are encoded using an audio encoder. The input mixing intensity coefficient is determined based on the predicted mixing intensity coefficient.

Images