Determining modifications to apply to a multi-channel audio signal and associated encoding and decoding

Abstract

The invention relates to a method for determining a set of corrections (Corr.) to be applied to a multi-channel audio signal, the set of corrections being determined on the basis of an item of information (Inf.B) representing a spatial image of the original multi-channel signal and an item of information (Inf.B) representing a spatial image of the original multi-channel signal that has been coded and then decoded. The invention also relates to a decoding method and an encoding method implementing the determination method, as well as associated coding and decoding devices.

Description

[Technical Field]

[0001] The present invention relates particularly to the encoding / decoding of ambiophonic (hereinafter also referred to as "ambisonic") spatial acoustic data. [Background technology]

[0002] The encoders / decoders (hereinafter referred to as "codecs") currently used in mobile telephony are monaural (single signal channel rendered for a single speaker). The 3GPPEVS ("Enhanced Voice Services") codec makes it possible to provide "ultra HD" quality (also called "high definition plus" or HD+ audio) with an ultra-wideband (SWB) audio bandwidth for signals sampled at 32 or 48 kHz, or a full-band (FB) audio bandwidth for signals sampled at 48 kHz, with audio bandwidths of 14.4 to 16 kHz in SWB mode (9.6 to 128 kbit / s) and 20 kHz in FB mode (16.4 to 128 kbit / s).

[0003] The next stage of evolution in the quality of operator-provided conversational services should include immersive services using devices such as smartphones equipped with multiple microphones, or remote presence, i.e., spatial audio conferencing or video conferencing equipment using 360° video, or "live" audio content sharing equipment that enables spatial 3D audio rendering that is far more immersive than mere 2D stereo rendering. With the widespread use of listening to mobile phones using audio headsets, and the emergence of advanced audio equipment (accessories such as 3D microphones, voice assistants with acoustic antennas, and virtual reality headsets), the capture and rendering of spatial audio scenes is now widespread enough to provide an immersive communication experience.

[0004] For this purpose, the future 3GPP standard "IVAS" (short for "Immersive Voice And Audio Services") proposes extending the EVS codec for immersive applications by accepting at least the following spatial audio formats (and combinations thereof) as codec input formats. - Stereo or 5.1 multi-channel (channel-based) format where each channel outputs to a speaker (e.g., L and R in stereo, or L, R, Ls, Rs and C in 5.1), - An object (object-based) format in which an audio object is described as an audio signal (generally monaural) associated with metadata that describes the attributes of the object (such as its position in space, the spatial width of the source, etc.), - An ambisonic (scene-based) format that describes the sound field at a given point, generally captured by a spherical microphone or synthesized in the domain of spherical harmonics.

[0005] Of particular interest below is the encoding of sound in an ambisonic format according to exemplary embodiments (at least some of the embodiments presented in relation to the present invention are also applicable to formats other than ambisonic).

[0006] Ambisonics is a method and system for recording (acoustically "encoding") and reproducing (acoustically "decoding") spatialized sound. A (primary) ambisonic microphone contains at least four capsules (typically cardioid or subcardioid) arranged at the vertices of a spherical grid, such as a regular tetrahedron. The audio channels associated with these capsules are referred to as the "A format." This format is converted into the "B format," in which the sound field is decomposed into four components (spherical harmonics) denoted as W, X, Y, and Z, corresponding to four simultaneous virtual microphones. Component W corresponds to capturing the sound field in all directions, while the more directional components X, Y, and Z are analogous to pressure gradient microphones directed along three orthogonal axes in space. Ambisonic systems are flexible in the sense that recording and rendering are separate and isolated. This allows for (acoustically) decoding for speakers of any configuration (e.g., binaural, 5.1, or 7.1.4 multi-channel (with rising) "surround" sound). The ambisonic approach can be generalized to more than four channels in the B format, and this generalized representation is commonly called "HOA" (short for "Higher-Order Ambisonics"). By decomposing the sound into more spherical harmonics, the spatial rendering accuracy when rendering for speakers is improved.

[0007] The Mth-order ambisonic signal is given by K = (M + 1). 2It contains 1,000 components, and in the case of first-order ambisonic (M=1), there are 4 components W, X, Y, and Z, generally called FOA (First-Order Ambisonics). There is also a "planar" variation of ambisonic (W, X, Y) that decomposes sounds defined within a plane, which is generally the horizontal plane. In this case, the number of components is K=2M+1 channels. First-order ambisonic (4 channels: W, X, Y, Z), planar first-order ambisonic (3 channels: W, X, Y), and higher-order ambisonic will all be referred to as "ambisonic" below for ease of reading, and the processing operations presented are applicable regardless of whether it is planar or non-planar, and regardless of the number of ambisonic components.

[0008] Hereinafter, "ambisonic signal" is the name given to a B-format signal of a predetermined order that has a specific number of ambisonic components. This also includes hybrid cases, for example, where there are only 8 secondary channels (instead of 9). More precisely, in the secondary case, there are usually 5 channels (usually denoted as R, S, T, U, V) in addition to 4 primary channels (W, X, Y, Z), and one of the higher-order channels (e.g., R) can be ignored. The signal processed by the encoder / decoder takes the form of a series of blocks of acoustic samples, which will be referred to below as "frames" or "subframes".

[0009] Furthermore, mathematical notation below will follow the following conventions. - Scalar: s or N (lowercase is a variable, uppercase is a constant) The - operator Re(.) indicates the real part of a complex number. -Vector: u (bold lowercase) -Matrix: A (bold capital)

[0010] Notation A T and A H The symbols ∫ represent the transpose and Hermitian transpose (transpose and conjugate) of A, respectively. - A one-dimensional discrete-time signal s(i) defined over a time interval of length L, i=0,...,L-1 is represented as a row vector. s=[s(0),...,s(L-1)]

[0011] This is to avoid using parentheses: s=[s0,...,s L-1 It can also be written as ]. - A K-dimensional multidimensional discrete-time signal b(i), defined over a time interval of length L i=0,...,L-1, is represented by a matrix of size L×K.

number

[0012] This is to avoid using parentheses B=[B ij It can also be written as i=0,...K-1, j=0...L-1. -A 3D point having Cartesian coordinates (x,y,z) can be converted to spherical coordinates (r,Θ,φ), where r is the distance to the origin, Θ is the azimuth angle, and φ is the elevation angle. Here, without loss of generality, we use mathematical notation in which the elevation angle is defined with respect to the horizontal plane (0xy). This invention can be easily adapted to other definitions, including notation used in physics in which the azimuth angle is defined with respect to the axis Oz. Furthermore, we will not discuss here the notation conventions known in prior art related to ambisonics regarding the order of ambisonic components (including the abbreviations ACN for Ambisonic Channel Number, SID for Single Index Designation, and FuMA for Furse-Malham) and the normalization of ambisonic components (SN3D, N3D, maxN). More detailed information can be found, for example, in the following resources available online. https: / / en.wikipedia.org / wiki / Ambisonic_data_exchange_formats By convention, in an ambisonic signal, the first component generally corresponds to the omnidirectional component W.

[0013] The simplest way to encode an ambisonic signal is to use a monaural encoder and apply it in parallel to all channels, where the bit allocation may differ depending on the channel. This method is referred to here as "multi-mono". The multi-mono method can be extended to multi-stereo encoding (where pairs of channels are encoded separately by stereo codecs), or more generally, to the use of multiple parallel instances of the same core codec. [Overview of the project] [Problems that the invention aims to solve]

[0014] One such embodiment is shown in Figure 1. The input signal is divided into channels (one mono channel or multiple channels) by block 100. These channels are separately encoded by blocks 120-122 based on a predetermined distribution and bit allocation. Their bitstreams are multiplexed (block 130), transmitted and / or stored, then demultiplexed (block 140) to apply decoding in order to reconstruct the decoded channels (blocks 150-152), and then recombined (block 160).

[0015] The resulting quality varies depending on the core encoding and decoding used (blocks 120-122 and 150-152), and generally only extremely high bitrates should suffice. For example, in the case of multi-mono, EVS encoding is considered (from a perceptual standpoint) semi-transparent at a bitrate of at least 48 kbit / s per channel (mono), and therefore, a minimum bitrate of 4 × 48 = 192 kbit / s can be obtained for a first-order ambisonic signal. Because the multi-mono encoding method does not consider inter-channel correlation, spatial deformation occurs due to the addition of various artifacts such as the appearance of ghost sources, diffuse sound, or displacement of the sound source trajectory. Encoding ambisonic signals using this method leads to a decrease in the degree of spatialization.

[0016] This provides an alternative method of encoding parameters rather than encoding all channels of a stereo or multi-channel signal separately. In this type of encoding, after a processing operation called "downmixing," the input multi-channel signal is reduced to a smaller number of channels, which are then encoded and transmitted, along with the encoding of additional spatial information. Parametric decoding involves increasing the number of channels after decoding the transmitted channels, using a processing operation called "upmixing" (typically performed via decorrelation) and spatial synthesis based on the decoded additional spatial information. An example of stereo parametric encoding is given by the 3GPPe-AAC+ codec. Note that the downmixing operation also leads to a decrease in the degree of spatialization, in which case the spatial image is modified. [Means for solving the problem]

[0017] This invention aims to improve upon the prior art.

[0018] For this purpose, we propose a method for determining a set of modifications to be applied to a multichannel acoustic signal, the set of modifications being determined from information representing the spatial image of the original multichannel signal and from information representing the encoded and then decoded spatial image of the original multichannel signal.

[0019] The determined set of modifications applied to the decoded multichannel signal thus allows for the suppression of spatial degradation caused by encoding and possibly channel reduction / increase operations. The application of these modifications thus allows for the restoration of the spatial image of the decoded multichannel signal that is closest to the spatial image of the original multichannel signal.

[0020] In a particular embodiment, the set of modifications is determined in the entire time domain (one frequency band). In some variations, this is performed in the time domain by frequency subbands. This makes it possible to adapt the modifications according to the frequency band.

[0021] In other variations, this is performed in the real or complex transformation domain (typically the frequency domain), such as the Short-Time Discrete Fourier Transform (STFT) or Modified Discrete Cosine Transform (MDCT).

[0022] The present invention also relates to a method for decoding a multichannel acoustic signal, comprising the following steps. - The steps include receiving a bitstream containing an encoded audio signal from the original multichannel signal and information representing a spatial image of the original multichannel signal, - The steps include decoding the received encoded audio signal to obtain the decoded multi-channel signal, - The step of decoding information representing the spatial image of the original multi-channel signal, - A step of determining information representing the spatial image of the decoded multichannel signal, -The step of determining a set of modifications to be applied to the decoded signal using the above-described determination method, - A step of modifying the decoded multichannel signal using the determined set of modifications.

[0023] Thus, in this embodiment, the decoder can determine the modifications to be applied to the decoded multichannel signal from information representing the spatial image of the original multichannel signal received from the encoder. The information received from the encoder is therefore limited. The decoder is responsible for both determining and applying the modifications.

[0024] The present invention also relates to a method for encoding a multichannel acoustic signal, comprising the following steps. - The step of encoding the audio signal from the original multi-channel signal, - A step of determining information that represents the spatial image of the original multi-channel signal, - A step of locally decoding the encoded audio signal to obtain the decoded multi-channel signal, - A step of determining information representing the spatial image of the decoded multichannel signal, -The step of determining a set of modifications to be applied to the decoded multi-channel signal using the above-described determination method, - A step of encoding the determined set of modifications.

[0025] In this embodiment, the encoder determines the set of modifications to be applied to the decoded multi-channel signal and transmits it to the decoder. Therefore, the encoder takes the lead in determining these modifications.

[0026] In a first particular embodiment of the above-described decoding method or encoding method, the information representing the spatial image is a covariance matrix, and the step of determining the set of modifications further includes the following steps: - The steps of obtaining a weight matrix containing weight vectors associated with a set of virtual speakers, - A step of determining the spatial image of the original multichannel signal from the acquired weight matrix and the covariance matrix of the received original multichannel signal, - A step of determining the spatial image of the decoded multichannel signal from the acquired weight matrix and the determined covariance matrix of the decoded multichannel signal, - A step of calculating the ratio of the spatial image of the original multichannel signal to the spatial image of the decoded multichannel signal in the direction of the speakers of the virtual speaker set in order to obtain a set of gains.

[0027] According to this embodiment, the method using rendering in the speaker makes it possible to transmit only a limited amount of data from the encoder to the decoder. In fact, for a given order M, K = (M+1) (for the same number of virtual speakers) 2 Sending a small number of coefficients is sufficient, but it is recommended to use more virtual speakers and therefore more points for more stable correction. Furthermore, the correction can be easily interpreted in terms of the gain associated with the virtual speakers.

[0028] In another modified embodiment, if the encoder directly determines the energy of the signal in various directions and transmits this spatial image of the original multi-channel signal to the decoder, the determination of a set of modifications to the decoding method further includes the following steps: - The steps of obtaining a weight matrix containing weight vectors associated with a set of virtual speakers, - A step of determining the spatial image of the decoded multichannel signal from the acquired weight matrix and from information representing the spatial image of the determined decoded multichannel signal, - A step of calculating the ratio of the spatial image of the original multichannel signal to the spatial image of the decoded multichannel signal in the direction of the speakers of the virtual speaker set in order to obtain a set of gains.

[0029] To ensure that the correction values ​​are not too extreme, the decoding or encoding method includes a step of limiting the value of the obtained gain to at least one threshold.

[0030] This set of gains constitutes a set of modifications, which may take the form of a modification matrix containing, for example, the set of gains determined in this way.

[0031] In a second particular embodiment of the decoding or encoding method, the information representing the spatial image is a covariance matrix, and the step of determining a set of modifications includes the step of determining a transformation matrix by matrix decomposition of two covariance matrices, the transformation matrix constituting the set of modifications.

[0032] This embodiment has the advantage of performing correction directly in the ambisonic region in the case of ambisonic multichannel signals. Therefore, the step of converting a signal rendered for speakers to the ambisonic region is avoided. This embodiment also allows for the optimization of the correction to be mathematically optimal, even if it requires the transmission of more coefficients compared to the method of rendering for speakers. In fact, for order M, and therefore for a certain number of components K = (M+1) 2In this case, the number of coefficients transmitted is K × (K+1) / 2. To avoid excessive amplification across a specific frequency range, normalization coefficients are determined and applied to the transformation matrix.

[0033] If the set of modifications is represented by a transformation matrix or modification matrix as described above, the decoded multichannel signal is modified by the determined set of modifications by applying the set of modifications to the decoded multichannel signal, i.e., directly in the ambisonic domain in the case of an ambisonic signal.

[0034] In an embodiment where rendering in the speaker is performed by a decoder, the decoded multichannel signal is modified using a set of modifications determined in the following steps. - The steps include acoustically decoding the decoded multichannel signal using a set of virtual speakers, - The step of applying the obtained set of gains to the signal obtained from acoustic decoding, - The step of acoustically encoding the modified signal obtained from acoustic decoding in order to obtain the components of the multi-channel signal, - A step of summing the components of the multichannel signal thus obtained in order to obtain a corrected multichannel signal.

[0035] In one variant embodiment, the above-described decoding, gain application, and coding / summation steps are grouped into a direct correction operation using a correction matrix. This correction matrix may be applied directly to the decoded multichannel signal, which has the advantage of directly correcting the ambisonic region as described above.

[0036] In a second embodiment in which the encoding method performs a method for determining a set of modifications, the decoding method includes the following steps: - The steps of receiving a bitstream which includes an encoded audio signal from the original multichannel signal and an encoded set of modifications to be applied to the decoded multichannel signal, the set of modifications encoded using the above-described encoding method, - The steps include decoding the received encoded audio signal to obtain the decoded multi-channel signal, - The step of decoding the encoded set of modifications, - A step of modifying a decoded multichannel signal by applying a set of decoded modifications to the decoded multichannel signal.

[0037] In this embodiment, the encoder determines which modifications to be applied directly to the decoded multichannel signal in the ambisonic region, and the decoder applies these modifications directly to the multichannel signal decoded in the ambisonic region.

[0038] In this case, the set of modifications may be a transformation matrix or a modification matrix containing a set of gains.

[0039] In one variant of a decoding method used for rendering to a speaker, the decoding method includes the following steps: - The steps of receiving a bitstream containing an encoded audio signal from the original multichannel signal and an encoded set of modifications to be applied to the decoded multichannel signal, the set of modifications encoded using the encoding method described above, - The steps include decoding the received encoded audio signal to obtain the decoded multi-channel signal, - The step of decoding the encoded set of modifications, -The decoded multichannel signal is processed in the following steps, i.e. The steps include: • Acoustically decoding the decoded multichannel signal using a set of virtual speakers, The steps include applying the obtained set of gains to the signal obtained from acoustic decoding, • The steps include acoustically encoding the modified signal obtained from acoustic decoding in order to acquire the components of the multi-channel signal, - A step in which the components of the multichannel signal thus obtained are summed up in order to obtain a corrected multichannel signal, and the correction is performed using the decoded correction set.

[0040] In this embodiment, the encoder determines the modifications to be applied to the signal obtained from acoustic decoding of the virtual speaker set, and the decoder applies these modifications to the signal obtained from acoustic decoding, and then, in the case of an ambisonic multichannel signal, converts these signals back into the ambisonic domain.

[0041] In one variant embodiment, the above-described decoding, gain application, and encoding / summation steps are grouped into a direct modification operation using a modification matrix. This modification is then performed directly by applying the modification matrix to the decoded multichannel signal, such as an ambisonic signal. As mentioned above, this has the advantage of performing the modification directly in the ambisonic region.

[0042] The present invention also relates to a decoding device including a processing circuit that performs the decoding method described above.

[0043] The present invention also relates to a decoding device including a processing circuit that performs the encoding method described above.

[0044] The present invention also relates to a computer program that includes instructions for performing the above-described decoding or encoding method when executed by a processor.

[0045] The present invention last relates to a processor-readable storage medium that stores a computer program including instructions for performing the above-described decoding or encoding method.

[0046] Other features and advantages of the present invention will become apparent upon closer examination of the following description of specific embodiments presented through brief, illustrative, and non-limiting examples and accompanying drawings. [Brief explanation of the drawing]

[0047] [Figure 1] The above-mentioned multi-mono encoding using conventional technology is shown. [Figure 2]The steps of a method for determining a set of modifications according to one embodiment of the present invention are shown in the form of a flowchart. [Figure 3] A first embodiment of an encoder and decoder, an encoding method and a decoding method according to the present invention is shown. [Figure 4] A first detailed embodiment of the block that determines the set of modifications is shown. [Figure 5] A second detailed embodiment of the block that determines the set of modifications is shown. [Figure 6] A second embodiment of an encoder and decoder, an encoding method and a decoding method according to the present invention is shown. [Figure 7] Multiple examples of structural embodiments of an encoder and decoder according to one embodiment of the present invention are shown. [Modes for carrying out the invention]

[0048] The methods described below are based in particular on correcting spatial degradation to ensure that the spatial image of the decoded signal is as close as possible to the original signal. Unlike known parametric encoding methods for stereo, i.e., multi-channel signals, in which perceptual cues are encoded, the present invention is not based on perceptual interpretation of spatial image information because the ambisonic region is not directly "audible".

[0049] Figure 2 shows the main steps performed to determine the set of modifications to apply to the encoded and then decoded multi-channel signal.

[0050] The input to this determination method is the original multichannel signal B having dimensions K × L (i.e., K components of L time or frequency samples). In step S1, information representing the spatial image of the original multichannel signal is extracted.

[0051] What is interesting here is the case of multi-channel signals having ambisonic representation as described above. The present invention can also be applied to other types of multi-channel signals, such as B-format signals that have been modified, such as suppressing certain components (e.g., suppressing the second-order R component to maintain only eight channels) or matrixing of the B-format to pass to an equivalent region (referred to as the "equivalent spatial region"), as described in the 3GPPTS26.260 specification, for example. Another example of matrixing is shown in "Channel Mapping 3" of the IETFOpus codec and in 3GPPTS26.918 (Clause 6.1.6.3).

[0052] Here, "spatial image" refers to the distribution of acoustic energy of an ambisonic acoustic scene in various directions within space. In some variations, the spatial image describing the acoustic scene generally corresponds to positive values ​​evaluated in various predetermined directions within space, for example, in the form of MUSIC (Multiple Signal Classification) pseudospectrals sampled in these directions or histograms of the direction of arrival (the direction of arrival is determined by discretization given by the predetermined direction). These positive values ​​can be interpreted as energy, and to simplify the description of the present invention, we consider them as follows.

[0053] The spatial image associated with an ambisonic acoustic scene therefore represents relative acoustic energy (or more generally, a positive value) as a function of various directions in space. In this invention, the information representing the spatial image may be, for example, a covariance matrix calculated between channels of a multichannel signal or energy information associated with the direction from which the sound was emitted (associated with the direction of a virtual speaker distributed across a unit sphere).

[0054] The set of modifications applied to a multichannel signal is information that can be defined by a set of gains associated with the direction from which the sound was emitted, and may be in the form of a modification matrix that includes such set of gains or a transformation matrix.

[0055] The covariance matrix of the multi-channel signal B is obtained, for example, in step S1. As described below with respect to FIGS. 3 and 6, the matrix is calculated, for example, as follows. Within the normalization factor, C = B.B T (In the case of real numbers) Or within the normalization factor, C = Re(B.B H (In the case of complex numbers)

[0056] In some variants, an operation for temporally smoothing the covariance matrix may be used. In the case of multi-channel signals in the time domain, the covariance can be recursively estimated (one sample at a time) in the following form. Cij(n) = n / (n + 1)Cij(n - 1) + 1 / (n + 1)bi(n)bj(n)

[0057] In one variant embodiment, energy information is obtained in various directions (associated with the directions of virtual speakers distributed over the unit sphere). For this purpose, for example, the SRP ("Steered-Response Power") method described later with respect to FIGS. 3 and 4 is applied. In some variants, other spatial image calculation methods (MUSIC pseudo-spectrum, histogram of direction of arrival) may be used.

[0058] Multiple embodiments for encoding the original multi-channel signal are conceivable and are described below.

[0059] In the first embodiment, each channel b of B in step S2 k , k = 0,.., K - 1 is encoded using multi-mono encoding, and each channel b k is encoded separately. In some variant embodiments, channel b kMultistereo coding is also possible, in which the signals are encoded into separate pairs. 5.1 A conventional example of an input signal uses two separate stereo coding operations, L / R and Ls / Rs, together with the C and LFE (low frequency only) mono coding operations. In the case of ambisonic signals, multistereo coding may be applied to an equivalent multichannel signal obtained after matrixing the ambisonic components (B format) or channels into B format. For example, in the first order, channels W, X, Y, and Z can be transformed into four transformed channels, and two pairs of channels are encoded separately and then decoded back into B format. An example is shown in the Opus codec ("Channel Mapping 3") and the latest version of the 3GPPTR 26.918 specification (Clause 6.1.6.3).

[0060] In other variations, step S2 may also use concatenated multichannel coding, such as the MPEG-H3D audio codec for ambisonic (scene-based) formats. In this case, the codec concatenates and encodes the input channels. In the MPEG-H example, this concatenation coding is broken down into several steps for an ambisonic signal, including extracting and encoding the dominant monaural source, extracting the ambiance (typically into a first-order ambisonic signal), and encoding metadata describing all extracted channels (referred to as "carrier channels") and acoustic beamform vectors for extracting the dominant channel. Concatenated multichannel coding makes it possible to utilize the relationships between all channels to perform all bit allocations that take into account all audio content, for example, by extracting the dominant source and ambiance.

[0061] In a preferred embodiment, the exemplary embodiment of step S2 is multi-mono encoding performed using the 3GPPEVS codec as described above. However, the method according to the present invention can thus be used independently of the core codec (multi-mono, multi-stereo, concatenated encoding) used to represent the channels to be encoded.

[0062] The signal encoded in this bitstream format may be decoded in step S3 by the encoder's local decoder or by a decoder after transmission. This signal is a multichannel signal.

number

[0063] Steps S2a, S2b, S3a, and S3b represent a variant embodiment of the encoding and decoding of the multi-channel signal B. The difference from the encoding in step S2 described above lies in the use of additional processing operations to reduce the number of channels ("downmix") in step S2a and increase the number of channels ("upmix") in step S3b. These encoding and decoding steps (S2b, S3a) are the same as steps S2 and S3, except that steps S2b and S3a have fewer input and output channels.

[0064] One example of downmixing a first-order ambisonic input signal is one in which only the W channel is maintained. For ambisonic input signals of an order greater than 1, the first four components W, X, Y, and Z may be acquired as a downmix (and thus the signal is truncated to the first order). In some variations, a subset of the ambisonic components (e.g., eight second-order channels without component R) may be acquired as a downmix, and cases where it is represented as a matrix are also conceivable. For example, a stereo downmix might be L = W - Y + 0.3 * X, R = W + Y + 0.3 *The signal is acquired in X format (using only the FOA channel). One example of upmixing a monaural signal is to apply various room-space impulse responses (SRIRs) or various (all-pass) decorrelation filters in the time or frequency domain. An exemplary embodiment of decorrelation in the frequency domain is shown, for example, in reference 3GPPS4-180975, pCR to 26.118 on Dolby VRStream audio profile candidate (Clause X.6.2.3.5).

[0065] The signal B' obtained from this "downmix" processing operation is encoded in step S2b using a core codec (multi-mono, multi-stereo, concatenated encoding) with a mono or multi-mono scheme, for example, a 3GPPEVS codec. The input audio signal from encoding step S2b and the output audio signal from decoding step S3a have fewer channels than the original multi-channel audio signal. In this case, the spatial image represented by the core codec is already significantly degraded even before encoding. In extreme cases, encoding only the W channel reduces the number of channels to a single mono channel. Subsequently, the input signal is limited to a single audio channel, and therefore the spatial image is lost. The method according to the present invention makes it possible to describe and reconstruct this spatial image so that it is as close as possible to the spatial image of the original multi-channel signal.

[0066] The multichannel signal decoded at the output side of the upmix step in S3b of this modified embodiment

number

[0067] In step S4, the information representing the spatial image of the decoded multichannel signal is the multichannel signal decoded by two variant examples (S2~S3 or S2a~S2b~S3a~S3b)

number

[0068] In step S5, information representing the original multi-channel signal and the decoded multi-channel signal is used to determine a set of modifications to be applied to the decoded multi-channel signal in order to suppress spatial degradation.

[0069] Two embodiments will be described below with reference to Figures 4 and 5 to illustrate the steps described above.

[0070] The method described in Figure 2 can be performed in the time domain using the entire frequency band (in the case of a single band) or frequency subbands (in the case of multiple bands), and without changing the operation of the method, each subband is then processed separately. When the method is performed using subbands, the set of modifications is therefore determined for each subband, resulting in extra costs in terms of calculation and data transmitted to the decoder compared to the single-band case. The subband division may be uniform or non-uniform. For example, the spectrum of a signal sampled at 32 kHz may be divided according to various deformation examples. - Four bandwidths, each with a width of 1, 3, 4, and 8 kHz, or 2, 2, 4, and 8 kHz. - 24 bark bands (from 100Hz in the low frequency range to 3.5~4kHz in the last subband) -The 24 bark bands can be grouped into sets of 4 or 6 consecutive band blocks to form 6 or 4 "clumped" bands, respectively.

[0071] Other divisions are also possible, including cases with different sampling frequencies (e.g., 16 or 48 kHz), such as the ERB band (short for "Equivalent Square Bandwidth") - or into thirds of an octave.

[0072] In some variations, the present invention can also be carried out in a transformed domain, such as the domain of the Short-Time Discrete Fourier Transform (STFT) or the domain of the Modified Discrete Cosine Transform (MDCT).

[0073] Several embodiments for performing the determination of the set of modifications and applying the set of modifications to the decoded signal are described below.

[0074] Please recall here a known technique for encoding sound sources in an ambisonic format. A monaural sound source can be artificially spatialized by multiplying its signal by the values ​​of spherical harmonics associated with the direction of the source (assuming the signal is carried by a plane wave) in order to obtain an equal number of ambisonic components. This involves the step of calculating the coefficients of each spherical harmonic of a desired order at a position determined by the azimuth angle Θ and the elevation angle φ. B=Y(Θ, φ).s Here, s is the monaural signal to be spatialized, and Y(Θ,φ) is the coding vector that defines the coefficients of the spherical harmonics associated with the direction (Θ,φ) in the Mth order. An example of an encoded vector is given using the SN3D notation convention for the first-order case, and by the following formula for the order of the SID or FuMa channel.

number

[0075] In some variant examples, other normalization notation conventions (e.g., maxN, N3D) and channel order (e.g., ACN) may be used, and various embodiments are thus adapted to conventions used for the normalization order of one or more ambisonic components (FOA or HOA). This is equivalent to modifying the order of rows Y(Θ,φ) or multiplying these rows by a predetermined constant.

[0076] For higher orders, the coefficients Y(Θ,φ) of spherical harmonics can be found in B. Rafaely's book "Fundamentals of Spherical Array Processing," Springer, 2015. In general, for order M, K = (M+1) 2 There are several ambisonic signals.

[0077] Similarly, recall here some concepts regarding ambisonic rendering by speakers. Ambisonic sound is not intended to be heard in this way. To be heard immersively through speakers or headphones, a rendering (also called "rendering") "decoding" step in an acoustic sense must be performed. Typically distributed on a sphere with a unit radius, with direction (Θ) in azimuth and elevation angles. n ,φ n Consider the case of N (virtual or physical) speakers, n=0,...,N-1, where n is known. The decoding considered here is the signal from the speaker s n This is a linear operation that involves applying matrix D to the ambisonic signal B in order to obtain the signal s n teeth,

number

[0078] Matrix D is

number

[0079] There are several methods for "decoding" in an acoustic sense. One method, known as "basic decoding" and also called "mode matching," is based on encoding a matrix E associated with all directions of a virtual speaker. E=[Y(θ0,φ0)...Y(θ N-1 ,φ N-1 )]

[0080] According to this method, matrix D is typically defined as the pseudo-inverse of E. E:D = pinv(E) = D T (DD T ) -1

[0081] Alternatively, a method also known as the "projection" method gives similar results for a specific regular distribution of directions and is given by the following equation:

number

[0082] In the latter case, for each direction of the subscript n,

number

[0083] In connection with the present invention, such a matrix would function as a directional beamforming matrix describing a method for acquiring signals that characterize directions in space in order to perform analysis and / or spatial transformations.

[0084] In relation to the present invention, it is useful to describe the interconversion that passes through the speaker domain to the ambisonic domain. The sequential application of the two conversions should accurately reproduce the original ambisonic signal, provided that no intermediate modifications are applied to the speaker domain. The interconversion is therefore defined as the execution of a pseudo-inverse conversion of D. pinv(D).S=D T (DD T ) -1 .S

[0085] K=(M+1) 2 Therefore, a matrix D of size K×K can be inversed under certain conditions, in which case B=D -1 It is .S.

[0086] In the "mode matching" method, we find that pinv(D) = E. In some variant examples, other methods of decoding using D with the corresponding inverse transform E may be used, and the only condition that must be satisfied is that the combination of decoding using D and the inverse transform using E must achieve a complete reconstruction (assuming no intermediate processing operations are performed between acoustic decoding and acoustic coding).

[0087] Such variations can be given, for example, as follows. - The regularization term is D T (DD T +εI) -1 The format is "mode matching" decoding, where ε is a small value (e.g., 0.01). - Conventional "in-phase" or "maximum rE" decoding known in the prior art, -Or, it is a variation where the distribution in the direction of the speaker is not regular on a spherical surface.

[0088] Figure 3 shows a first embodiment of an encoding and decoding apparatus that performs an encoding and decoding method including a method for determining a set of modifications as described with reference to Figure 2.

[0089] In this embodiment, the encoder calculates information representing the spatial image of the original multi-channel signal and transmits it to the decoder so that spatial degradation caused by encoding can be corrected. This makes it possible to reduce spatial artifacts in the decoded ambisonic signal during the decoding process.

[0090] The encoder, therefore, receives a multi-channel input signal in the form of an ambisonic representation FOA, or HOA, i.e., a hybrid representation having a subset of ambisonic components up to a given partial ambisonic order, the latter of which is actually included in a similar manner to the FOA or HOA, with zero missing ambisonic components, and the ambisonic order is given by the minimum order necessary to include all the given components. Thus, without loss of generality, we will consider the description of the FOA or HOA case below.

[0091] In the embodiments described above, the input signal is sampled at 32 kHz. The encoder preferably operates in frames of length 20 ms, i.e., L = 640 samples per frame at 32 kHz. In some variations, other frame lengths and sampling frequencies are possible (e.g., L = 480 samples per 10 ms frame at 48 kHz). In one preferred embodiment, encoding is performed in the time domain (in one or more bandwidths), but in some variations, the present invention may be performed in a transformed domain, for example, after a Short-Time Discrete Fourier Transform (STFT) or Modified Discrete Cosine Transform (MDCT).

[0092] Depending on the encoding embodiment used, a block 310 can be performed to reduce the number of channels (DMX), as described with respect to Figure 2, and the input to block 311 is the output signal B' of block 310 if downmixing is performed, and otherwise it is signal B. In one embodiment, when downmixing is applied, this is, for example, to retain only the W channel of the first-order ambisonic input signal, and to retain only the first four ambisonic components W, X, Y, and Z of the ambisonic input signal of order > 1 (thus truncating the signal down to the first order). Other types of downmixing (as described above in conjunction with the selection of a subset of channels and / or matrixing) can also be performed without modifying the method according to the present invention.

[0093] Block 311 outputs the audio signal b' of block 310 if the downmix step has been performed. kor the audio signal b of the original multi-channel signal B k This is encoded. This signal corresponds to the ambisonic component of the original multi-channel signal, if no channel reduction operation had been applied.

[0094] In a preferred embodiment, block 311 uses multimono coding (COD) with fixed or variable assignments, and the core codec is a standard 3GPPEVS codec. In this multimono scheme, each channel b k or b' k These are encoded separately by a single instance of the codec. However, in some variations, other encoding methods, such as multi-stereo encoding or concatenated multi-channel encoding, are also possible. This, therefore, provides the encoded audio signal obtained from the original multi-channel signal in the form of a bitstream at the output of the encoding block 311, which is then sent to the multiplexer 340.

[0095] Optionally, block 320 performs subband splitting. In some variants, this subband splitting may reuse equivalent processing operations performed in block 310 or 311, where the splitting in block 320 is performed.

[0096] In a preferred embodiment, the channels of the original multi-channel audio signal are divided into four frequency subbands, each with a width of 1 kHz, 3 kHz, 4 kHz, and 8 kHz (this is equivalent to dividing the frequency into 0-1000, 1000-4000, 4000-8000, and 8000-16000 Hz). This division may be performed as a Short-Time Discrete Fourier Transform (STFT), band-pass filtering in the Fourier domain (by applying a frequency mask), and an inverse transform with added overlap. In this case, the subbands are still sampled at the same original frequencies, and the processing operations according to the present invention are applied in the time domain. In some variations, a filter bank may be used for critical sampling. Note that the subband division operation generally involves processing delays depending on the type of filter bank implemented. According to the present invention, temporal alignment may be applied before and after encoding / decoding and / or before extraction of spatial image information so that the spatial image information is temporally synchronized with the corrected signal.

[0097] In some variations, full-band processing may be performed, or the sub-band division may differ as described above.

[0098] In other variations, the signal obtained from the conversion of the original multi-channel audio signal is used directly, and the present invention is applied to the converted region along with the division of the converted region into subbands.

[0099] In the following description, the various encoding and decoding steps described above are written to involve processing operations in the (real or complex) time or frequency domain with a single frequency band, for the sake of brevity.

[0100] Furthermore, it is also possible to optionally perform high-pass filtering in each subband in the form of a second-order elliptic IIR filter (typically with a cutoff frequency of 20 or 50 Hz), for example, with a cutoff frequency preferably set to 20 or 50 Hz (50 Hz in some variant examples). This preprocessing avoids potential bias in subsequent covariance estimation during encoding. Without this preprocessing, the modifications performed in block 390, described later, tend to amplify low frequencies while full-band processing is being performed.

[0101] Block 321 determines the information (Inf.B) that represents the spatial image of the original multi-channel signal.

[0102] In one embodiment, this information is energy information associated with the direction from which the sound was emitted (associated with the direction of virtual speakers distributed on a unit sphere).

[0103] For this purpose, a virtual 3D sphere with a unit radius is defined, and this 3D sphere is oriented in the direction of the nth speaker (Θ n ,φ n The signal is discretized by N points ("point" virtual speakers) whose positions are defined in spherical coordinates. The speakers are typically (quasi)uniformly arranged on the surface of a sphere. The number of virtual speakers N is determined as a discretization with at least N=K points, where M is the ambisonic order of the signal and K=(M+1) 2That is, N≧K. For example, this discretization can be performed using the Lebedev quadrature method, according to references: VI Lebedev, and DNLaikov, "A quadrature formula for the sphere of the 131st algebraic order of accuracy", Doklady Mathematics, vol.59, no.3, 1999, pp.477-481, or Pierre Lecomte, Philippe-Aubert GAUthier, Shristophe Langrenne, Alexandre Garcia and Alain Berry, On the use of a Lebedev grid for Ambisonics, AES Convention 139, New York, 2015.

[0104] In some variations, other discretization methods may be used, such as the Fliege discretization with at least N=K points (N≧K) as described in the reference: J. Fliege and U. Maier, “A two-stage approach for computing cubature formulae for the sphere”, Technical Report, Dortmund University, 1999, or the discretization using points from a “spherical t-design” as described in the paper “Mclaren's Improved Snub Cube and Other New Spherical Designs in Three Dimensions”, Discrete and Computational Geometry, 15(1996), pp.429-441 by RH Hardin and NJASloane.

[0105] From the discretization described above, the spatial image of the multichannel signal can be determined. One possible method is the SRP (Steered-Response Power) method. In fact, this method calculates the short-term energies coming from various directions defined with respect to azimuth and elevation angles. For this purpose, as described above, a weight matrix of ambisonic components is calculated, similar to rendering for N speakers, and then this matrix is ​​applied to the multichannel signal to sum the contributions of the components and generate a set of N audio beams (or "beamformers").

[0106] Direction of the nth speaker (Θ n ,φ n The signal from the acoustic beam to s n =d n Given by .B, where d n is a weight (row) vector that gives the acoustic beamforming coefficient for a given direction, and B is a matrix of size K × L representing an ambisonic signal (B format) with K components over a time width of length L.

[0107] The equation S=DB is derived from the set of signals from N acoustic beams. Here,

number

[0108] Each direction (Θ n ,φ n The short-term energy over a time interval of length L at ) is given by the following equation: σ n 2 =s n .s n T =(d n .B).(d n .B) T =d n .BB T .d n T =dn .Cd n T Here C=BB T (In the case of real numbers) or Re(BB H )(in the case of complex numbers) is the covariance matrix of B.

[0109] Each item σ n 2 =s n .s n T This corresponds to all directions (Θ) of the discretization of a 3D sphere by a virtual speaker. n ,φ n ) can be calculated in this way.

[0110] The spatial image Σ is given by the following equation. Σ=[σ0 2 ,…,σ N-1 2 ] In addition to the SRP method, modified examples for calculating spatial images Σ may be used. -value d n This may vary depending on the type of acoustic beamforming used (total delay, MVDR, LCMV, etc.). The present invention also relates to matrix D and spatial image Σ=[σ0 2 ,…,σ N-1 2 ] This can also be applied to these variant examples that calculate [the result]. -MUSIC (Multiple Signal Classification) method also provides another way to compute spatial images using a subspace scheme.

[0111] The present invention also relates to spatial images Σ=[σ0 2 ,…,σ N-1 2 ] This can be applied to the modified example used to calculate the following: This is calculated by diagonalizing the covariance matrix, and the direction (Θ n ,φ n This corresponds to the MUSIC pseudospectrum evaluated against ). - The spatial image can be calculated from, for example, the histogram of the (primary) intensity vectors in the paper "Direction estimation based on sound intensity vectors" by S. Tervo, Proc. EUSIPCO, 2009, or from a generalization to pseudo-intensity vectors. In this case, the histogram (taking as the value the number of occurrences of the direction of arrival values in a given direction (Θ n , φ n )) is interpreted as a set of energies in a given direction.

[0112] Block 330 then quantizes the spatial image thus determined, for example, by scalar quantization to 16 bits per coefficient (by directly using the floating-point representation truncated to 16 bits). In some variants, other scalar or vector quantization methods are also possible.

[0113] In another embodiment, the information representing the spatial image of the original multi-channel signal is the (sub-band) covariance matrix of input channel B. This matrix is (in the case of real numbers) calculated as C = B.B within the range of the normalization coefficient T .

[0114] When the present invention is executed in the complex-valued transform domain, this covariance is calculated as C = Re(B.B H ) within the range of the normalization coefficient.

[0115] In some variants, an operation for temporally smoothing the covariance matrix may be used. In the case of multi-channel signals in the time domain, the covariance can be recursively estimated (one sample at a time).

[0116] Since the covariance matrix C (of size K×K) is symmetric by definition, only one of the lower or upper triangular matrices is sent to the quantization block 330 that encodes (Q)K(K + 1) / 2 coefficients, where K is the number of ambisonic components.

[0117] This block 330 quantizes these coefficients (by directly using a 16-bit truncated floating-point representation) for example, by 16-bit scalar quantization for each coefficient. In some variations, other methods of scalar or vector quantization of the covariance matrix can be performed. For example, the maximum value (maximum variance) of the covariance matrix can be calculated, and then scalar quantization can be performed with logarithmic steps to a smaller number of bits (e.g., 8 bits) to normalize the values ​​of the upper (or lower) triangular matrix of the covariance matrix by its maximum value.

[0118] In some variant examples, the covariance matrix C can be regularized before being quantized in the form C+εI.

[0119] The quantized values ​​are sent to the multiplexer 340.

[0120] In this embodiment, the decoder receives a bitstream in the demultiplexer block 350 that includes an encoded audio signal obtained from the original multichannel signal and information representing the spatial image of the original multichannel signal.

[0121] Block 360 decodes the covariance matrix or other information representing the spatial image of the original signal (Q -1 Block 370 decodes (DEC) the audio signal represented by the bitstream.

[0122] In one embodiment of encoding and decoding that does not perform downmix and upmix steps, the decoded multichannel signal

number

[0123] In an embodiment using a downmix step for encoding, the decoded audio signal sent to the input of the upmix block 371 by decoding performed in block 370

number

[0124] Block 371 therefore performs an optional step (UPMIX) to increase the number of channels. In one embodiment of this step, a monaural signal

number

number

number

[0125] Block 372 performs an optional step (SB) to divide the signal into subbands in order to obtain subbands in either the time domain or the transformed domain. The inverse transform step, in block 391, aggregates the subbands to reconstruct the multi-channel signal at the output.

[0126] Block 375 represents the spatial image of the decoded multichannel signal, similar to how it was described in Block 321 (with respect to the original multichannel signal).

number

number

[0127] As described in block 321, in one embodiment, this information is energy information associated with the direction from which the sound was emitted (associated with the direction of a virtual speaker distributed on a unit sphere). As described above, the spatial image of the decoded multichannel signal can be determined using the SRP method (etc.).

[0128] In another embodiment, this information is the channel covariance matrix of the decoded multichannel signal.

[0129] This covariance matrix is ​​therefore obtained as follows: that is, within the range of the normalization coefficients

number

number

[0130] In some variant examples, an operation to smooth the covariance matrix in time may be used. For multi-channel signals in the time domain, the covariance can be estimated recursively (one sample at a time).

[0131] Original multi-channel signal (Inf.B) and decoded multi-channel signal (Inf.

number

number

[0132] Two specific embodiments of this determination will be described with reference to Figures 4 and 5.

[0133] In the embodiment shown in Figure 4, a method using (explicit or implicit) rendering in the virtual speaker is used, while in the embodiment shown in Figure 5, a method based on Cholesky factorization is used.

[0134] Block 390 in Figure 3 performs CORR (Correction of Decoded Multichannel Signal) on the decoded multichannel signal using the set of corrections determined in Block 380 in order to obtain the corrected decoded multichannel signal.

[0135] Figure 4, therefore, shows one embodiment of the step of determining the set of modifications. This embodiment is performed using rendering in a virtual speaker.

[0136] In this embodiment, the information representing the spatial images of the original multichannel signal and the decoded multichannel signal is represented by their respective covariance matrices C and

number

[0137] In this case, blocks 420 and 421 determine the spatial images of the original multichannel signal and the decoded multichannel signal, respectively.

[0138] For this purpose, as mentioned above, the direction of the nth speaker (Θ n ,φ n A virtual 3D sphere with a unit radius, whose direction in spherical coordinates is defined by ), is discretized by N points ("point" virtual speakers).

[0139] Multiple discretization methods were defined above.

[0140] From the discretization described above, the spatial image of the multichannel signal can be determined. As mentioned above, one possible method is the SRP method (etc.), which calculates the short-term energy coming from various directions defined with respect to azimuth and elevation angles.

[0141] Using this method or another type of method listed above, for the original multi-channel signal (IMGB) at 420 and the decoded multi-channel signal (IMG

Number

Number

Number

[0142] If the information (InfB) representing the spatial image of the original signal received and decoded by the decoder at 360 is the spatial image itself, that is, energy information (or positive values) associated with the direction in which the sound is emitted (associated with the directions of virtual speakers distributed on the unit sphere), there is no need to calculate this at 420 anymore. This spatial image is then directly used in block 430 described below.

[0143] Similarly, if the determination at 375 of the information representing the spatial image of the decoded multi-channel signal (Inf

Number

[0144] From the spatial images Σ and

Number

number

number

[0145] The energy ratio is directional (Θ n ,φ n ) and depending on the frequency band, it can be extremely large. Block 440 provides the gain g n The maximum value that can be taken is arbitrarily restricted (restriction g) n ) becomes possible. n 2 and

number

[0146] In one possible embodiment, g n A threshold is applied to the value. Any value greater than the threshold is forced to be equal to the threshold. The threshold may be set to 6dB, for example, so that the gain values ​​outside the ±6dB range saturate at ±6dB.

[0147] This gain g n The set of modifiers, therefore, constitutes the set of modifications to be applied to the decoded multi-channel signal.

[0148] This set of gains is received at the input side of the modification block 390 in Figure 3.

[0149] A correction matrix that can be directly applied to a decoded multichannel signal is, for example, of the form G=E.diag([g0...g N-1]).D can be defined as, where D and E are the acoustic decoding and coding matrices defined above. This matrix G is the modified output ambisonic signal (

number

number

[0150] The breakdown of the steps performed for correction is described here. Block 390 is the corresponding predetermined gain g n This is applied to each virtual speaker. By applying this gain, it becomes possible to obtain the same energy as the original signal through that speaker.

[0151] The rendering of the decoded signal at each speaker is modified in this way.

[0152] The acoustic coding step, for example, ambisonic coding using matrix E, is then performed to obtain components of the multichannel signal, such as ambisonic components. These ambisonic components are used to modify the output multichannel signal.

number

[0153] In some variant examples, the covariance matrix of the encoded and then decoded multichannel signal.

number

Number

[0154] Only the value of the first coefficient R of the matrix R corresponding to the omnidirectional component (W channel) is applied to R as a normalization coefficient and retained to avoid an overall gain increase due to the correction matrix G. 00

Number

Number

Number

[0155] In some variants, it is sufficient to calculate only a subset of the matrix elements to determine R 00 (and thus g norm ), so the normalization coefficient g norm can be determined without calculating the entire matrix R.

[0156] The matrix G or G norm obtained in this way corresponds to a set of corrections to be applied to the decoded multi-channel signal.

[0157] Here, FIG. 5 shows another embodiment of a method for determining the set of corrections performed in block 380 of FIG. 3.

[0158] In this embodiment, information representing the spatial images of the original multi-channel signal and the decoded multi-channel signal is covariance matrix C and

Number

[0159] In this embodiment, rendering is not performed for virtual speakers to correct the spatial image of the multi-channel signal. In particular, for an ambisonic signal, correction of the spatial image is directly calculated within the ambisonic domain.

[0160] For this purpose, the converted matrix T is applied to the decoded signal

Equation

[0161] Therefore, what is sought is the following equation

Equation

Equation

Equation

[0162] In this embodiment, the above equation is solved using a factorization known as Cholesky factorization.

[0163] Given a matrix A of size n×n, the Cholesky factorization determines a (lower or upper) triangular matrix L such that A = LL <o T (in the real case), A = LL H (in the complex case). For the factorization to be possible, the matrix A must be a positive definite symmetric matrix (in the real case) or a positive definite Hermitian matrix (in the complex case), and in the real case, the diagonal coefficients of L are strictly positive.

[0164] For real numbers, a matrix M of size n×n can be said to be positive definite symmetric if it is symmetric (M T =M) and positive definite (

number

[0165] For a symmetric matrix M, it can be verified that the matrix is ​​positive definite if all its eigenvalues ​​are strictly positive (λ i This is the case when >0). The eigenvalue is positive (λ i If the matrix is ​​≥ 0, it is said to be a positive semi-definite matrix.

[0166] A matrix M of size n×n is said to be positive definite symmetric Hermitian if it is Hermitian (M H =M) and positive definite (

number

[0167] Cholesky factorization is used, for example, to find solutions to a system of linear equations of the form Ax = b. For example, in the case of complex numbers, Cholesky factorization can be used to find the solution of A. H Convert to and solve Ly=b, then L H It is possible to solve x=y.

[0168] Similarly, Cholesky factorization is A=U T U (for real numbers) and A = U H It can be written as U (for complex numbers), where U is an upper triangular matrix.

[0169] In the embodiments described here, we will only consider the case of Cholesky factorization by a triangular matrix L without losing generality.

[0170] Cholesky factorization, therefore, is given by matrix C = LL, provided that matrix C is positive definite symmetric. TThis makes it possible to decompose it into two triangular matrices. This yields the following equation.

number

[0171] Using identifiers

number

[0172] In other words

number

[0173] covariance matrix C and

number

[0174] It should be noted here that matrix L and

number

[0175] Block 510 therefore forces the covariance matrix C to be positive definite. For this purpose, to ensure that the matrix is ​​indeed positive definite, the value ε is added to the diagonal coefficients of the matrix (Fact. C is C for factorization). That is, C = C + εI, where ε is, for example, 10 -9 It is a small value set to and I is the identity matrix.

[0176] Similarly, block 520 has a matrix

number

number

[0177] Two covariance matrices C and

number

number

[0178] In some variant examples, alternative solutions may be implemented by decomposition into eigenvalues.

[0179] Eigenvalue decomposition ("eigenvalue decomposition") is the process of factorizing a real or complex matrix A of size n × n into the following form: A=QΛQ -1 Here, Λ is the eigenvalue λ. i It is a diagonal matrix containing the eigenvectors, where Q is the matrix of eigenvectors.

[0180] If the matrix is ​​made up of real numbers, the following equation holds: A=QΛQ T

[0181] For complex numbers, the decomposition is A = QΛQ H It is written as follows.

[0182] In this case, the next thing to look for is

number

number

number

[0183] Find the following expression using the identifier.

number

[0184] In other words, the following equation holds true.

number

[0185] The stability of interframe solutions is typically not as good as when using the Cholesky factorization method. This instability is exacerbated by further computational approximations that can potentially be amplified during the decomposition into eigenvalues.

[0186] In some variations, the diagonal matrix is ​​given by the following equation:

number

number

number

[0187] In this embodiment, the relative energy difference between the decoded ambisonic signal and the modified ambisonic signal can be extremely large, particularly in the case of high frequencies, which can be significantly degraded by the encoder, as in multi-mono EVS coding. A regularization term may be added to avoid excessive amplification of specific frequency ranges. Block 640 optionally plays the role of normalizing (Norm.T) the modification.

[0188] In a preferred embodiment, the normalization coefficient is calculated so as not to amplify the frequency range.

[0189] Covariance matrix of a multi-channel signal after encoding and decoding.

number

number

[0190] The first coefficient R of matrix R corresponds to the omnidirectional component (W channel) 00 Only the value of is retained in order to apply it to T as a normalization coefficient and to avoid the increase in total gain due to the modification matrix T.

number

number

number

[0191] In some variant examples, R 00 (Therefore, g normSince it is sufficient to calculate only a subset of matrix elements to determine the normalization coefficient g, norm This can be determined without computing the entire matrix R.

[0192] T or T obtained in this way norm The matrix corresponds to a set of modifications applied to the decoded multi-channel signal.

[0193] In this embodiment, block 390 in Figure 3 is a modified output ambisonic signal (

number

[0194] A second embodiment of the encoder / decoder according to the present invention is described below, wherein the method for determining the set of modifications is performed in the encoder. This embodiment is illustrated in Figure 6, which therefore shows a second embodiment of an encoding and decoding device that performs an encoding and decoding method including a method for determining the set of modifications as described above with respect to Figure 2.

[0195] In this embodiment, the encoder performs a method for determining a set of modifications (e.g., gain associated with direction) and then transmits the set of modifications to the decoder. The decoder decodes the set of modifications for application to the decoded multichannel signal. This embodiment therefore includes performing local decoding in the encoder, which is represented by blocks 612-613.

[0196] Blocks 610, 611, 620, and 621 are identical to blocks 310, 311, 320, and 321, respectively, as described with reference to Figure 3.

[0197] The information representing the spatial image of the original multi-channel signal (Inf.B) is therefore obtained on the output side of block 621.

[0198] Block 612 performs local decoding (DEC_loc) similar to the encoding performed in block 611.

[0199] This local decoding may include, or preferably may be integrated with, block 611, a complete decoding from the bitstream.

[0200] In one embodiment of encoding and decoding that does not perform downmix and upmix steps, the decoded multichannel signal

number

[0201] In an embodiment where the downmix step at 610 is used for encoding, the decoded audio signal is sent to the input of the upmix block 613 by local decoding performed at block 612.

number

[0202] Block 613 then performs an optional step (UPMIX) to increase the number of channels. In one embodiment of this step, this is a monaural signal

number

number

number

[0203] Block 614 performs an optional step (SB) to divide the time domain or the transformed domain into subbands in order to obtain subbands in either the time domain or the transformed domain.

[0204] Block 615 represents the spatial image of the decoded multichannel signal (Inf

number

number

[0205] In a manner similar to blocks 621 and 321, in one embodiment, this information is energy information associated with the direction from which the sound was emitted (associated with the direction of a virtual speaker distributed on a unit sphere). As described above, the spatial image of the decoded multichannel signal can be determined using an SRP method (such as the modified example above).

[0206] In another embodiment, this information is the channel covariance matrix of the decoded multichannel signal.

[0207] This covariance matrix is ​​then obtained as follows: That is, (in the case of real numbers) within the range of normalization coefficients

number

number

[0208] Original multi-channel signal (Inf.B) and decoded multi-channel signal (Inf.

number

number

[0209] Two specific embodiments of this determination are possible, and have been described with reference to Figures 4 and 5.

[0210] In the embodiment shown in Figure 4, a method using rendering in the speaker is employed, while in the embodiment shown in Figure 5, a method performed directly in the ambisonic region and based on Cholesky factorization or decomposition into eigenvalues ​​is employed.

[0211] Therefore, if the embodiment of Figure 4 is applied to 630, the determined set of modifications is the set of directions defined by the set of virtual speakers (Θ n ,φ n Set of gains g for ) n This set of gains can be determined in the form of a modification matrix G, as described with reference to Figure 4. This set of gains (corr.) is then coded with 640. The coding of this set of gains is the modification matrix G or G norm It may be something that encodes.

[0212] A matrix G of size K × K is symmetric, and therefore, according to the present invention, G or G normNote that only the lower or upper triangular matrix, i.e., K × (K+1) / 2 values, can be encoded. Generally, the values ​​of the diagonal terms are positive. In one embodiment, matrix G or G norm The values ​​are encoded using scalar quantization (with or without a sign bit) depending on whether the value is an off-diagonal term or not. norm In several variations using G norm Since the first value of the diagonal term (corresponding to the omnidirectional component) is always 1, its encoding and transmission can be omitted. For example, in the case of first-order ambisonic with K=4 channels, this is equivalent to transmitting only 9 values ​​instead of K×(K+1) / 2=10 values. In some variations, other scalar or vector quantization methods (with or without prediction) may be used.

[0213] If the embodiment in Figure 5 is applied at 630, the determined set of modifications is the transformation matrix T or T norm It is then encoded with 640.

[0214] Note that the matrix T of size K×K is a triangular matrix in the variant using Cholesky factorization, and a symmetric matrix in the variant using eigenvalue decomposition. Therefore, according to the present invention, T or T norm Only the lower or upper triangular matrix, i.e., K × (K+1) / 2 values, can be encoded.

[0215] Generally, the values ​​of the diagonal terms are positive. In one embodiment, matrix T or T norm The values ​​are encoded using scalar quantization (with or without a sign bit) depending on whether they are off-diagonal terms or not. In some variant examples, other scalar or vector quantization methods (with or without prediction) may be used. norm In the variant using T norm Since the first value of the diagonal term (corresponding to the omnidirectional component) is always 1, its encoding and transmission can be omitted. For example, in the case of first-order ambisonic with K=4 channels, this is equivalent to transmitting only 9 values ​​instead of K×(K+1) / 2=10 values.

[0216] Block 640 then encodes the determined set of modifications and sends the encoded set of modifications to the multiplexer 650.

[0217] The decoder receives a bitstream in the demultiplexer block 660 containing the encoded audio signal obtained from the original multichannel signal and a set of encoded modifications to be applied to the decoded multichannel signal.

[0218] Block 670 decodes the coded set of modifications (Q -1 Block 680 decodes (DEC) the encoded audio signal received in the stream.

[0219] In one embodiment of encoding and decoding that does not perform downmix and upmix steps, the decoded multichannel signal

number

[0220] In an embodiment using a downmix step for encoding, the decoded audio signal sent to the input of the upmix block 681 is decoded by the decoding performed in block 680.

number

[0221] Block 681 then performs an optional step (UPMIX) to increase the number of channels. In one embodiment of this step, a monaural signal

number

number

number

[0222] Block 682 performs an optional step (SB) to divide the signal into subbands to obtain either a subband in the time domain or in the transformed domain, and Block 691 groups the subbands to reconstruct the output multichannel signal.

[0223] Block 690 is a modified decoded multichannel signal correction (

number

[0224] In one embodiment, the set of corrections is a set of gains as described with reference to Figure 4, and this set of gains is received on the input side of the correction block 690. The set of gains is, for example, G=E.diag([g0...g N-1 ]).D or G norm =g norm If it is a form of a modification matrix that can be directly applied to a decoded multichannel signal defined in the form of .G, then this matrix G or G norm Next, the corrected output ambisonic signal (

number

number

[0225] Block 690 is a set of gains g nIf it receives, block 690 has the corresponding gain g n This is applied to each virtual speaker. By applying this gain, it becomes possible to obtain the same energy as the original signal through that speaker.

[0226] The rendering of the decoded signals for each speaker is modified in this way.

[0227] An acoustic coding step, such as ambisonic coding, is then performed to obtain components of the multichannel signal, such as ambisonic components. These ambisonic components are ultimately used to modify the output multichannel signal.

number

[0228] As described with reference to Figure 5, in one embodiment where the set of modifications is a transformation matrix, the transformation matrix T decoded at 670 is received on the input side of the modification block 690.

[0229] In this embodiment, block 690 is the modified output ambisonic signal (

number

[0230] Although the present invention is applicable to ambisonic formats, in some variations, other formats (multichannel, object, etc.) can be converted to ambisonic in order to apply the methods performed by the various embodiments described above. An exemplary embodiment of such a conversion from a multichannel or object format to an ambisonic format is described in Figure 2 of the 3GPPTS26.259 specification (v15.0.0).

[0231] Figure 7 shows an encoding device DCOD and a decoding device DDEC within the scope of the present invention, which are redundant (in the sense of "reversible") and connected to each other by a communication network RES.

[0232] The encoding device DCOD typically includes processing circuits that include the following: -Memory MEM1 for storing computer program instruction data within the scope of the present invention (these instructions may be distributed between encoder DCOD and decoder DDEC), - An interface INT1 receives the original multi-channel signal B, for example, an ambisonic signal distributed across various channels (e.g., four primary channels W, Y, Z, X), with the intention of compressing and encoding it within the scope of the present invention. - A processor PROC1 that processes the signal by executing computer program instructions stored in memory MEM1 with the intention of receiving and encoding the signal, - COM1 is a communication interface that transmits encoded signals over the network.

[0233] The DDEC (Decoder Decoder) typically includes its own processing circuitry, which includes the following: -Memory MEM2 for storing computer program instruction data within the scope of the present invention (these instructions may be distributed between the encoder DCOD and decoder DDEC as described above), - Within the scope of the present invention, an interface COM2 receives encoded signals from the network RES with the intention of compressing and decoding them. -Processor PROC2 processes these signals by executing computer program instructions stored in memory MEM2, with the intention of decoding them. - Modified decoded signal (

number

[0234] Of course, Figure 7 shows an example of a structural embodiment of a codec (encoder or decoder) within the scope of the concept of the present invention. Figures 3 to 6 described above detail more functional embodiments of these codecs.

Claims

1. 1. A method for determining a set of modifications (Corr.) to be applied to a multi-channel audio signal, said set of modifications being derived from information representative of a spatial image of the original multi-channel signal (Inf. B) and from information representative of the spatial image of the original multi-channel signal which has been coded and then decoded (Inf. [0010] ) is determined from the determination method.

2. The method of claim 1 , wherein the set of modifications is determined by frequency subbands.

3. receiving (350) an encoded audio signal from an original multi-channel signal and a bitstream containing information representative of a spatial image of said original multi-channel signal; Decoding (370) the received encoded audio signal to obtain a decoded multi-channel signal; Decoding (360) information representative of a spatial image of the original multi-channel signal; determining (375) information representative of a spatial image of the decoded multi-channel signal; - determining (380) a set of modifications to be applied to the decoded signal using a method according to claim 1 or 2; modifying (390) the decoded multi-channel signal using the determined set of modifications; 23. A method for decoding a multi-channel audio signal, comprising:

4. A step of encoding (611) an audio signal from an original multi-channel signal; - determining (621) information representative of a spatial image of the original multi-channel signal; locally decoding (612) the encoded audio signal to obtain a decoded multi-channel signal; determining (615) information representative of a spatial image of the decoded multi-channel signal; - determining (630) a set of modifications to be applied to the decoded multi-channel signal using a determination method according to claim 1 or 2; encoding (640) the set of determined modifications; 23. A method for encoding a multi-channel audio signal, comprising:

5. The information representative of the aerial image is a covariance matrix, and the step of determining the set of corrections further comprises: obtaining a weight matrix comprising weight vectors associated with the set of virtual speakers; determining a spatial image of the original multi-channel signal from the obtained weighting matrix and from the covariance matrix of the original multi-channel signal; determining a spatial image of the decoded multi-channel signal from the obtained weighting matrix and from the determined covariance matrix of the decoded multi-channel signal; calculating a ratio between the spatial image of the original multi-channel signal and the spatial image of the decoded multi-channel signal in the direction of a speaker of the set of virtual speakers to obtain a set of gains; The method of decoding according to claim 3 , comprising:

6. The information representative of the aerial image is a covariance matrix, and the step of determining the set of corrections further comprises: obtaining a weight matrix comprising weight vectors associated with the set of virtual speakers; determining a spatial image of the original multi-channel signal from the obtained weighting matrix and from the covariance matrix of the original multi-channel signal; determining a spatial image of the decoded multi-channel signal from the obtained weighting matrix and from the determined covariance matrix of the decoded multi-channel signal; calculating a ratio between the spatial image of the original multi-channel signal and the spatial image of the decoded multi-channel signal in the direction of a speaker of the set of virtual speakers to obtain a set of gains; The encoding method of claim 4, comprising:

7. The received information representative of a spatial image of the original multi-channel signal is the spatial image of the original multi-channel signal, and the step of determining the set of modifications further comprises: obtaining a weight matrix comprising weight vectors associated with the set of virtual speakers; determining a spatial image of the decoded multi-channel signal from the obtained weighting matrix and from the determined information representative of a spatial image of the decoded multi-channel signal; calculating a ratio between the spatial image of the original multi-channel signal and the spatial image of the decoded multi-channel signal in the direction of a speaker of a set of virtual speakers to obtain a set of gains; The method of decoding according to claim 3 , comprising:

8. 4. The method of claim 3, wherein the information representative of the spatial image is a covariance matrix, and the step of determining the set of modifications comprises the step of determining a transformation matrix via a matrix decomposition of two covariance matrices, the transformation matrix constituting the set of modifications.

9. 5. The method of claim 4, wherein the information representative of the spatial image is a covariance matrix, and the step of determining the set of modifications comprises the step of determining a transformation matrix via a matrix decomposition of two covariance matrices, the transformation matrix constituting the set of modifications.

10. 9. A method for decoding according to claim 3, 5, 7 or 8, wherein the decoded multi-channel signal is modified by a set of modifications determined by applying the set of modifications to the decoded multi-channel signal.

11. the decoded multi-channel signal is decoded according to the determined set of modifications; acoustically decoding the decoded multi-channel signal with the defined set of virtual speakers; applying the obtained set of gains to a signal resulting from the acoustic decoding; acoustically encoding a modified signal resulting from said acoustic decoding to obtain components of said multi-channel signal; summing said components of the multi-channel signal thus obtained to obtain a modified multi-channel signal; 8. The method of decoding according to claim 5 or 7, modified by:

12. receiving a bitstream comprising an encoded audio signal from an original multi-channel signal and an encoded set of modifications to be applied to the decoded multi-channel signal, the modification set being encoded using the encoding method of claim 6; decoding the received encoded audio signal to obtain a decoded multi-channel signal; decoding the set of encoded modifications; the decoded multi-channel signal, acoustically decoding the decoded multi-channel signal with a set of virtual speakers; - applying a set of gains obtained to the signal obtained from said acoustic decoding; - acoustically encoding modified signals resulting from said acoustic decoding to obtain components of said multi-channel signal; - summing said components of the multi-channel signal thus obtained to obtain a modified multi-channel signal; and modifying the decoded set of modifications in 23. A method for decoding a multi-channel audio signal, comprising:

13. 13. A decoding device comprising processing circuitry for carrying out a decoding method according to any one of claims 3, 5, 7, 8 or 10 to 12.

14. Encoding device comprising processing circuitry for carrying out the encoding method according to any one of claims 4, 6 or 9.

15. A processor-readable storage medium having stored thereon a computer program comprising instructions for carrying out the decoding method according to any one of claims 3, 5, 7, 8 or 10 to 12.

16. A processor-readable storage medium having stored thereon a computer program comprising instructions for carrying out the encoding method of any one of claims 4, 6 or 9.

Images