Determine the corrections to be applied to multichannel audio signals, and the associated encoding and decoding

Abstract

The present invention relates to a method for determining a set of corrections (Corr.) to be performed on a multi-channel sound signal, wherein the set of corrections is determined based on information (Inf.B) representing a spatial image of an original multi-channel signal and information (Inf.B) representing a spatial image of the original multi-channel signal that has been encoded and then decoded. The present invention also relates to a decoding method and an encoding method implementing the determination method, and to associated encoding devices and decoding devices.

Description

Technical Field

[0001] The present invention relates to encoding / decoding of spatialized sound data, in particular in the context of ambisonics (hereinafter also denoted "ambisonic"). Background Art

[0002] The coders / decoders (hereinafter referred to as "codecs") currently used in mobile phones are monophonic (a single signal channel to be presented on a single speaker). The 3GPP EVS ("Enhanced Voice Services") codec enables the provision of "Ultra-HD" sound quality (also known as "High Definition Plus" or HD+ Voice), wherein a Super Wideband (SWB) audio band is provided for signals sampled at 32 kHz or 48 kHz or a Full Band (FB) audio band is provided for signals sampled at 48 kHz; the audio bandwidth is 14.4 kHz to 16 kHz in SWB mode (9.6 kbit / s to 128 kbit / s) and 20 kHz in FB mode (16.4 kbit / s to 128 kbit / s).

[0003] The next quality evolution in conversational services provided by operators should consist of immersive services using terminals such as smartphones equipped with multiple microphones, or telepresence or 360° video spatialized audio conferencing or video conferencing equipment, or even "live" audio content sharing equipment with spatialized 3D sound rendering that is more immersive than simple 2D stereo rendering. With the increasing adoption of listening on mobile phones with audio headsets and the emergence of advanced audio devices (such as 3D microphones, voice assistants with acoustic antennas, virtual reality headsets and other accessories), it is now common enough to capture and render spatialized sound scenes to provide an immersive communication experience.

[0004] To this end, the future 3GPP standard "IVAS" ("Immersive Voice and Audio Services") plans to extend the EVS codec for immersion by accepting at least the following listed spatialized sound formats (and their combinations) as codec input formats:

[0005] Stereo or 5.1 multichannel (channel-based) formats, where each channel feeds one speaker (e.g. L and R in stereo or L, R, Ls, Rs, and C in 5.1);

[0006] object-based formats, where a sound object is described as an audio signal (usually mono) associated with metadata describing the properties of the object (position in space, source spatial width, etc.),

[0007] An ambisonics (scene-based) format that describes the sound field at a given point, typically captured by spherical microphones or synthesized in the spherical harmonics domain.

[0008] The following typically focuses on encoding sound in an Ambisonics format via exemplary embodiments (at least some aspects presented below in connection with the present invention can also be applied to formats other than Ambisonics).

[0009] Ambisonics is a method for recording ("encoding" in the acoustic sense) spatialized sound and a system for reproducing ("decoding" in the acoustic sense). An Ambisonics microphone (1st order) comprises at least four capsules (typically cardioid or subcardioid) arranged on a spherical grid (e.g., the vertices of a regular tetrahedron). The audio channels associated with these capsules are referred to as "A-format". This format is converted to "B-format", in which the sound field is decomposed into four components (spherical harmonics) represented by W, X, Y, Z, which correspond to four coincident virtual microphones. Component W corresponds to the omnidirectional capture of the sound field, while the more directional components X, Y and Z are similar to pressure gradient microphones oriented along three orthogonal spatial axes. The Ambisonics system is a flexible system in the sense that recording and rendering are separate and distinct. It allows (acoustically) decoding of any configuration of speakers (e.g., binaural, 5.1 or 7.1.4 omnidirectional channels (with elevation) "surround" sound). The Ambisonics approach can be generalized to more than four channels in the B-format, and this generalized representation is often referred to as "HOA" ("Higher Order Ambisonics"). Decomposing the sound into more spherical harmonics improves the accuracy of the spatial rendering when rendered on loudspeakers.

[0010] An M-order Ambisonics signal comprises K=(M+1)2 components and in the 1st order (if M=1) there are four components W, X, Y and Z which are usually referred to as FOA (first order Ambisonics). There are also so-called "planar" Ambisonics variants (W, X, Y) which decompose the sound defined on a plane, usually a horizontal plane. In this case, the number of components is K=2M+1 channels. 1st order Ambisonics (4 channels: W, X, Y, Z), planar 1st order Ambisonics (3 channels: W, X, Y) and higher order Ambisonics are hereinafter indiscriminately referred to as "Ambisonics" for ease of reading, the processing operations presented being applicable independently of the planar or non-planar type and the number of Ambisonics components.

[0011] In the following, an "ambisonic signal" will be the name given to a predetermined order signal in B format having a certain number of ambidextrous components. This also includes the hybrid case where, for example, there are only 8 channels (instead of 9) in the 2nd order - more precisely, there are 4 1st order channels (W, X, Y, Z) plus usually 5 channels (usually denoted R, S, T, U, V) in the 2nd order, and one of the higher order channels (e.g. R) may for example be omitted.

[0012] The signal to be processed by the encoder / decoder is in the form of consecutive blocks of sound samples, hereinafter referred to as "frames" or "sub-frames".

[0013] In addition, in the following text, mathematical symbols follow the following conventions:

[0014] Scalar: s or N (lowercase for variables or uppercase for constants)

[0015] The operator Re(.) represents the real part of a complex number.

[0016] Vector: u (lowercase, bold)

[0017] Matrix: A (uppercase, bold)

[0018] Symbol A T and A H Indicates the transpose and Hermitian transpose (transpose and conjugate) of A, respectively.

[0019] The one-dimensional discrete-time signal s(i) defined in the time interval i=0,…,L-1 of length L is represented by the following row vector

[0020] s=[s(0),...,s(L-1)].

[0021] It can also be written as: s=[s0,…,s L-1 ] to avoid using parentheses.

[0022] A multidimensional discrete-time signal b(i) defined in a time interval i=0,...,L-1 of length L and having K dimensions is represented by the following matrix of size L×K:

[0023]

[0024] It can also be expressed as: B = [B ij ], i=0,…K-1, j=0…L-1 to avoid using parentheses.

[0025] 3D points with Cartesian coordinates (x,y,z) can be converted to spherical coordinates where r is the distance from the origin, θ is the azimuth and is the elevation angle. Without loss of generality, the mathematical convention of defining elevation angles relative to the horizontal plane (0xy) is used here; the invention can be easily adapted to other definitions, including the convention used in physics where azimuth angles are defined relative to the axis Oz.

[0026] Furthermore, the conventions known from prior art Ambisonics technology regarding the order of Ambisonics components (including ACN Ambisonics channel number, SID single index nomenclature, FuMA Furse-Malham) and the normalization of Ambisonics components (SN3D, N3D, maxN) are not hinted at here. More details can be found in resources available online, for example: https: / / en.wikipedia.org / wiki / Ambisonic_data_exchange_formats

[0027] By convention, the first component of an Hi-Fi stereo signal usually corresponds to the omnidirectional component W.

[0028] The simplest approach for encoding an high-fidelity stereo signal is to use a mono encoder and apply it in parallel to all channels, with possibly different bit allocations depending on the channel. This approach is referred to herein as "multi-mono". The multi-mono approach can be extended to multi-stereo encoding (where pairs of channels are encoded separately by a stereo codec) or more generally to multiple similar instances using the same core codec.

[0029] Such an embodiment is shown in Figure 1. The input signal is divided into channels (a mono channel or multiple channels) by block 100. The channels are individually encoded based on a predetermined profile and bit allocation by blocks 120 to 122. The bitstreams of the channels are multiplexed (block 130) and after transmission and / or storage, the bitstreams are demultiplexed (block 140) to apply decoding to reconstruct the decoded channels (blocks 150 to 152), which are reassembled (block 160).

[0030] The associated quality varies depending on the core encoding and decoding used (blocks 120 to 122 and 150 to 152) and is generally satisfactory only at very high bit rates. For example, in the case of multi-mono, EVS encoding may be considered quasi-transparent (from a perceptual point of view) at a bit rate of at least 48 kbit / s per channel (mono); thus, for a 1st-order high-fidelity stereo signal, a minimum bit rate of 4×48=192 kbit / s is obtained. Since the multi-mono coding method does not take into account the inter-channel correlation, the method produces spatial deformations and various artifacts, such as the appearance of ghost sources, diffuse noise or displacement of source tracks. Therefore, encoding high-fidelity stereo signals using this method leads to a degradation of spatialization.

[0031] For stereo or multichannel signals, an alternative to encoding all channels separately is given by parametric coding. For this type of coding, the input multichannel signal is reduced to a smaller number of channels, which are encoded and transmitted after a processing operation called "downmixing", and additional spatialization information is also encoded. Parametric decoding consists in increasing the number of channels after decoding the transmitted channels using a processing operation called "upmixing" (typically implemented by decorrelation) and spatial synthesis based on the decoded additional spatialization information. 3GPP e-AAC + An example of stereo parametric coding is given in the codec . It should be noted that the downmix operation also results in a degradation of the spatialization; in this case, the spatial image is modified. Summary of the invention

[0032] The present invention aims to improve the prior art.

[0033] To this end, the present invention proposes a method for determining a set of corrections to be performed on a multi-channel sound signal, wherein the set of corrections is determined based on information representing a spatial image of an original multi-channel signal and information representing a spatial image of an original encoded and then decoded multi-channel signal.

[0034] Thus, the determined set of corrections to be applied to the decoded multi-channel signal enables limiting the spatial degradations due to the encoding and possible channel drop / addition operations. Thus, implementing the corrections enables restoring a spatial image of the decoded multi-channel signal that is closest to the spatial image of the original multi-channel signal.

[0035] In a specific embodiment, the set of corrections is determined in the full-band time domain (one frequency band). In some variants, the set of corrections is performed in the time domain by frequency sub-bands. This enables adaptation of the corrections according to the frequency band.

[0036] In other variants, the set of corrections is performed in the real or complex transform domain (typically the frequency domain) of the Short Time Discrete Fourier Transform (STFT), Modified Discrete Cosine Transform (MDCT) type, or the like.

[0037] The present invention also relates to a method for decoding a multi-channel sound signal, the method comprising the following steps:

[0038] receiving a bitstream comprising an encoded audio signal from an original multi-channel signal and information representing a spatial image of the original multi-channel signal;

[0039] Decoding the received encoded audio signal and obtaining a decoded multi-channel signal;

[0040] decoding information representing a spatial image of an original multi-channel signal;

[0041] determining information representing a spatial image of the decoded multi-channel signal;

[0042] Using the above determination method to determine a set of corrections to be performed on the decoded signal;

[0043] The decoded multi-channel signal is corrected using the determined set of corrections.

[0044] Thus, in this embodiment, the decoder is able to determine the correction to be performed on the decoded multi-channel signal from information received from the encoder representing the spatial image of the original multi-channel signal. The information received from the encoder is thus limited. The decoder is responsible for determining and applying the correction.

[0045] The present invention also relates to a method for encoding a multi-channel sound signal, the method comprising the following steps:

[0046] encoding an audio signal from an original multi-channel signal;

[0047] determining information representing a spatial image of an original multi-channel signal;

[0048] Locally decoding the encoded audio signal and obtaining a decoded multi-channel signal;

[0049] determining information representing a spatial image of the decoded multi-channel signal;

[0050] Using the above determination method to determine a set of corrections to be performed on the decoded multi-channel signal;

[0051] The determined set of corrections is encoded.

[0052] In this embodiment, the encoder determines a set of corrections to be performed on the decoded multi-channel signal and transmits the set of corrections to the decoder.

[0053] Therefore, the encoder initiates this correction determination.

[0054] In a first specific embodiment of the decoding method as described above or the encoding method as described above, the information representing the spatial image is a covariance matrix, and determining the set of corrections further comprises the following steps:

[0055] obtaining a weighting matrix including weighting vectors associated with a set of virtual loudspeakers;

[0056] Determine a spatial image of the original multi-channel signal according to the obtained weighting matrix and a covariance matrix of the received original multi-channel signal;

[0057] determining a spatial image of the decoded multi-channel signal according to the obtained weighting matrix and the determined covariance matrix of the decoded multi-channel signal;

[0058] A ratio between the spatial image of the original multi-channel signal and the spatial image of the decoded multi-channel signal in directions of speakers in the set of virtual speakers is calculated to obtain a set of gains.

[0059] According to this embodiment, this method of using the presentation on the loudspeaker enables only a limited amount of data to be transmitted from the encoder to the decoder. In fact, for a given order M, K=(M+1) to be transmitted 2 A coefficient (associated with the same number of virtual loudspeakers) may be sufficient, but for a more stable correction it may be advisable to use more virtual loudspeakers and therefore transmit more points. Furthermore, the correction can be easily understood from the perspective of the gains associated with the virtual loudspeakers.

[0060] In another variant embodiment, if the encoder directly determines the energy of the signals in different directions and transmits this spatial image of the original multi-channel signal to the decoder, determining the set of corrections for the decoding method further comprises the following steps:

[0061] obtaining a weighting matrix including weighting vectors associated with a set of virtual loudspeakers;

[0062] determining a spatial image of the decoded multi-channel signal according to the obtained weighting matrix and the information representing the determined spatial image of the decoded multi-channel signal;

[0063] A ratio between the spatial image of the original multi-channel signal and the spatial image of the decoded multi-channel signal in directions of speakers in the set of virtual speakers is calculated to obtain a set of gains.

[0064] In order to ensure that the correction value is not too extreme, the decoding method or the encoding method comprises a step of limiting the gain value obtained according to at least one threshold value.

[0065] The set of gains constitutes the set of corrections and may, for example, take the form of a correction matrix comprising the set of gains thus determined.

[0066] In a second particular embodiment of the decoding method or the encoding method, the information representative of the spatial image is a covariance matrix and determining the set of corrections comprises a step of determining a transformation matrix by matrix decomposition of two covariance matrices, the transformation matrix constituting the set of corrections.

[0067] This embodiment has the advantage that in the case of an Ambisonics multi-channel signal the correction is made directly in the Ambisonics domain. The step of transforming the signal rendered on the loudspeaker into the Ambisonics domain is thus avoided. This embodiment further enables the correction to be optimized so that it is optimal in a mathematical sense, even if it requires the transmission of a greater number of coefficients than the method for rendering on the loudspeaker. In fact, for an order M and therefore a number of components K=(M+1) 2 , the number of coefficients to be transmitted is K×(K+1) / 2.

[0068] In order to avoid over-amplification beyond certain frequency ranges, a normalization factor is determined and applied to the transformation matrix.

[0069] If the set of corrections is represented by a transformation matrix or correction matrix as described above, the decoded multi-channel signal is corrected by means of the determined set of corrections by applying the set of corrections to the decoded multi-channel signal, that is to say directly in the ambidextrous domain in case of an ambidextrous signal.

[0070] In an embodiment implemented by a decoder for rendering on a loudspeaker, the decoded multi-channel signal is corrected using the determined set of corrections in the following steps:

[0071] acoustically decoding the decoded multi-channel signal on a defined set of virtual loudspeakers;

[0072] applying the obtained set of gains to the signal produced by the acoustic decoding;

[0073] acoustically encoding the correction signal resulting from the acoustic decoding to obtain components of the multi-channel signal;

[0074] The components of the multi-channel signal thus obtained are summed to obtain a corrected multi-channel signal.

[0075] In a variant embodiment, the above decoding, applying gain and encoding / summing steps are combined together into a direct correction operation using a correction matrix. This correction matrix can be applied directly to the decoded multi-channel signal, which has the advantage of making the correction directly in the high-fidelity stereo domain as described above.

[0076] In a second embodiment in which the encoding method implements the method to determine the set of corrections, the decoding method comprises the following steps:

[0077] receiving a bitstream comprising an encoded audio signal from an original multi-channel signal and an encoded set of corrections to be performed on a decoded multi-channel signal, the set of corrections having been encoded using the encoding method described above;

[0078] Decoding the received encoded audio signal and obtaining a decoded multi-channel signal;

[0079] decoding the encoded set of corrections;

[0080] The decoded multi-channel signal is corrected by applying the decoded set of corrections to the decoded multi-channel signal.

[0081] In this embodiment, the encoder determines the corrections to be performed on the decoded multi-channel signal directly in the Ambisonics domain, and the decoder applies these corrections to the decoded multi-channel signal directly in the Ambisonics domain.

[0082] In this case, the set of corrections may be a transformation matrix or a correction matrix comprising a set of gains.

[0083] In a variant embodiment of the decoding method presented on a loudspeaker, the decoding method comprises the following steps:

[0084] receiving a bitstream comprising an encoded audio signal from an original multi-channel signal and an encoded set of corrections to be performed on a decoded multi-channel signal, the set of corrections having been encoded using a coding method as described above;

[0085] Decoding the received encoded audio signal and obtaining a decoded multi-channel signal;

[0086] decoding the encoded set of corrections;

[0087] The decoded set of corrections is used to correct the decoded multi-channel signal in the following steps:

[0088] acoustically decoding the decoded multi-channel signal on a defined set of virtual loudspeakers;

[0089] applying the obtained set of gains to the signal produced by acoustic decoding;

[0090] acoustically encoding the correction signal generated by the acoustic decoding to obtain components of the multi-channel signal;

[0091] The components of the multi-channel signal thus obtained are summed to obtain a corrected multi-channel signal.

[0092] In this embodiment, the encoder determines corrections to be made to the signals resulting from the acoustic decoding over a set of virtual loudspeakers, and the decoder applies these corrections to the signals resulting from the acoustic decoding and then transforms these signals back to the Ambisonics domain in the case of Ambisonics multi-channel signals.

[0093] In a variant embodiment, the above decoding, applying gain and encoding / summing steps are combined together into a direct correction operation using a correction matrix. The correction is then performed directly by applying the correction matrix to the decoded multi-channel signal (e.g., an Ambisonics signal). As described above, this embodiment has the advantage of making the correction directly in the Ambisonics domain.

[0094] The invention also relates to a decoding device comprising a processing circuit for implementing the decoding method as described above.

[0095] The invention also relates to a decoding device comprising a processing circuit for implementing the encoding method as described above.

[0096] The invention relates to a computer program comprising instructions for implementing a decoding method or an encoding method as described above when executed by a processor.

[0097] The invention finally relates to a storage medium which can be read by a processor and which stores a computer program comprising instructions for executing a decoding method or an encoding method as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] Other characteristics and advantages of the invention will become more clearly apparent after reading the following description of particular embodiments, given by way of simple illustrative and non-limiting examples, and the description of the accompanying drawings, in which:

[0099] FIG1 illustrates multi-mono coding according to the prior art and as described above;

[0100] Figure 2 The steps of a method for determining a set of corrections according to one embodiment of the present invention are presented in the form of a flow chart;

[0101] Figure 3 A first embodiment of an encoder and a decoder, an encoding method and a decoding method according to the present invention are presented;

[0102] Figure 4 A first detailed embodiment of a block for determining the set of corrections is presented;

[0103] Figure 5 A second detailed embodiment of a block for determining the set of corrections is presented;

[0104] Figure 6 A second embodiment of the encoder and decoder, the encoding method and the decoding method according to the present invention are presented; and

[0105] Figure 7 An example of a structural embodiment of an encoder and a decoder according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0106] The method described below is based on correcting spatial degradations, specifically in order to ensure that the spatial image of the decoded signal is as close as possible to the original signal. Unlike known parametric coding methods for stereo or multichannel signals that encode perceptual cues, the present invention is not based on a perceptual interpretation of the spatial image information, since the high-fidelity stereo domain is not directly "audible".

[0107] Figure 2 The main steps implemented to determine a set of corrections to be applied to an encoded and then decoded multi-channel signal are shown.

[0108] An original multi-channel signal B of size K x L (that is to say K components of L time or frequency samples) is at the input of the determination method. In step S1 , information representing the spatial image of the original multi-channel signal is extracted.

[0109] As mentioned above, the focus here is on the case of multi-channel signals with a high-fidelity stereo representation. The invention can also be applied to other types of multi-channel signals, such as B-format signals with modifications such as suppression of certain components (e.g. suppression of the 2nd order R components to keep only 8 channels) or matrixing of the B-format to an equivalent domain (called "equivalent spatial domain"), as described in the 3GPP TS 26.260 specification - another example of matrixing is given by the "Channel Mapping 3" of the IETF Opus codec and in the 3GPP TS 26.918 specification (clause 6.1.6.3).

[0110] The name "spatial image" given here refers to the distribution of the acoustic energy of a high-fidelity stereo sound scene in various directions in space; in some variants, this spatial image describing the sound scene typically corresponds to positive values ​​evaluated in various predetermined directions in space, for example in the form of a MUSIC (Multiple Signal Classification) pseudo-spectrum sampled in these directions or a histogram of arrival directions (where the arrival directions are counted according to a discretization given by predetermined directions); these positive values ​​can be interpreted as energies and as shown below to simplify the description of the invention.

[0111] Therefore, the spatial image associated with an ambidextrous sound scene represents the relative acoustic energy (or more generally positive values) as a function of the various directions in space. In the present invention, the information representing the spatial image may be, for example, a covariance matrix calculated between the channels of a multichannel signal or energy information associated with the directions from which the sounds originate (associated with the directions of virtual loudspeakers distributed on a unit sphere).

[0112] The set of corrections to be applied to the multi-channel signal is information that may be defined by a set of gains associated with the directions from which the sounds originate, which information may be in the form of a correction matrix comprising the set of gains or a transformation matrix.

[0113] The covariance matrix of the multi-channel signal B is obtained in step S1, for example. Figure 3 and Figure 6 As described, the matrix is ​​calculated as follows:

[0114] C=BB T to within the normalization factor (in the case of real numbers)

[0115] or

[0116] C=Re(BB H ) to be within the normalization factor (in the complex case).

[0117] In some variants, an operation that temporally smoothes the covariance matrix may be used. In the case of multi-channel signals in the time domain, the covariance may be estimated recursively (sample by sample) as follows:

[0118] Cij(n)=n / (n+1)Cij(n 1)+1 / (n+1)bi(n)bj(n).

[0119] In a variant embodiment, energy information in various directions (associated with the directions of virtual loudspeakers distributed on the unit sphere) is obtained. To this end, for example, a late reference Figure 3 and Figure 4 The SRP (“Steering Response Power”) method described in the previous section is described in detail. In some variants, other spatial image calculation methods (MUSIC pseudospectrum, histogram of directions of arrival) can be used.

[0120] Several embodiments for encoding an original multi-channel signal are contemplated and described herein.

[0121] In the first embodiment, the various channels b of B k , k=0, ..., K-1 are encoded. In step S2, multi-mono encoding is used. Each channel b k In some variant embodiments, channel b is encoded in separate pairs. kMulti-stereo encoding for encoding is also possible. A conventional example for a 5.1 input signal is to use two separate stereo encoding operations, L / R and Ls / Rs, and C and LFE (low frequency only) mono encoding operations; for the case of hi-fi stereo, multi-stereo encoding can be applied to the hi-fi stereo components (B-format) or to the equivalent multi-channel signal obtained after matrixing the channels in B-format - for example, in order 1, the channels W, X, Y, Z can be converted into four transformed channels and two pairs of channels encoded separately and converted back to B-format when decoding. An example is given in the latest version of the Opus codec ("Channel Mapping 3") and in the 3GPP TR 26.918 specification (clause 6.1.6.3).

[0122] In other variants, joint multi-channel coding may also be used in step S2, such as the MPEG-H 3D Audio codec in the Ambisonics (scene-based) format; in this case, the codec jointly encodes the input channels. In the MPEG-H example, for an Ambisonics signal, this joint coding is decomposed into multiple steps, such as extracting the main mono source and encoding it, extracting the background sound (typically reduced to a 1st order Ambisonics signal), encoding all extracted channels (called "transport channels") and metadata describing the acoustic beamforming vectors to extract the main channel. Joint multi-channel coding enables the relationship between all channels to be exploited, for example to extract the main audio source and the background sound or to perform an overall bit allocation that takes into account all audio content.

[0123] In a preferred embodiment, an exemplary embodiment of step S2 is a multi-mono encoding performed using the 3GPP EVS codec as described above. However, the method according to the invention can therefore be used independently of the core codec (multi-mono, multi-stereo, joint encoding) used to represent the channels to be encoded.

[0124] The signal thus encoded in the form of a bit stream can be decoded in step S3 by a local decoder of the encoder or by a decoder after transmission. The signal is decoded to recover the multi-channel signal (for example by using multiple EVS decoders for multi-mono decoding). The sound channel.

[0125] Steps S2a, S2b, S3a, S3b represent a variant embodiment of the encoding and decoding of the multi-channel signal B. The difference to the encoding of step S2 described above is the use of additional processing operations to reduce the number of channels in step S2a ("downmix") and to increase the number of channels in step S3b ("upmix"). These encoding and decoding steps (S2b and S3a) are similar to steps S2 and S3, except that in steps S2b and S3a the number of corresponding input and output channels is lower.

[0126] One example of a downmix of a 1st order Ambisonics input signal is to keep only W channels; for an Ambisonics input signal of order>1, the first 4 components W, X, Y, Z may be used as the downmix (thus truncating the signal to 1st order). In some variants, a subset of the Ambisonics components (e.g. 8 2nd order channels without the component R) may be used as the downmix, and matrixed cases may also be considered, such as a stereo downmix obtained in the form: L=W Y+0.3*X, R=W+Y+0.3*X (using only the FOA channels).

[0127] One example of upmixing a mono signal consists in applying various spatial room impulse responses (SRIR) or various decorrelation filters (all-pass type) in the time domain or frequency domain. An exemplary embodiment of decorrelation in the frequency domain is given, for example, in the document 3GPP S4-180975, pCR to 26.118 on Dolby VRStream Audio Profile Candidates (clause X.6.2.3.5).

[0128] The signal B' resulting from this "downmix" processing operation is encoded in a step S2b with a core codec (multi-mono, multi-stereo, joint coding), for example using a mono or multi-mono method with a 3GPP EVS codec. The number of channels of the input audio signal from the encoding step S2b and the output audio signal from the decoding step S3a is less than the number of channels of the original multi-channel audio signal. In this case, the spatial image represented by the core codec is substantially degraded even before encoding. In the extreme case, the number of channels is reduced to a single mono channel by encoding only W channels; the input signal is then limited to a single audio channel and the spatial image is therefore lost. The method according to the invention makes it possible to describe this spatial image and reconstruct it as close as possible to the spatial image of the original multi-channel signal.

[0129] At the output of the upmixing step in S3b of this variant embodiment, the decoded multi-channel signal recover.

[0130] In step S4, the decoded multi-channel signal is obtained according to the two variants (S2-S3 or S2a-S2b-S3a-S3b). In the same way as for the original image, this information may be a covariance matrix calculated on the decoded multi-channel signal or energy information associated with the direction from which the sound originates (or, equivalently, associated with an imaginary point on the unit sphere).

[0131] The information representing the original multi-channel signal and the decoded multi-channel signal is used in step S5 to determine a set of corrections to be performed on the decoded multi-channel signal in order to limit spatial degradations.

[0132] The following will refer to Figure 4 and Figure 5 Two embodiments are described to illustrate this step.

[0133] Figure 2 The method described in can be implemented in the time domain, in the frequency full band (with a single frequency band) or by frequency subbands (with multiple frequency bands), and this does not change the operation of the method, each subband is then processed separately. If the method is performed by subbands, the set of corrections is determined according to the subbands, which causes additional costs in terms of calculation and data transmitted to the decoder compared to the case of a single frequency band. The division of subbands can be uniform or non-uniform. For example, the spectrum of a signal sampled at 32kHz can be divided according to various variants:

[0134] 4 frequency bands with widths of 1kHz, 3kHz, 4kHz and 8kHz or even 2kHz, 2kHz, 4kHz and 8kHz

[0135] 24 Bark bands (from 100Hz width at low frequencies to 3.5-4kHz at the last sub-band)

[0136] These 24 Bark bands can be grouped together into blocks of 4 or 6 contiguous bands to form a set of 6 or 4 "aggregate" bands respectively.

[0137] Other divisions are possible (eg ERB band - "Equivalent Rectangular Bandwidth" - or 1 / 3 of an octave), including the case of different sampling frequencies (eg 16 kHz or 48 kHz).

[0138] In some variants, the invention may also be implemented in a transform domain, for example in the domain of the Short Time Discrete Fourier Transform (STFT) or in the domain of the Modified Discrete Cosine Transform (MDCT).

[0139] Several embodiments for implementing the determination of the set of corrections and for applying the set of corrections to the decoded signal are now described.

[0140] Here is a known technique for encoding a sound source in an Ambisonics format. A monophonic sound source can be artificially spatialized by multiplying its signal by the value of a spherical harmonic associated with its direction of origin (assuming that this signal is carried by a plane wave) to obtain the same number of Ambisonics components. This involves calculating the value of the spherical harmonic associated with the direction of its origin (assuming that this signal is carried by a plane wave). Determine the coefficients of each spherical harmonic at the location:

[0141] B=Y(θ,φ).s

[0142] where s is the mono signal to be spatialized and is defined in the direction of the Mth order An example of a coding vector for the associated spherical harmonic coefficients is given below for the 1st order with the SN3D convention and the order of SID or FuMa channels:

[0143]

[0144] In some variants, other normalization conventions (e.g., maxN, N3D) and channel orders (e.g., ACN) may be used, and the various embodiments may then be adapted according to the convention for one or more normalized orders for the high-fidelity stereo components (FOA or HOA). This amounts to modifying the line or multiply the rows by a predefined constant.

[0145] For higher orders, the coefficients of the spherical harmonics are It can be found in the following book: B. Rafaely, Fundamentals of Spherical Array Processing, Springer, 2015. In general, for the order M, there exists K = (M + 1) 2 A high-fidelity stereo signal.

[0146] Likewise, several concepts related to high-fidelity stereophonic rendered by speakers will be mentioned here. High-fidelity stereophonic does not mean listening as such; for immersive listening on speakers or headphones, a "decoding" step of the acoustic scene must be implemented, also called rendering ("renderer"). Considering the distribution of the sound waves on a sphere (typically with unit radius and its direction The case of N (virtual or physical) loudspeakers in azimuth and elevation is known. Decoding, as considered here, consists in applying the matrix D to the high-fidelity stereo signal B to obtain the loudspeaker signals s n These signals can be combined into a matrix S = [s0,…s N-1 ], S = DB, where

[0147]

[0148] The matrix D can be decomposed into a single row matrix d n , that is to say

[0149]

[0150] d n can be viewed as a weighting vector for the nth speaker, used to reassemble the components of the high-fidelity stereo signal and calculate the signal played on the nth speaker: n =d n .B.

[0151] There are various methods for "decoding" in an acoustic scene. The so-called "basic decoding" method (also called "pattern matching") is based on an encoding matrix E associated with all directions of virtual loudspeakers:

[0152]

[0153] According to this method, the matrix D is typically defined as the pseudo-inverse matrix of E: D = pinv(E) = D T (DD T ) -1

[0154] As an alternative, what may be called the "projection" method gives similar results for certain regular distributions of directions and is described by the following equation:

[0155]

[0156] In the latter case, it can be seen that for each direction of index n,

[0157]

[0158] In the context of the present invention, such a matrix will be used as a directional beamforming matrix describing how to obtain signal characteristics of directions in space to perform analysis and / or spatial transformation.

[0159] In the context of the present invention, it is useful to describe the reciprocal transform that passes from the loudspeaker domain to the Ambisonics domain. If no intermediate modifications are applied in the loudspeaker domain, the consecutive application of these two transforms should reproduce the original Ambisonics signal exactly. The reciprocal transform is thus defined as the pseudo-inverse of D:

[0160] pinv(D).S=D T (DD T ) -1 .S

[0161] When K=(M+1) 2 When the size K×K matrix D can be inverted in some cases, and in this case: B = D -1 .S

[0162] In the case of the "pattern matching" method, it seems that pinv(D) = E. In some variants, other methods for decoding using D, and the corresponding inverse transformation E, may be used; the only condition to be satisfied is that the combination of decoding using D and inverse transformation using E should give a perfect reconstruction (when no intermediate processing operations are performed between acoustic decoding and acoustic encoding).

[0163] This variant is given, for example, by the following method:

[0164] - "Pattern matching" decoding, with conditioning items of the form:

[0165] where ε is a low value (e.g. 0.01),

[0166] - "In-phase" or "max-rE" decoding, known from prior art

[0167] Or a variation where the distribution of speaker directions on the sphere is irregular.

[0168] Figure 3 A first embodiment of an encoding device and a decoding device is shown, which is used to implement a method including determining a reference Figure 2 The encoding and decoding methods of the set of correction methods are described.

[0169] In this embodiment, the encoder computes information representing the spatial image of the original multi-channel signal and transmits this information to the decoder to enable it to correct the spatial degradation caused by the encoding. This enables the spatial artifacts in the decoded high-fidelity stereo signal to be reduced during decoding.

[0170] Thus, the encoder receives a multi-channel input signal, such as an Ambisonics representation FOA or HOA or a mixed representation of a subset of Ambisonics components up to a given partial Ambisonics order - the latter case is in fact included in an equivalent manner in the FOA or HOA case, where the missing Ambisonics components are zero and the Ambisonics order is given by the minimum order required to include all defined components. Therefore, without loss of generality, the description of the FOA or HOA case is considered below.

[0171] In the embodiment thus described, the input signal is sampled at 32 kHz. The encoder operates with frames preferably 20 ms long, that is to say L=640 samples per frame at 32 kHz. In some variants, other frame lengths and sampling frequencies are possible (e.g. 10 ms at 48 kHz, L=480 samples per frame).

[0172] In a preferred embodiment, encoding is performed in the time domain (on one or more frequency bands), but in some variants the invention may be implemented in the transform domain, for example after a Short Time Discrete Fourier Transform (STFT) or a Modified Discrete Cosine Transform (MDCT).

[0173] Depending on the encoding embodiment used, as shown in reference Figure 2 As explained, a block 310 for reducing the number of channels (DMX) may be implemented; the input of block 311 is the signal B' at the output of block 310 when downmixing is implemented, otherwise the signal B. In one embodiment, if downmixing is applied, this consists in keeping only the W channels for, for example, a 1st order Ambisonics input signal, and in keeping only the first 4 Ambisonics components W, X, Y, Z for an Ambisonics input signal of order>1 (thus truncating the signal to 1st order). Other types of downmixing (such as the selection of a subset of channels and / or matrixed downmixing described above) may be implemented without modifying the method according to the invention.

[0174] If a downmixing step is performed, block 311 downmixes the audio signal b′ of B′ at the output of block 310. k Encode, or encode the audio signal b of the original multi-channel signal B k If no processing to reduce the number of channels is applied, the signal corresponds to the high-fidelity stereo component of the original multi-channel signal.

[0175] In a preferred embodiment, block 311 uses multi-mono coding (COD) with fixed or variable allocation, where the core codec is the standard 3GPP EVS codec. In this multi-mono approach, each channel b k or b' k The encoding is performed individually by an instance of the codec; however, in some variants, other encoding methods are possible, such as multi-stereo encoding or joint multi-channel encoding. Thus, at the output of the encoding block 311, an encoded audio signal resulting from the original multi-channel signal is given in the form of a bit stream sent to a multiplexer 340.

[0176] Optionally, block 320 performs a division into subbands. In some variants, this division into subbands may reuse equivalent processing operations performed in blocks 310 or 311; the splitting of block 320 is utilized here.

[0177] In a preferred embodiment, the channels of the original multi-channel audio signal are divided into 4 frequency sub-bands with widths of 1 kHz, 3 kHz, 4 kHz, 8 kHz respectively (equivalent to dividing the frequencies into 0-1000 Hz, 1000-4000 Hz, 4000-8000 Hz and 8000-16000 Hz). The division can be implemented by a short-time discrete Fourier transform (STFT), a bandpass filtering in the Fourier domain (by applying a frequency mask) and an inverse transform with overlap addition. In this case, the sub-bands remain sampled at the same original frequency and the processing operations according to the invention are applied in the time domain; in some variants, a filter bank with critical sampling can be used. It should be noted that the operation of sub-band division generally involves a processing delay depending on the type of filter bank implemented; according to the invention, time alignment can be applied before or after encoding-decoding and / or after extraction of the spatial image information so that the spatial image information is well synchronized in time with the corrected signal.

[0178] In some variants, full-band processing may be performed, or the division into sub-bands may be different, as explained above.

[0179] In other variants, the signal resulting from the transformation of the original multi-channel audio signal is used directly and the invention is applied to the transform domain divided into subbands in the transform domain.

[0180] In the rest of the description, the various steps of encoding and decoding are described as if they involve processing operations in the (real or complex) time domain or frequency domain with a single frequency band to simplify the description.

[0181] A high-pass filtering (with a cut-off frequency typically of 20 Hz or 50 Hz) may also be optionally implemented in each subband, for example in the form of a 2nd order elliptic IIR filter, with a cut-off frequency preferably set to 20 Hz or 50 Hz (50 Hz in some variants). This pre-processing avoids possible bias in the subsequent covariance estimation during encoding; without this pre-processing, the correction implemented in block 390, described later, would tend to amplify low frequencies during full-band processing.

[0182] Block 321 determines (Inf.B) information representing the spatial image of the original multi-channel signal.

[0183] In one embodiment, the information is energy information associated with the direction from which the sound originates (associated with the directions of virtual speakers distributed on the unit sphere).

[0184] To this end, a virtual 3D sphere with unit radius is defined, which is discretized by N points (“point” virtual speakers), whose positions in spherical coordinates are given by the direction of the nth speaker Definition. The loudspeakers are typically placed on a sphere in a (quasi-)uniform manner. The number of virtual loudspeakers N is determined as a discretization with at least N=K points, where M is the high-fidelity stereo order of the signal and K=(M+1) 2 , that is to say N ≥ K. The discretization can be performed, for example, using the “Lebedev” quadrature method, according to the following references: VI Lebedev and DN Laikov, “A quadrature formula for the sphere of the 131st algebraic order of accuracy”, Doklady Mathematics, Vol. 59, No. 3, 1999, pp. 477-481 or Pierre Lecomte, Philippe-Aubert Gauthier, Christophe Langrenne, Alexandre Garcia and Alain Berry, “On the use of a Lebedev grid for Ambisonics”, AES Convention 139, New York, 2015.

[0185] In other variations, other discretizations may be used, such as a Fliege discretization with at least N = K points (N ≥ K), as described in J. Fliege and U. Maier, "A two-stage approach for computing cubature formulae for the sphere", Technical Report, Dortmund University, 1999, or a discretization using points from a "spherical t design" as described in RH Hardin and NJASloane, "McLaren's Improved Snub Cube and Other New Spherical Designs in Three Dimensions", Discrete and Computational Geometry, 15 (1996), pp. 429-441.

[0186] From this discretization, the spatial image of the multichannel signal can be determined. One possible method is for example the SRP ("Steering Response Power") method. In fact, this method consists in calculating the short-term energy coming from various directions defined in terms of azimuth and elevation. To this end, as explained above, a weighting matrix of the high-fidelity stereo components is calculated, similar to the rendering on N loudspeakers, and this matrix is ​​then applied to the multichannel signal to sum the contributions of the components and generate a set of N beams (or "beamformers").

[0187] From the direction of the nth speaker The signal of the sound beam is given by: n =d n .B

[0188] Among them, d n is a vector of weights (rows) giving the acoustic beamforming coefficients for a given direction, and B is a matrix of size K x L representing an Ambisonics signal (B format) with K components in a time interval of length L.

[0189] A set of signals from N beams yields the equation: S = DB

[0190] in

[0191]

[0192] And S is a matrix of size N×L representing the signals of N virtual loudspeakers in a time interval of length L.

[0193] In each direction within a time segment of length L The short-term energy is:

[0194]

[0195] Where C = BB T (real case) or Re(BB H )(complex case) is the covariance matrix of B.

[0196] Each item For all directions corresponding to the discretization of the 3D sphere of the virtual loudspeaker Calculate in this way.

[0197] The spatial image ∑ is then given by:

[0198] ∑=[σ0 2 , ..., σ N-1 2 ]

[0199] Variants for computing the spatial image Σ other than the SRP method may be used.

[0200] Value d n may vary depending on the type of acoustic beamforming used (delay-sum, MVDR, LCMV, etc.). The invention also applies to those variants where the matrix D and the spatial image are calculated

[0201] ∑=[σ0 2 , ..., σ N-1 2 ]

[0202] The MUSIC (Multiple Signal Classification) method also provides another method to calculate spatial images in a subspace manner.

[0203] The invention also applies to computing this variant of the spatial image

[0204] ∑=[σ0 2 , ..., σ N-1 2 ]

[0205] This variant corresponds to the sum calculated by diagonalizing the covariance matrix for the direction Evaluated MUSIC pseudospectra.

[0206] The spatial image can be calculated from the histogram of intensity vectors (order 1), for example in S. Tervo, Direction estimation based on sound intensity vectors, Proc. EUSIPCO, 2009, or its generalization to pseudo-intensity vectors. In this case, the histogram (values ​​are in the predetermined direction) The number of occurrences of the arrival direction value in the θ is interpreted as a set of energies in a predetermined direction.

[0207] The block 330 then quantizes the spatial image thus determined, for example with scalar quantization on 16 bits per coefficient (by directly using a truncated floating point representation on 16 bits). In some variants, other scalar or vector quantization methods are possible.

[0208] In another embodiment, the information representing the spatial image of the original multi-channel signal is the covariance matrix (of the subbands) of the input channel B. The matrix is ​​calculated as follows:

[0209] C=BB T to be within the normalization factor (in the real case).

[0210] If the invention is implemented in the complex-valued transform domain, the covariance is calculated as follows:

[0211] C=Re(BB H )

[0212] to be within the normalization factor.

[0213] In some variants, an operation that temporally smoothes the covariance matrix may be used.In case of multi-channel signals in the time domain, the covariance may be estimated recursively (sample by sample).

[0214] The covariance matrix C (of size K×K) is symmetric by definition, and only one of the lower or upper triangle is passed to the quantization block 330, which encodes (Q) K(K+1) / 2 coefficients, K being the number of high-fidelity stereo components.

[0215] This block 330 quantizes the coefficients, for example with scalar quantization on 16 bits per coefficient (by directly using a truncated floating point representation on 16 bits). In some variants, other methods for scalar or vector quantization of the covariance matrix may be implemented. For example, the maximum value (maximum variance) of the covariance matrix may be calculated and then the values ​​of the upper (or lower) triangle of the covariance matrix normalized by the maximum value may be encoded on a smaller number of bits (e.g., 8 bits) using scalar quantization and a logarithmic step size.

[0216] In some variations, the covariance matrix C may be normalized and then quantized in the form C+εI.

[0217] The quantized value is sent to multiplexer 340 .

[0218] In this embodiment, in the multiplexer block 350, the decoder receives a bitstream comprising an encoded audio signal generated from an original multi-channel signal and information representing a spatial image of the original multi-channel signal.

[0219] Block 360 decodes the covariance matrix or other information representing the spatial image of the original signal (Q -1 Block 370 decodes (DEC) the audio signal represented by the bitstream.

[0220] In one embodiment of encoding and decoding without implementing the downmixing and upmixing steps, the decoded multi-channel signal is obtained at the output of the decoding block 370

[0221] In an embodiment in which encoding is performed using a downmixing step, the decoding implemented in the block 370 makes it possible to obtain a decoded audio signal which is sent to the input of the upmixing block 371

[0222] Therefore, block 371 implements an optional step of increasing the number of channels (upmixing). In one embodiment of this step, for a channel of a mono signal, It is about warping the signal using various spatial room impulse responses (SRIR) These SRIRs are defined in terms of the original high-fidelity stereo order of B. Other decorrelation methods are possible, such as applying an all-pass decorrelation filter to the signal various channels.

[0223] Block 372 implements an optional step of subband division (SB) to obtain subbands in the time domain or in the transform domain. In block 391, an inversion step combines the subbands to recover the multi-channel signal at the output.

[0224] Block 375 determines (Inf ) representing the spatial image of the decoded multi-channel signal in a similar manner as described for block 321 (for the original multi-channel signal). ) information, this time applied to the decoded multi-channel signal obtained at the output of block 371 or block 370 according to the decoding embodiment

[0225] In the same manner as described for block 321, in one embodiment, the information is energy information associated with the direction from which the sound originates (associated with the directions of virtual loudspeakers distributed on the unit sphere). As explained above, the SRP method (among others) can be used to determine the spatial image of the decoded multi-channel signal.

[0226] In another embodiment, the information is a covariance matrix of the channels of the decoded multi-channel signal.

[0227] The covariance matrix is ​​then obtained as follows:

[0228] (real case) or

[0229] (Complex case) to be within the normalization factor.

[0230] In some variants, an operation that temporally smoothes the covariance matrix may be used.In case of multi-channel signals in the time domain, the covariance may be estimated recursively (sample by sample).

[0231] According to the information (Inf.B) representing the spatial image of the original multi-channel signal and the information (Inf. ), for example, the covariance matrix C and Block 380 implements the process for determining (Det.Corr) as referenced Figure 2 Describes a set of correction methods.

[0232] refer to Figure 4 and Figure 5 Two specific embodiments of this determination are described.

[0233] exist Figure 4 In an embodiment of the present invention, a method using (explicit or implicit) presentation on a virtual speaker is used, and in Figure 5 In the embodiment of the present invention, a method based on Cholesky factorization is used.

[0234] Figure 3 Block 390 performs a correction (CORR) on the decoded multi-channel signal using the set of corrections determined by block 380 to obtain a corrected decoded multi-channel signal.

[0235] therefore, Figure 4 One embodiment of the step of determining a set of corrections is shown. This embodiment is performed using rendering on a virtual speaker.

[0236] In this embodiment, it is initially considered that the information representing the spatial image of the original multi-channel signal and the information representing the spatial image of the decoded multi-channel signal are the corresponding covariance matrices C and

[0237] In this case, blocks 420 and 421 determine the spatial image of the original multi-channel signal and the decoded multi-channel signal, respectively.

[0238] To this end, as described above, a virtual 3D sphere with unit radius is discretized by N points (“point” virtual loudspeakers), whose directions in spherical coordinates are given by the direction of the nth loudspeaker definition.

[0239] Several discretization methods have been defined above.

[0240] From this discretization, the spatial image of the multi-channel signal can be determined.As mentioned above, one possible method is the SRP method (among others) which consists in calculating the short-term energy from each direction defined in terms of azimuth and elevation.

[0241] This method or other types of methods as listed above may be used to determine the original multi-channel signal at 420 (IMG B) and 421 (IMG ) and the spatial image ∑ of the decoded multi-channel signal at (IS B and IS ).

[0242] If the signal (Inf B) representing the spatial image of the original signal received and decoded by the decoder at 360 is the spatial image itself, that is to say the energy information (or positive values) associated with the direction from which the sound originates (associated with the directions of virtual loudspeakers distributed on the unit sphere), it is no longer necessary to calculate this information at 420. This spatial image is then used directly by means of block 430 as described below.

[0243] Likewise, if the signal (Inf ) is the spatial image of the decoded multi-channel signal itself, then there is no need to calculate this information at 421. The spatial image is then used directly by block 430 as described below.

[0244] According to the spatial image ∑ and Block 430 is directed to Given each point, calculate (ratio) the energy σ of the original signal n 2 =∑ n The energy of the decoded signal The energy ratio between them. Therefore, a set of gains g is obtained using the following equation n :

[0245]

[0246] According to the direction and frequency band, the energy ratio can be large. Block 440 enables optional limitation (limit g n )Gain g n The maximum value that can be taken. We will recall here that it is denoted by σ n 2 and Positive values ​​of may more generally correspond to values ​​generated from the MUSIC pseudospectrum or from the discretization direction The values ​​produced by the arrival direction histogram in .

[0247] In one possible embodiment, a threshold is applied to g n Any value greater than the threshold is forced to be equal to the value of the threshold. The threshold may be set to 6 dB, for example, so that gain values ​​outside the interval ±6 dB are saturated at ±6 dB.

[0248] Therefore, the group gain g n The set of corrections to be performed on the decoded multi-channel signal is constructed.

[0249] exist Figure 3 The set of gains is received at the input of the correction block 390.

[0250] A correction matrix can be defined that can be directly applied to the decoded multi-channel signal, for example as G = E.diag ([g0 ... g N-1 ]).D, where D and E are the acoustic decoding and encoding matrices defined above. The matrix G is applied to the decoded multi-channel signal To obtain the corrected output high-fidelity stereo signal ( corr).

[0251] A breakdown of the steps implemented for the correction is now described. For each virtual loudspeaker, block 390 applies the corresponding previously determined gain g n Applying this gain results in the same amount of energy being delivered to the loudspeaker as the original signal.

[0252] The presentation of the decoded signal at each loudspeaker is thus corrected.

[0253] Then an acoustic coding step is performed, such as high-fidelity stereo coding using the matrix E, to obtain components of the multi-channel signal, such as high-fidelity stereo components. These high-fidelity stereo components are finally summed to obtain a corrected output multi-channel signal ( Corr). It is thus possible to accurately calculate the channels associated with the virtual loudspeakers, apply gains thereto, and then reconstruct the processed channels, or in an equivalent way to apply the matrix G to the signal to be corrected.

[0254] In some variants, the covariance matrix of the encoded and then decoded multi-channel signal may be used. The covariance matrix of the corrected signal in block 390 is calculated as:

[0255]

[0256] The first coefficient R of the matrix R corresponding only to the omnidirectional component (W channel) 00 The value of is retained to be applied as a normalization factor to R and to avoid an increase in the overall gain due to the correction matrix G:

[0257]

[0258] G norm =g norm .G

[0259] in

[0260]

[0261] in, Corresponding to the first coefficient of the covariance matrix of the decoded multi-channel signal.

[0262] In some variations, a normalization factor g may be determined norm There is no need to compute the entire matrix R, since computing only a subset of the matrix elements is sufficient to determine R 00 (and therefore g norm ).

[0263] So the matrix G or G norm corresponds to the set of corrections to be performed on the decoded multi-channel signal.

[0264] Now Figure 5 shows the method used to determine the Figure 3 Another embodiment of a set of correction methods implemented in block 380 of .

[0265] In this embodiment, it is considered that the information representing the spatial image of the original multi-channel signal and the information representing the spatial image of the decoded multi-channel signal are the corresponding covariance matrices C and

[0266] In this embodiment, no attempt is made to correct the spatial image of the multi-channel signal by the rendering performed on the virtual loudspeakers. In particular, for an Ambisonics signal, an attempt is made to compute the correction of the spatial image directly in the Ambisonics domain.

[0267] To this end, a transformation matrix T to be applied to the decoded signal is determined so that when the transformation matrix T is applied to the decoded signal The modified spatial image is then identical to the spatial image of the original signal B.

[0268] Therefore, we seek a matrix T that satisfies the following equation:

[0269] Where C = BB T is the covariance matrix of B and In the current frame The covariance matrix of .

[0270] In this embodiment, a decomposition known as a Culaski decomposition is used to solve the equation.

[0271] Given a matrix A of size n×n, the Cholakia decomposition consists in determining the (upper or lower) triangular matrix L such that A=LL T (real case) and A=LL H (Complex case). For possible decompositions, the matrix A can be a positive definite symmetric matrix (real case) or a positive definite Hermitian matrix (complex case); in the real case, the diagonal coefficients of L are strictly positive.

[0272] In the case of real numbers, if it is symmetric (M T =M) and positive definite (for x∈Rn Any value of \{0}, x T Mx>0), then the matrix M of size n×n is considered to be positive definite and symmetric.

[0273] For a symmetric matrix M, it can be shown that if all eigenvalues ​​are strictly positive (λ i >0), then the matrix is ​​positive definite. If the eigenvalue is positive (λ i ≥0), then the matrix is ​​considered to be positive semidefinite.

[0274] If the Hermitian H =M) and positive definite (for z∈C n Any value of \{0}, z H Mz is a real number > 0), then a matrix M of size n×n is called positive definite symmetric Hermitian.

[0275] For example, the Cholakia decomposition is used to obtain the solution of a linear system of equations of the type Ax = b. For example, in the case of complex numbers, the Cholakia decomposition can be used to transform A into LL H To solve Ly = b and then solve for L H x=y.

[0276] In an equivalent way, the Cholakia decomposition can be written as A = U T U (real case) and A = U H U (complex case), where U is an upper triangular matrix.

[0277] In the embodiments described here, only Cholesky decompositions with triangular matrices L are treated without loss of generality.

[0278] Therefore, the Cholakia decomposition allows the matrix C = LL under the condition that the matrix C is positive definite and symmetric. T decomposed into two triangular matrices. This gives the following equation:

[0279]

[0280] Use identification to obtain:

[0281]

[0282] That is:

[0283]

[0284] Because the covariance matrix C and is usually a positive semidefinite matrix, so the Culaski decomposition cannot be used as such.

[0285] It should be noted here that when the matrix L and When is lower (respectively upper) triangular, the transformation matrix T is also lower (respectively upper) triangular.

[0286] Therefore, block 510 forces the covariance matrix C to be positive definite. To do this, a value ε is added (Fact.C represents the factorization of C) to the diagonal coefficients of the matrix to ensure that the matrix is ​​actually positive definite: C = C + εI, where ε is set to 10, for example. -9 and I is the identity matrix.

[0287] Similarly, block 520 is accomplished by Modify the covariance matrix in the form To force the matrix to be positive definite, ε is set to 10, for example -9 and I is the identity matrix.

[0288] Once the two covariance matrices C and Being adjusted to be positive definite, block 530 calculates the associated Culaski decomposition and obtains the optimal transformation matrix T of the following form (Det.T):

[0289]

[0290] In some variants, the alternative solution may be performed via eigenvalue decomposition.

[0291] The decomposition of eigenvalues ​​("eigendecomposition") consists in factorizing a real or complex matrix A of size n×n in the following form:

[0292] A=QΛQ -1

[0293] Among them, Λ is a i is a diagonal matrix of and Q is the matrix of eigenvectors.

[0294] If the matrix is ​​real, then:

[0295] A=QΛQ T

[0296] In the case of complex numbers, the decomposition is written as: A = QΛQ H

[0297] In the present case, what is sought is a matrix T such that:

[0298] Where C = QΛQ t and

[0299]

[0300] That is:

[0301]

[0302] Use identification to obtain:

[0303]

[0304] That is:

[0305]

[0306] The stability of the solution from one frame to another is typically not as good as that of the Culaski decomposition method. This instability is exacerbated by the more important computational approximations during the decomposition of the eigenvalues, which may be larger.

[0307] In some variations, the diagonal matrix

[0308]

[0309] in

[0310] Λ=(λ0,...,λ K-1 )

[0311] You can press The format is calculated element by element

[0312] where sgn(.) is the sign function (+1 if positive, -1 otherwise) and ε is the regularization term (e.g. ε = 10 -9 ) to avoid division by zero.

[0313] In this embodiment, the relative difference in energy between the decoded ambidextrous signal and the corrected ambidextrous signal may be large, especially with respect to high frequencies, which may be strongly corrupted by encoders such as multi-mono EVS encoding. In order to avoid extreme amplification of certain frequency regions, a regularization term may be added. Block 640 is optionally responsible for normalizing the correction (Norm.T).

[0314] In a preferred embodiment, the normalization factor is therefore calculated so as not to amplify frequency regions.

[0315] According to the covariance matrix of the encoded and then decoded multi-channel signal and the transformation matrix T, the covariance matrix of the correction signal can be calculated as:

[0316]

[0317] The first coefficient R of the matrix R corresponding only to the omnidirectional component (W channel) 00 The value of is retained to be applied as a normalization factor to T and to avoid an increase in the overall gain due to the correction matrix T:

[0318]

[0319] T norm =g norm .T

[0320] in

[0321]

[0322] in, Corresponding to the first coefficient of the covariance matrix of the decoded multi-channel signal.

[0323] In some variations, a normalization factor g may be determined norm There is no need to compute the entire matrix R, since computing only a subset of the matrix elements is sufficient to determine R 00 (and therefore g norm ).

[0324] So we get the matrix T or T norm corresponds to the set of corrections to be performed on the decoded multi-channel signal.

[0325] Through this embodiment, Figure 3 Block 390 converts the transformation matrix T or T into the Ambisonics domain. norm The step of correcting the decoded multi-channel signal is performed directly on the decoded multi-channel signal to obtain a corrected output high-fidelity stereo signal ( corr).

[0326] A second embodiment of an encoder / decoder according to the invention will now be described, in which the method for correcting groups is implemented in the encoder. Figure 6 Thus, the figure shows a second embodiment of a coding device and a decoding device for implementing a method comprising determining a reference Figure 2 The encoding and decoding methods of the set of correction methods are described.

[0327] In this embodiment, the method for determining a set of corrections (e.g., gains associated with directions) is performed at the encoder, which then transmits the set of corrections to the decoder. The decoder decodes the set of corrections to apply them to the decoded multi-channel signal. Thus, this embodiment involves implementing local decoding at the encoder, and the local decoding is represented by blocks 612 to 613.

[0328] Boxes 610, 611, 620 and 621 are respectively equivalent to reference Figure 3 Blocks 310, 311, 320 and 321 are described.

[0329] Information representative of the spatial image of the original multi-channel signal (Inf. B) is thus obtained at the output of block 621 .

[0330] Block 612 performs local decoding (DEC_loc) consistent with the encoding performed by block 611 .

[0331] This local decoding may consist of a complete decoding of the bitstream from block 611 , or, preferably, may be integrated into block 611 .

[0332] In one embodiment of encoding and decoding without implementing the downmixing and upmixing steps, the decoded multi-channel signal is obtained at the output of the local decoding block 612

[0333] In the embodiment where the downmixing step at 610 is used for encoding, the local decoding implemented in block 612 enables obtaining a decoded audio signal sent to the input of the upmixing block 613

[0334] Therefore, block 613 implements an optional step of increasing the number of channels (upmixing). In one embodiment of this step, for a channel of a mono signal, It is about warping the signal using various spatial room impulse responses (SRIR) These SRIRs are defined in terms of the original high-fidelity stereo order of B. Other decorrelation methods are possible, such as applying an all-pass decorrelation filter to the signal various channels.

[0335] Block 614 implements an optional step of subband partitioning (SB) to obtain subbands in the time domain or transform domain.

[0336] Block 615 determines (Inf ) representing the spatial image of the decoded multi-channel signal in a similar manner as described for blocks 621 and 321 (for the original multi-channel signal). ) information, this time applied to the decoded multi-channel signal obtained at the output of block 612 or block 613 according to the embodiment of local decoding This block 615 is equivalent to Figure 3 375 in box.

[0337] In the same manner as for blocks 621 and 321, in one embodiment, the information is energy information associated with the direction from which the sound originates (associated with the directions of virtual loudspeakers distributed on the unit sphere). As explained above, the SRP method or the like (like the variants described above) can be used to determine the spatial image of the decoded multi-channel signal.

[0338] In another embodiment, the information is a covariance matrix of the channels of the decoded multi-channel signal.

[0339] The covariance matrix is ​​then obtained as follows:

[0340] to within the normalization factor (in the case of real numbers)

[0341] or

[0342]

[0343] to within the normalization factor (in the case of complex numbers)

[0344] According to the information (Inf.B) representing the spatial image of the original multi-channel signal and the information (Inf. ), for example, the covariance matrix C and Block 680 implements the process for determining (Det.Corr) as referenced Figure 2 Describes a set of correction methods.

[0345] Two specific embodiments of this determination are possible and have been referred to. Figure 4 and Figure 5 Give a description.

[0346] exist Figure 4 In an embodiment, a method using presentation on a loudspeaker is used, and in Figure 5 In an embodiment of the invention, a method based on Culasky decomposition or by decomposition of eigenvalues ​​implemented directly in the high-fidelity stereo domain is used.

[0347] Therefore, if at 630, Figure 4 In an embodiment, the set of corrections determined is a set of directions defined by a set of virtual loudspeakers A set of gains g n The set of gains can be determined in the form of a correction matrix G, as shown in reference Figure 4 Described in .

[0348] The set of gains (Corr.) is then encoded at 640. Encoding the set of gains may involve encoding the correction matrix G or G norm to encode.

[0349] It should be noted that the matrix G of size K×K is symmetric, so according to the present invention, only G or G norm The lower or upper triangle of G (i.e., K×(K+1) / 2 values) is encoded. Typically, the values ​​on the diagonal are positive. In one embodiment, scalar quantization (with or without a sign bit) is used to encode the matrix G or G, depending on whether the values ​​are off-diagonal. norm Encoding. norm In the variant ofnorm The encoding and transmission of the first value of the diagonal (corresponding to the omnidirectional component) of , since this value is always 1; for example, in the case of 1st-order ambisonics with K=4 channels, this is equivalent to transmitting only 9 values ​​instead of K×(K+1) / 2=10 values. In some variants, other scalar or vector quantization methods (with or without prediction) may be used.

[0350] If applied at 630 Figure 5 In an embodiment, the determined set of corrections is the transformation matrix T or T norm , the transformation matrix is ​​then encoded at 640 .

[0351] It should be noted that the matrix T of size K×K is triangular in the variant using Cholakia decomposition and symmetric in the variant using eigenvalue decomposition; therefore, according to the present invention, only T or T norm The lower or upper triangle (i.e., K×(K+1) / 2 values) of the RGB-D matrix is ​​encoded.

[0352] Typically, the values ​​on the diagonal are positive. In one embodiment, the matrix T or T is quantized using scalar quantization (with or without a sign bit), depending on whether the value is off-diagonal or not. norm In some variants, other scalar or vector quantization methods (with or without prediction) can be used. norm In the variant of norm Coding and transmission of the first value of the diagonal (corresponding to the omnidirectional component), since this value is always 1; for example, in the case of 1st order ambisonics with K=4 channels, this amounts to transmitting only 9 values ​​instead of K×(K+1) / 2=10 values.

[0353] Accordingly, block 640 encodes the determined set of corrections and sends the encoded set of corrections to multiplexer 650 .

[0354] In a multiplexer block 660, the decoder receives a bitstream comprising an encoded audio signal generated from the original multi-channel signal and the encoded set of corrections to be applied to the decoded multi-channel signal.

[0355] Block 670 decodes the encoded set of corrections (Q -1 Block 680 decodes (DEC) the encoded audio signal in the stream.

[0356] In one embodiment of encoding and decoding without implementing the downmixing and upmixing steps, the decoded multi-channel signal is obtained at the output of the decoding block 680

[0357] In an embodiment in which encoding is performed using a downmixing step, the decoding implemented in the block 680 makes it possible to obtain a decoded audio signal which is sent to the input of the upmixing block 681

[0358] Therefore, block 681 implements an optional step of increasing the number of channels (upmixing). In one embodiment of this step, for a mono signal channels, which consists in warping the signal using various spatial room impulse responses (SRIRs) These SRIRs are defined in terms of the original high-fidelity stereo order of B. Other decorrelation methods are possible, such as applying an all-pass decorrelation filter to the signal various channels.

[0359] Block 682 implements an optional step of subband splitting (SB) to obtain subbands in the time or transform domain, and block 691 groups the subbands together to recover the output multi-channel signal.

[0360] Block 690 applies a correction (CORR) to the decoded multi-channel signal using the set of corrections decoded at block 670 to obtain a corrected decoded multi-channel signal ( Corr).

[0361] In this group of calibration is the reference Figure 4 In one embodiment of a set of gains described, the set of gains is received at an input of correction block 690 .

[0362] If the set of gains takes the form of a correction matrix that can be directly applied to the decoded multi-channel signal, the correction matrix is ​​defined, for example, in the following form:

[0363] G=E.diag([g0...g N-1 ]).D or G norm =g norm .G, then the matrix G or G norm Applicable to decoded multi-channel signals To obtain the corrected output high-fidelity stereo signal ( Corr).

[0364] If block 690 receives a set of gains g n , then block 690 applies the corresponding gain g to each virtual speaker n Applying this gain results in the same amount of energy being delivered to the loudspeaker as the original signal.

[0365] The presentation of the decoded signal at each loudspeaker is thus corrected.

[0366] An acoustic encoding step, such as high-fidelity stereo encoding, is then performed to obtain components of the multi-channel signal, such as high-fidelity stereo components. These high-fidelity stereo components are then summed to obtain a corrected multi-channel output signal ( Corr).

[0367] In this calibration the reference Figure 5 In one embodiment of the transformation matrix described, the transformation matrix T decoded at 670 is received at the input of the correction block 690 .

[0368] By this embodiment, block 690 converts the transformation matrix T or T into the Ambisonics domain. norm directly applied to the decoded multi-channel signal to perform the step of correcting the decoded multi-channel signal to obtain a corrected output high-fidelity stereo signal ( corr).

[0369] Even though the present invention is applied in the case of Ambisonics, in some variants, other formats (multi-channel, object, etc.) may be converted to Ambisonics to implement the methods according to the various embodiments described. An exemplary embodiment of converting from a multi-channel or object format to an Ambisonics format is described in 3GPP TS 26.259 specification. Figure 2 (v15.0.0).

[0370] Figure 7 Shown are a coding device DCOD and a decoding device DDEC which, in the sense of the invention, are paired with one another (in the sense of "reversible") and are connected to one another via a communication network RES.

[0371] The coding device DCOD comprises a processing circuit which typically comprises:

[0372] a memory MEM1 for storing instruction data of a computer program in the sense of the invention (these instructions possibly being distributed between a coder DCOD and a decoder DDEC);

[0373] an interface INT1 for receiving an original multi-channel signal B, for example a high-fidelity stereo signal distributed over various channels (for example four 1st-order channels W, Y, Z, X), in order to compress and encode it in the sense of the invention;

[0374] a processor PROC1 for receiving the signal and processing it by executing computer program instructions stored in the memory MEM1 to encode the signal; and

[0375] The communication interface COM 1 is used to transmit the encoded signal via a network.

[0376] The decoding device DDEC comprises its own processing circuitry, which typically comprises:

[0377] a memory MEM2 for storing instruction data of a computer program in the sense of the invention (these instructions possibly being distributed between the encoder DCOD and the decoder DDEC as described above);

[0378] an interface COM2 for receiving coded signals from the network RES in order to compress and decode them in the sense of the invention;

[0379] a processor PROC2 for processing the signals by executing computer program instructions stored in the memory MEM2 to decode the signals; and

[0380] Output interface INT2, for delivering the corrected decoded signal ( Corr), for example in the form of high-fidelity stereo channels W...X, to present these signals.

[0381] Of course, the Figure 7 An example of a structural embodiment of a codec (encoder or decoder) in the sense of the present invention is shown. As mentioned above, Figures 3 to 6 More functional embodiments of these codecs are described in detail.

Claims

1. A method for determining a set of corrections to be performed on a multi-channel sound signal, wherein: The set of corrections is determined based on information representing a spatial image of an original multi-channel signal and information representing a spatial image of a decoded multi-channel signal, the decoded multi-channel signal representing that the original multi-channel signal has been encoded and then decoded.

2. The method of claim 1, wherein: The set of corrections is determined by frequency sub-band.

3. A method for decoding a multi-channel sound signal, the method comprising the following steps: receiving a bitstream comprising an encoded audio signal from an original multi-channel signal and information representing a spatial image of the original multi-channel signal; Decoding the received encoded audio signal and obtaining a decoded multi-channel signal; Decoding information representing a spatial image of the original multi-channel signal; determining information representing a spatial image of the decoded multi-channel signal; determining a set of corrections to be performed on the decoded multi-channel signal using the method of any one of claims 1 and 2; The decoded multi-channel signal is corrected using the determined set of corrections.

4. The method of claim 3, wherein: The information representing the spatial image is a covariance matrix, and determining the set of corrections further comprises the following steps: obtaining a weighting matrix including weighting vectors associated with a set of virtual loudspeakers; Determine a spatial image of the original multi-channel signal according to the obtained weighting matrix and a covariance matrix of the original multi-channel signal; determining a spatial image of the decoded multi-channel signal according to the obtained weighting matrix and the determined covariance matrix of the decoded multi-channel signal; A ratio between the spatial image of the original multi-channel signal and the spatial image of the decoded multi-channel signal in directions of speakers in the set of virtual speakers is calculated to obtain a set of gains.

5. The method of claim 3, wherein: The received information representing the spatial image of the original multi-channel signal is the spatial image of the original multi-channel signal, and determining the set of corrections further comprises the following steps: obtaining a weighting matrix including weighting vectors associated with a set of virtual loudspeakers; determining a spatial image of the decoded multi-channel signal according to the obtained weighting matrix and the information representing the determined spatial image of the decoded multi-channel signal; A ratio between the spatial image of the original multi-channel signal and the spatial image of the decoded multi-channel signal in directions of speakers in the set of virtual speakers is calculated to obtain a set of gains.

6. The method of claim 3, wherein: The information representative of the spatial image is a covariance matrix, and determining the set of corrections comprises a step of determining a transformation matrix by matrix decomposition of the two covariance matrices, the transformation matrix constituting the set of corrections.

7. The method according to any one of claims 4 to 6, wherein: The decoded multi-channel signal is corrected by the determined set of corrections by applying the set of corrections to the decoded multi-channel signal.

8. The method according to any one of claims 4 and 5, wherein: The decoded multi-channel signal is corrected by the determined set of corrections in the following steps: acoustically decoding the decoded multi-channel signal on a defined set of virtual loudspeakers; applying the obtained set of gains to the signal produced by the acoustic decoding; acoustically encoding the correction signal resulting from the acoustic decoding to obtain components of the multi-channel signal; The components of the multi-channel signal thus obtained are summed to obtain a corrected multi-channel signal.

9. A method for encoding a multi-channel sound signal, the method comprising the following steps: encoding an audio signal from an original multi-channel signal; determining information representing a spatial image of the original multi-channel signal; Locally decoding the encoded audio signal and obtaining a decoded multi-channel signal; determining information representing a spatial image of the decoded multi-channel signal; determining a set of corrections to be performed on the decoded multi-channel signal using the method of any one of claims 1 and 2; The determined set of corrections is encoded.

10. The method of claim 9, wherein: The information representing the spatial image is a covariance matrix, and determining the set of corrections further comprises the following steps: obtaining a weighting matrix including weighting vectors associated with a set of virtual loudspeakers; Determine a spatial image of the original multi-channel signal according to the obtained weighting matrix and a covariance matrix of the original multi-channel signal; determining a spatial image of the decoded multi-channel signal according to the obtained weighting matrix and the determined covariance matrix of the decoded multi-channel signal; A ratio between the spatial image of the original multi-channel signal and the spatial image of the decoded multi-channel signal in directions of speakers in the set of virtual speakers is calculated to obtain a set of gains.

11. The method of claim 9, wherein: The information representative of the spatial image is a covariance matrix, and determining the set of corrections comprises a step of determining a transformation matrix by matrix decomposition of the two covariance matrices, the transformation matrix constituting the set of corrections.

12. A method for decoding a multi-channel sound signal, the method comprising the following steps: Receiving a bitstream comprising an encoded audio signal from an original multi-channel signal and an encoded set of corrections to be performed on a decoded multi-channel signal, the set of corrections having been encoded using a method as claimed in any one of claims 9 to 11; Decoding the received encoded audio signal and obtaining a decoded multi-channel signal; decoding the encoded set of corrections; The decoded multi-channel signal is corrected by applying the decoded set of corrections to the decoded multi-channel signal.

13. A method for decoding a multi-channel sound signal, the method comprising the following steps: receiving a bitstream comprising an encoded audio signal from an original multi-channel signal and an encoded set of corrections to be performed on a decoded multi-channel signal, the set of corrections having been encoded using the method as claimed in claim 10; Decoding the received encoded audio signal and obtaining a decoded multi-channel signal; decoding the encoded set of corrections; The decoded set of corrections is used to correct the decoded multi-channel signal in the following steps: acoustically decoding the decoded multi-channel signal on a set of virtual loudspeakers; applying the obtained set of gains to the signal produced by acoustic decoding; acoustically encoding the correction signal generated by the acoustic decoding to obtain components of the multi-channel signal; The components of the multi-channel signal thus obtained are summed to obtain a corrected multi-channel signal.

14. A decoding device comprising processing circuitry for implementing the method of any one of claims 3 to 8 or 12 to 13.

15. An encoding device comprising processing circuitry for implementing the method of any one of claims 9 to 11.

16. A storage medium readable by a processor and storing a computer program comprising instructions for executing the method of any one of claims 3 to 8 or 12 to 13 or the method of any one of claims 9 to 11.

Images