Adaptive audio object encoding / decoding method and apparatus in immersive audio systems

Abstract

This invention discloses an adaptive audio object encoding and decoding method and apparatus for use in immersive audio systems. The encoding method first performs activity detection on the audio signal spectrum, calculates the merging factor, adaptively divides the signal into sub-bands based on the merging factor, extracts side information, and then compresses and quantizes the side information before combining it with the downmixed signal stream to form an encoded bitstream. The decoding method first decomposes the input bitstream to obtain the side information bitstream and the downmixed signal bitstream, then synthesizes them to obtain the audio object signal. A post-processor converts this signal into a time-domain signal, renders it, and outputs personalized multi-audio objects. The apparatus includes a conversion module, a detection module, a grouping module, an extraction module, an encoding module, a decoding module, a synthesis module, and a rendering module. This invention reduces aliasing distortion of audio objects and improves the quality of decoded audio objects by adaptively grouping sub-bands based on the activity of the audio signal, thus meeting the user's immersive listening needs.

Description

Technical Field

[0001] This invention belongs to the field of digital audio signal processing technology, specifically relating to an adaptive audio object encoding and decoding method and apparatus applied to immersive audio systems. Background Technology

[0002] With the continuous development of augmented reality and virtual reality technologies, users have increasingly higher demands for spatial audio, hoping that it can respond to different environments displayed visually. How to effectively encode audio signals is a pressing issue that needs to be addressed to enhance the immersive experience of spatial audio.

[0003] Traditional stereo or multichannel coding typically encodes different signals as separate independent signals, with the bit rate directly proportional to the number of channels. This is therefore limited by current network bandwidth constraints. Channel-based coding systems, due to strict limitations on the number and placement of speakers, struggle to meet personalized user interaction needs. Spatial Audio Object Coding (SAOC), on the other hand, utilizes redundancy between audio objects by sharing spatial parameters among multiple objects, reducing the bit rate and enabling effective encoding of audio signals even with bandwidth fluctuations. It also adapts to different configurations, including single-channel, stereo, binaural, and multichannel, enhancing the immersive experience of spatial audio.

[0004] SAOC divides each frame of signal into 28 sub-bands and extracts the spatial parameters of multiple objects and the single-channel downmixed signal. By utilizing the redundancy between the object sub-bands, it achieves simultaneous transmission of multiple object signals at a lower bit rate. However, because multiple audio objects share spatial parameters, the decoded audio object contains signals from other objects, resulting in spectral aliasing distortion. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an adaptive audio object encoding and decoding method and apparatus for use in immersive audio systems, thereby solving the problem of severe aliasing distortion in the recovered object signals in existing object encoding methods.

[0006] To achieve the above objectives, the present invention provides an adaptive audio object encoding method applied to immersive audio systems, comprising the following steps:

[0007] Step A1: Input independent multi-audio objects to convert multi-channel time-domain audio signals into frequency-domain signals;

[0008] Step A2: Perform activity detection on the audio signal spectrum and calculate the merging factor;

[0009] Step A3: Based on the merging factor, adaptive subband grouping is performed on the audio signal;

[0010] Step A4: Extract the side information of the audio object based on sub-band grouping;

[0011] Step A5: Compress the edge information to obtain a low-dimensional feature representation of the edge information;

[0012] Step A6: Quantize the low-dimensional feature representation of the edge information to obtain the edge information code stream;

[0013] Step A7: Downmix the audio frequency domain signal to obtain the downmixed signal, and use mono encoding to obtain the downmixed signal bitstream;

[0014] Step A8: Combine the side information bitstream and the downmixed signal bitstream to obtain the encoded bitstream.

[0015] Furthermore, in step A1, multiple audio object signals are acquired based on a preset sampling frequency to form a multi-channel audio signal. The audio signal is then processed by frame division, and the framed signal is windowed based on a window function with a preset overlap rate. The multi-channel audio signal is then transformed from the time domain to the frequency domain by an improved Discrete Cosine Transform (MDCT) to obtain the audio signal spectrum.

[0016] Furthermore, in step A2, a preset energy threshold algorithm is used to obtain the active state of a single frequency point of each frame of the audio object signal. The calculation method is as follows:

[0017]

[0018] In the formula, i is the frame number, m is the frequency point number, j is the audio object number, and TF j (i,m) is the frequency point active status flag, Ethh j () represents the set threshold, P j (,m) represents the energy at the frequency point.

[0019] Based on the active state, the merging factor of the binary values ​​is obtained, and the calculation method is as follows:

[0020]

[0021] In the formula, mF(i,m) is the merging factor of the binary values, and J is the number of objects.

[0022] Furthermore, in step A3, when the merging factor of a frequency point is 0, the frequency point and the next frequency point are merged into a sub-band; when the merging factor of a frequency point is 1, the frequency point is treated as a separate sub-band.

[0023] Furthermore, in step A6, a preset quantization table is used to quantize the low-dimensional feature representation of the side information to obtain the side information code stream.

[0024] This invention also provides an adaptive audio object decoding method for use in immersive audio systems, used to decode the encoding generated by the above-mentioned technical solutions, and specifically includes the following sub-steps:

[0025] Step B1: Decompose the encoded bitstream to obtain the side information bitstream and the downmixed signal bitstream;

[0026] Step B2: Dequantize the edge information bitstream to obtain the edge information decoded signal;

[0027] Step B3: Decompress the edge information decoding signal to obtain the reconstructed edge information;

[0028] Step B4: Decode the downmixed signal stream to obtain the reconstructed downmixed signal;

[0029] Step B5: The reconstructed side information and downmixed signal are synthesized to obtain a multi-audio object;

[0030] Step B6: Convert the multi-audio object into a time-domain audio signal;

[0031] Step B7: Render the multi-audio objects based on the rendering matrix, gain matrix, or HRTF parameters, and output personalized multi-audio object signals.

[0032] Furthermore, in step B2, a preset quantization table is used to look up the data corresponding to the quantization value of the side information to obtain the side information decoding signal.

[0033] Furthermore, in step B3, a deconvolutional decoder is used to recover high-dimensional side information from low-dimensional side information representation to obtain reconstructed side information.

[0034] Furthermore, in step B7, if a multi-speaker, multi-frequency object is used for playback, a rendering matrix or gain matrix is ​​used to render the time-domain signal of the decoded single object to obtain a personalized multi-audio object signal; if headphones are used for playback of the multi-audio object, an HRTF function is used to render the time-domain signal of the decoded single object to obtain a personalized multi-audio object signal.

[0035] The present invention also provides an adaptive audio object encoding and decoding device for use in an immersive audio system, comprising a conversion module, a detection module, a grouping module, an extraction module, an encoding module, a decoding module, a synthesis module, and a rendering module.

[0036] The conversion module is used to acquire multi-channel audio signals and convert them into audio frequency domain signals.

[0037] The detection module is used to detect the activity of each object and calculate the merging factor.

[0038] The grouping module is used to adaptively group each frame of signal into subbands based on the merging factor.

[0039] The extraction module is used to extract edge information based on sub-band grouping.

[0040] The encoding module is used to encode the side information and the downmixed signal to obtain the side information and downmixed signal bitstream.

[0041] The decoding module is used to decode the side information and the downmixed signal to obtain the decoded side information and downmixed signal.

[0042] The synthesis module is used to reconstruct the audio object signal from the decoded side information and the downmixed signal.

[0043] The rendering module is used to render multiple audio objects based on a rendering matrix, gain matrix, or HRTF function, and output personalized multi-audio object signals.

[0044] Compared with the prior art, the present invention has the following advantages:

[0045] This invention transmits multiple audio signals simultaneously, performs adaptive sub-band division based on the degree of aliasing, and compresses the signals using the nonlinear relationship between side information. It features low bit rate and high audio quality, enhances the immersive experience of spatial audio, and meets users' immersive listening needs. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating the encoding method of an embodiment of the present invention.

[0047] Figure 2 This is a flowchart illustrating the decoding method according to an embodiment of the present invention.

[0048] Figure 3 This is a schematic diagram of the structure of the encoding and decoding device according to an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0050] Currently, in many mature spatial audio applications, although object coding technology is employed, each frame of signal is typically divided into 28 subbands. Because multiple audio objects share spatial parameters, the decoded audio object contains signals from other objects, resulting in spectral aliasing distortion. To address this technical problem, this invention provides an adaptive audio object encoding / decoding method and apparatus for immersive audio systems, which adaptively groups audio signals into subbands based on the activity of the audio objects. Figure 1 This is a flowchart of the adaptive audio object encoding method for immersive audio systems provided by the present invention, as shown below. Figure 1 As shown, it includes the following steps:

[0051] Step A1: Input an independent multi-audio object to convert the multi-channel time-domain audio signal into a frequency-domain signal.

[0052] Multiple audio object signals are acquired based on a preset sampling frequency to form a multi-channel audio signal. The sampling frequency of the audio signal can be 11025, 22050, 24000, 44100, 48000 Hz, etc., and this embodiment uses 48000 Hz. The audio signal is segmented into frames, and the segmented signal is windowed based on a window function with a preset overlap rate. The multi-channel audio signal is transformed from the time domain to the frequency domain by the Modified Discrete Cosine Transform (MDCT) to obtain the audio signal spectrum. Commonly used window functions include Hamming window, Hanning window, Kaiser-Bessel window, etc., and common overlap methods such as 50% and 2 / 3 overlap can be used. The number of points in the MDCT transform can be common values ​​such as 1024 and 2048. This embodiment selects a Kaiser-Bessel window with 50% overlap and a window length of 1024 sampling points.

[0053] Step A2: Perform activity detection on the audio signal spectrum and calculate the merging factor.

[0054] The active state of a single frequency point in each frame of the audio object is obtained by using a preset energy threshold algorithm on the input audio frequency domain signal. The calculation method is as follows:

[0055]

[0056] In the formula, i is the frame number, m is the frequency point number, j is the audio object number, and TF j (i,m) is the frequency point active status flag, Ethh j () represents the set threshold, P j (,m) represents the energy at the frequency point.

[0057] Based on the active state, the merging factor of the binary values ​​is obtained, and the calculation method is as follows:

[0058]

[0059] In the formula, mF(i,m) is the merging factor of the binary values, and J is the number of objects.

[0060] Step A3: Based on the merging factor, adaptive subband grouping is performed on the audio signal.

[0061] When the merging factor of a frequency point is 0, the frequency point and the next frequency point are merged into a sub-band. When the merging factor of a frequency point is 1, the frequency point is treated as a separate sub-band.

[0062] Step A4: Extract the side information of the audio object based on sub-band grouping.

[0063] Based on the sub-band grouping determined by the frequency activity of the object, the side information of the audio object is extracted.

[0064] Step A5: Compress the edge information to obtain a low-dimensional feature representation of the edge information.

[0065] A convolutional neural network and a residual block encoder are used to compress high-dimensional edge information to obtain a low-dimensional representation of the edge information.

[0066] Step A6: Quantize the low-dimensional feature representation of the edge information to obtain the edge information code stream.

[0067] The low-dimensional feature representation of the side information is quantized using a preset quantization table to obtain the side information code stream.

[0068] Step A7: The audio frequency domain signal is downmixed to obtain a downmixed signal, and then a downmixed signal bitstream is obtained using mono encoding.

[0069] Encoders such as AAC, Opus, and Speex can be used to encode single-channel downmixed signals. In this embodiment, an AAC encoder is selected.

[0070] Step A8: Combine the side information bitstream and the downmixed signal bitstream to obtain the encoded bitstream.

[0071] The quantized side information bitstream, the mono-coded downmixed signal, and the side information bitstream are combined to obtain the coded bitstream output.

[0072] Figure 2 This is a flowchart of the adaptive audio object decoding method applied to an immersive audio system provided by the present invention, as shown below. Figure 2 As shown, it includes the following steps:

[0073] Step B1: Decompose the encoded bitstream to obtain the side information bitstream and the downmixed signal bitstream.

[0074] Step B2: Dequantize the edge information bitstream to obtain the edge information decoded signal.

[0075] The data corresponding to the quantized value of the side information is found by using a preset quantization table, and the side information decoding signal is obtained.

[0076] Step B3: Decompress the edge information decoding signal to obtain the reconstructed edge information.

[0077] A deconvolutional decoder is used to recover high-dimensional side information from low-dimensional side information representation, thus obtaining the reconstructed side information.

[0078] Step B4: Decode the downmixed signal stream to obtain the reconstructed downmixed signal.

[0079] The single-channel downmixed signal is decoded using decoders such as AAC, Opus, and Speex to obtain the reconstructed downmixed signal. In this embodiment, the AAC decoder is selected.

[0080] Step B5 involves synthesizing the reconstructed side information and the downmixed signal to obtain a multi-audio object.

[0081] The reconstructed edge information is multiplied with the downmixed signal to recover the object signal energy.

[0082] Step B6 converts the multi-audio object into a time-domain audio signal.

[0083] Multiple audio object signals are transformed from the frequency domain to the time domain to obtain multiple object time domain signals.

[0084] Step B7: Render the multi-audio object based on the rendering matrix, gain matrix, or HRTF function, and output a personalized multi-audio object signal.

[0085] If a multi-speaker, multi-frequency object is used for playback, a rendering matrix or gain matrix is ​​used to render the time-domain signal of the decoded individual object, resulting in a personalized multi-audio object signal. If headphones are used for playback of multi-audio objects, a Head Related Transfer Function (HRTF) is used to render the time-domain signal of the decoded individual object, resulting in a personalized multi-audio object signal. In this embodiment, after restoring the original audio object signal, a renderer is applied to the spatial orientation of the audio, and different rendering techniques are used for different playback devices to achieve personalized multi-audio objects.

[0086] like Figure 3As shown, the present invention also provides an adaptive audio object encoding / decoding device for use in an immersive audio system, comprising a conversion module 41, a detection module 42, a grouping module 43, an extraction module 44, an encoding module 45, a decoding module 46, a synthesis module 47, and a rendering module 48. Specifically: the conversion module 41 is used to acquire multi-channel audio signals and convert them into audio frequency domain signals; the detection module 42 is used to detect the activity of each object and calculate the merging factor; the grouping module 43 is used to adaptively group each frame of signal according to the merging factor; the extraction module 44 is used to extract side information based on the sub-band grouping; the encoding module 45 is used to encode the side information and the downmixing signal to obtain the side information and downmixing signal bitstream; the decoding module 46 is used to decode the side information and the downmixing signal to obtain the decoded side information and downmixing signal; the synthesis module 47 is used to reconstruct the audio object signal from the decoded side information and the downmixing signal; and the rendering module 48 is used to render multiple audio objects based on a rendering matrix, a gain matrix, or an HRTF function, outputting personalized multi-audio object signals.

[0087] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to replace them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. An adaptive audio object coding method applied in an immersive audio system, characterized in that, The method comprises the following steps: Step A1, inputting independent multi-audio objects, converting a multi-channel time-domain audio signal into a frequency-domain signal; Step A2, performing activity detection on the spectrum of the audio signal, and calculating a merging factor; An energy threshold algorithm is used to obtain the activity state of each frequency point of each frame of the audio object from the input audio frequency-domain signal, and the specific calculation method is as follows: （1） In the formula, is a frame number, is a frequency point number, is an audio object number, is a frequency point active state flag, is a set threshold value, is a frequency point energy; According to the activity state, a binary merging factor is obtained, and the specific calculation method is as follows: （2） wherein a merge factor that is a binary value, is the number of objects; Step A3, performing adaptive sub-band grouping on the audio signal based on the merging factor; When the merging factor of a frequency point is 0, the frequency point and the next frequency point are merged into one sub-band; when the merging factor of a frequency point is 1, the frequency point is taken as a sub-band alone; Step A4, extracting side information of the audio object based on the sub-band grouping; Step A5, compressing the side information to obtain a low-dimensional feature expression of the side information; Step A6, quantizing the low-dimensional feature expression of the side information to obtain a side information code stream; Step A7, downmixing the audio frequency-domain signal to obtain a downmix signal, and using single-channel encoding to obtain a downmix signal code stream; Step A8, synthesizing the side information code stream and the downmix signal code stream to obtain an encoded code stream.

2. A method for adaptive audio object coding for use in immersive audio systems as defined in claim 1, characterized by: In step A1, a plurality of audio object signals are collected based on a preset sampling frequency to form a multi-channel audio signal. The audio signal is frame processed, and the frame processed signal is window processed based on a preset overlap rate window function. The multi-channel audio signal is converted from time domain to frequency domain through improved MDCT to obtain the spectrum of the audio signal.

3. A method for adaptive audio object coding for use in immersive audio systems as defined in claim 1, characterized by: In step A6, the low-dimensional feature expression of the side information is quantized using a preset quantization table to obtain a side information code stream.

4. An adaptive audio object decoding method applied in an immersive audio system, characterized in that: The method is used for decoding the encoding generated by the method of any one of claims 1-3, and the specific implementation comprises the following steps: Step B1, decomposing the encoded code stream to obtain a side information code stream and a downmix signal code stream; Step B2, dequantizing the side information code stream to obtain a side information decoding signal; Step B3, decompressing the side information decoding signal to obtain reconstructed side information; Step B4, decoding the downmix signal code stream to obtain a reconstructed downmix signal; Step B5, synthesizing the reconstructed side information and the downmix signal to obtain multi-audio objects; Step B6, converting the multi-audio objects into time-domain audio signals; Step B7, rendering the multi-audio objects based on a rendering matrix or a gain matrix or an HRTF function to output personalized multi-audio object signals.

5. A method for adaptive audio object decoding for use in an immersive audio system as defined in claim 4, characterized by: In step B2, the data corresponding to the side information quantization value is obtained by searching a preset quantization table, to obtain a side information decoding signal.

6. A method for adaptive audio object decoding for use in an immersive audio system as defined in claim 4, characterized by: In step B3, the high-dimensional side information is recovered from the low-dimensional side information expression by using a deconvolution decoder, to obtain reconstructed side information.

7. A method for adaptive audio object decoding for use in an immersive audio system as defined in claim 4, characterized by: In step B7, if multi-audio box multi-frequency objects are used for playback, the decoded single object time-domain signal is rendered by using a rendering matrix or a gain matrix to obtain personalized multi-audio object signals; if earphones are used for multi-audio object playback, the decoded single object time-domain signal is rendered by using an HRTF function to obtain personalized multi-audio object signals.

8. An adaptive audio object codec apparatus for use in an immersive audio system, characterized by: The system comprises a conversion module, a detection module, a grouping module, an extraction module, an encoding module, a decoding module, a synthesis module and a rendering module; the conversion module is used for collecting multi-channel audio signals and converting the multi-channel audio signals into audio frequency domain signals; The detection module is used for detecting the activity of each object and calculating a merging factor; the grouping module is used for adaptively grouping each frame of signal according to the merging factor; the extraction module is used for extracting side information according to the sub-band grouping; and the encoding module is used for encoding the side information and downmix signals to obtain side information and downmix signal code streams; The decoding module is used for decoding the side information and downmix signals to obtain decoded side information and downmix signals; and the synthesis module is used for reconstructing the audio object signals from the decoded side information and downmix signals; The rendering module is used for rendering the multi-audio objects based on a rendering matrix or a gain matrix or an HRTF function and outputting personalized multi-audio object signals.

Images