Method and apparatus for processing audio for classifying a scene

Abstract

The audio processing method comprises the following steps: obtaining a first audio signal corresponding to a first frame; extracting a first feature vector by inputting the first audio signal to a first neural network; obtaining a temporal correlation vector representing a similarity between the first feature vector and at least one second feature vector extracted from at least one second audio signal corresponding to at least one second frame temporally preceding the first frame; and classifying the scene of the first audio signal by inputting the first feature vector, the at least one second feature vector, and the temporal correlation vector into a second neural network.

Description

Technical Field

[0001] The present disclosure relates to a method and apparatus for scene classification of audio, and more particularly, to a method and apparatus for classifying scenes of audio signals and processing the audio signals according to results of the scene classification. Background Art

[0002] According to the expansion of over-the-top (OTT) services, the increase in the resolution of televisions (TVs), and the enlargement of screens of electronic devices (such as tablets), the demand of viewers who want to experience immersive sound (such as theater content in a home environment) is increasing. In order to meet such viewer needs, it is necessary to process audio signals according to scene types by classifying the scenes of audio signals.

[0003] In addition, it is necessary to process an audio signal of a three-dimensional audio channel layout in which channels are arranged in front of a listener (three-dimensional audio channel layout in front of a listener) by considering the sound representation of entities (sound sources) on the screen. When processing an audio signal of a three-dimensional audio channel layout, it is necessary to perform efficient downmixing by classifying scenes of the audio signal.

[0004] Therefore, due to the increase in the number of scene types, there is an emerging demand for an audio scene classification module with a low latency, high classification accuracy and low complexity structure for audio signal scene classification. Summary of the invention

[0005] Solution to the problem

[0006] Provided are a method and apparatus for processing an audio signal from a lower channel layout of a multi-channel audio signal by using a scene classification result.

[0007] Provided are a method and apparatus for down-mixing or up-mixing a multi-channel audio signal according to a scene classification result.

[0008] According to an embodiment of the present disclosure, an audio processing method may include: obtaining a first audio signal corresponding to a first frame, extracting a first feature vector by inputting the first audio signal into a first neural network, obtaining a time correlation vector representing the similarity between the first feature vector and at least one second feature vector extracted from at least one second audio signal corresponding to at least one second frame temporally preceding the first frame, and classifying a scene of the first audio signal by inputting the first feature vector, at least one second feature vector, and the time correlation vector into a second neural network.

[0009] According to an embodiment of the present disclosure, an audio processing device includes a memory storing at least one instruction, and at least one processor configured to execute at least one instruction stored in the memory. The at least one processor can be configured to execute at least one instruction to obtain a first audio signal corresponding to a first frame, extract a first feature vector by inputting the first audio signal into a first neural network, obtain a time correlation vector representing the similarity between the first feature vector and at least one second feature vector extracted from at least one second audio signal corresponding to at least one second frame temporally preceding the first frame, and classify the scene of the first audio signal by inputting the first feature vector, at least one second feature vector, and the time correlation vector into a second neural network.

[0010] According to an embodiment of the present disclosure, an audio processing method includes: obtaining downmix related information including a downmix audio signal corresponding to an original audio signal and a scene classification result from a bitstream, demixing the downmix audio signal based on the downmix related information, and reconstructing an audio signal including at least one channel based on the demixed audio signal. The scene classification result can be obtained based on a time correlation vector between a first feature vector corresponding to a first frame of the original audio signal and at least one second feature vector corresponding to at least one second frame that is temporally preceding the first frame of the original audio signal.

[0011] According to an embodiment of the present disclosure, an audio processing device includes a memory storing at least one instruction, and at least one processor configured to execute the at least one instruction. The at least one processor may be further configured to obtain downmix related information including a downmix audio signal and a scene classification result from an original audio signal, demix the downmix audio signal based on the downmix related information, and reconstruct the audio signal based on the demix audio signal. The scene classification result may be obtained based on a time correlation vector between a first feature vector corresponding to a first frame of the original audio signal and at least one second feature vector corresponding to at least one second frame that is temporally preceding the first frame of the original audio signal.

[0012] According to an embodiment of the present disclosure, there is provided a computer-readable recording medium having recorded thereon a program for executing an audio processing method on a computer. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which:

[0014] Figure 1A is a block diagram of an audio scene classifier according to an embodiment of the present disclosure;

[0015] Figure 1BIs used to describe Figure 1A A block diagram of the detailed operation of the feature extractor in ;

[0016] Figure 1C Is used to describe Figure 1A A block diagram of the detailed operation of the temporal correlation analyzer in FIG.

[0017] Figure 1D Is used to describe Figure 1A A block diagram of the detailed operation of the estimator in ;

[0018] Figure 2A is a block diagram of a multi-channel audio signal processor according to an embodiment of the present disclosure;

[0019] Figure 2B is a diagram showing a scene classification result according to an embodiment of the present disclosure;

[0020] Figure 3A is a diagram for describing a scalable audio channel layout structure according to an embodiment of the present disclosure;

[0021] Figure 3B is a diagram for describing an example of a scalable audio channel layout structure;

[0022] Figure 4A is a block diagram of an audio encoding device according to an embodiment of the present disclosure;

[0023] Figure 4B is a block diagram of an audio encoding device according to an embodiment of the present disclosure;

[0024] Figure 4C is a block diagram of a multi-channel audio signal processor according to an embodiment of the present disclosure;

[0025] Figure 4D is a diagram for describing the detailed operation of an audio signal classifier;

[0026] Figure 5A is a block diagram of a multi-channel audio decoding device according to an embodiment of the present disclosure;

[0027] Figure 5B is a block diagram of a multi-channel audio decoding device according to an embodiment of the present disclosure;

[0028] Figure 5C is a block diagram of a multi-channel audio signal reconstructor according to an embodiment of the present disclosure;

[0029] Figure 5D is a block diagram of an upmix channel audio generator according to an embodiment of the present disclosure;

[0030] Fig. 6A is a block diagram of an audio encoding device according to an embodiment of the present disclosure;

[0031] Figure 6B is a block diagram of an error removal related information generator according to an embodiment of the present disclosure;

[0032] Fig. 7A is a block diagram of an audio decoding device according to an embodiment of the present disclosure;

[0033] Figure 7B is a block diagram of a multi-channel audio signal reconstructor according to an embodiment of the present disclosure;

[0034] Fig. 8A is a diagram for describing the order and rule of transmitting the audio stream in each channel group by the audio encoding device according to an embodiment of the present disclosure;

[0035] Figure 8B and Figure 8C Each is a diagram for describing a mechanism of gradual down-mixing according to an embodiment of the present disclosure;

[0036] Fig.9A is a block diagram of an audio encoding device according to an embodiment of the present disclosure;

[0037] Fig. 9B is a block diagram of an audio encoding device according to an embodiment of the present disclosure;

[0038] Fig. 10A is a block diagram of an audio decoding device according to an embodiment of the present disclosure;

[0039] Fig. 10B is a block diagram of an audio decoding device according to an embodiment of the present disclosure;

[0040] Fig.11 is a block diagram of an audio processing device according to an embodiment of the present disclosure;

[0041] Fig.12 is a block diagram of an audio processing device according to an embodiment of the present disclosure;

[0042] Fig.13 is a flowchart of an audio processing method according to an embodiment of the present disclosure;

[0043] Fig.14A is a flowchart of an audio processing method according to an embodiment of the present disclosure;

[0044] Fig. 14B is a flowchart of an audio processing method according to an embodiment of the present disclosure;

[0045] Fig.15 is a flowchart of an audio processing method according to an embodiment of the present disclosure;

[0046] Fig.16is a flowchart of an audio processing method according to an embodiment of the present disclosure;

[0047] Fig.17 is a flowchart of an audio processing method according to an embodiment of the present disclosure; and

[0048] Fig.18 is a block diagram of an audio processing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0049] All terms used herein, including descriptive or technical terms, should be interpreted as having meanings that are obvious to those of ordinary skill in the art. However, these terms may have different meanings according to the intentions of those of ordinary skill in the art, precedents, or the emergence of new technologies. In addition, the applicant may arbitrarily select some terms, and in this case, the meaning of the selected terms will be described in detail in the specific embodiments of the present disclosure. Therefore, the terms used herein must be defined together with the description in the entire specification based on the meaning of the terms.

[0050] Unless it has a significantly different meaning in the context, an expression used in the singular may encompass an expression in the plural. The terms used herein (including technical or scientific terms) may have the same meaning as those generally understood by those of ordinary skill in the art described in this disclosure. In addition, terms including ordinal numbers, such as "first" or "second" used herein, may be used to describe various elements, but those elements are not limited by the above terms. The above terms are only used to distinguish one element from another element.

[0051] When a component "includes" or "contains" an element, unless there is a specific description to the contrary, the component may also include other elements without excluding other elements. Terms such as "unit" or "module" used in the present disclosure indicate a unit that processes at least one function or movement, and the unit and module may be implemented by hardware or software, or by a combination of hardware and software.

[0052] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement the present disclosure. However, embodiments of the present disclosure can be implemented in many different forms and are not limited to those discussed herein. In addition, in the accompanying drawings, in order to clearly describe the present disclosure, parts that are not related to the description are omitted, and the same reference numerals represent the same elements throughout the specification.

[0053] Herein, “deep neural network (DNN)” or “neural network” is a representative example of an artificial neural network model that simulates brain nerves, and is not limited to an artificial neural network model that uses a specific algorithm.

[0054] Herein, "parameters" may be values ​​used in the operation process of each layer constituting a neural network, and may include, for example, weights (and biases) used when applying input values ​​to a predetermined calculation formula. Parameters may be expressed in the form of a matrix. For example, a parameter in the form of a matrix may be referred to as a "kernel" or a "filter". Parameters may be values ​​set as a result of training, and may be updated as needed by separate training data.

[0055] In this specification, a "hyperparameter" is a value that changes according to user or manufacturer settings in order to train a neural network, and may include, for example, the number of layers, the number of neurons included in each layer, activation function, learning rate, optimizer, batch size, iteration count, dropout probability, etc., but is not limited to these.

[0056] In the specification, an “audio processing device” refers to any device that takes an audio signal as input and processes the input audio signal, and for example, the “audio processing device” may include an “audio scene classifier”, an “audio encoding device” and / or an “audio decoding device”.

[0057] Herein, a "multi-channel audio signal" may mean an audio signal of n channels (where n is an integer greater than 2). A "mono-channel audio signal" may be a one-dimensional audio signal, a "stereo channel audio signal" may be a two-dimensional audio signal, and a "multi-channel audio signal" may be a three-dimensional audio signal.

[0058] Herein, "channel (speaker) layout" may refer to a combination of at least one channel, and may specify a spatial arrangement of channels (speakers). A channel used herein is a channel through which an audio signal is actually output, and thus may be referred to as a rendering channel.

[0059] For example, the channel layout may be an "XYZ channel layout". Herein, X may be the number of surround channels, Y may be the number of subwoofer channels, and Z may be the number of height channels. The channel layout may specify the spatial positions of surround channels / subwoofer channels / height channels.

[0060] Examples of “channel (speaker) layout” may include 1.0.0 channel (or mono channel) layout, 2.0.0 channel (or stereo channel) layout, 5.1.0 channel layout, 5.1.2 channel layout, 5.1.4 channel layout, 7.1.0 layout, 7.1.2 layout, and 3.1.2 channel layout, but “channel layout” is not limited thereto and various other channel layouts may exist.

[0061] The channels specified by “channel (speaker) layout” may be called various names, but may be collectively named for ease of explanation.

[0062] The channels that make up a "channel (speaker) layout" may be named based on the channels' respective spatial locations.

[0063] For example, the first surround channel of the 1.0.0 channel layout may be named a single channel. For the 2.0.0 channel layout, the first surround channel may be named an L2 channel, and the second surround channel may be named an R2 channel.

[0064] Herein, "L" indicates a channel located on the left side of the listener, and "R" indicates a channel located on the right side of the listener. "2" indicates that the number of surround channels is 2. "L" may also be referred to as "FL", and "R" may also be referred to as "FR".

[0065] For a 5.1.0 channel layout, a first surround channel may be named an L5 channel, a second surround channel may be named an R5 channel, a third surround channel may be named a C channel, a fourth surround channel may be named an Ls5 channel, and a fifth surround channel may be named an Rs5 channel. Herein, "C" indicates a channel located at the center of the listener. "s" refers to a channel located on one side. The first subwoofer channel of the 5.1.0 channel layout may be named a low frequency effect (LFE) channel. Herein, LFE may refer to a low frequency effect. In other words, the LFE channel may be a channel for outputting low frequency sound effects.

[0066] The surround channels of the 5.1.2 channel layout and the 5.1.4 channel layout may be named the same as the surround channels of the 5.1.0 channel layout. Similarly, the subwoofer channels of the 5.1.2 channel layout and the 5.1.4 channel layout may be named the same as the subwoofer channels of the 5.1.0 channel layout.

[0067] 5.1.2 The first height channel of the channel layout can be named H15 channel. Here, H represents the height channel. The second height channel can be named Hr5 channel.

[0068] For a 5.1.4 channel layout, the first height channel may be named Hfl channel, the second height channel may be named Hfr channel, the third height channel may be named Hbl channel, and the fourth height channel may be named Hbr channel. Here, f indicates a front channel relative to a listener, and b indicates a rear channel relative to a listener.

[0069] For a 7.1.0 channel layout, the first surround channel may be named L channel, the second surround channel may be named R channel, the third surround channel may be named C channel, the fourth surround channel may be named Ls channel, the fifth surround channel may be named Rs channel, the sixth surround channel may be named Lb channel, and the seventh surround channel may be named Rb channel.

[0070] The surround channels of the 7.1.2 channel layout and the 7.1.4 channel layout may be named the same as the surround channels of the 7.1.0 channel layout. Similarly, the corresponding subwoofer channels of the 7.1.2 channel layout and the 7.1.4 channel layout may be named the same as the subwoofer channels of the 7.1.0 channel layout.

[0071] For a 7.1.2 channel layout, the first height channel may be named Hl7 channel, and the second height channel may be named Hr7 channel.

[0072] For a 7.1.4 channel layout, a first height channel may be named an Hfl channel, a second height channel may be named an Hfr channel, a third height channel may be named an Hbl channel, and a fourth height channel may be named an Hbr channel.

[0073] For a 3.1.2 channel layout, a first surround channel may be named an L3 channel, a second surround channel may be named an R3 channel, and a third surround channel may be named a C channel. A first subwoofer channel of a 3.1.2 channel layout may be named an LFE channel. For a 3.1.2 channel layout, a first height channel may be named an Hfl3 channel (or a Tl channel), and a second height channel may be named an Hfr3 channel (or a Tr channel).

[0074] In this article, some channels may be named differently according to the channel layout, but may represent the same channel. For example, the H15 channel and the H17 channel may be the same channel. Similarly, the Hr5 channel and the Hr7 channel may be the same channel.

[0075] The channels are not limited to the above-mentioned channel names, and various other channel names may be used.

[0076] For example, the L2 channel may be named L" channel, the R2 channel may be named R" channel, the L3 channel may be named ML3(L') channel, the R3 channel may be named MR3(R') channel, the Hfl3 channel may be named MHL3 channel, the Hfr3 channel may be named MHR3 channel, the Ls5 channel may be named MSL5(Ls') channel, the Rs5 channel may be named MSR5 channel, the Hl5 channel may be named MHL5(Hl') channel. The Hr5 channel may be named MHR5(Hr') channel, and the C channel may be named MC channel.

[0077] The channels of the channel layout for the above layout can be named as in Table 1.

[0078] Table 1

[0079] Channel layout Channel Name 1.0.0 Single Channel 2.0.0 L2 / R2 5.1.0 L5 / C / R5 / Ls5 / Rs5 / LFE 5.1.2 L5 / C / R5 / Ls5 / Rs5 / Hl5 / Hr5 / LFE 5.1.4 L5 / C / R5 / Ls5 / Rs5 / Hfl / Hfr / Hbl / Hbr / LFE 7.1.0 L / C / R / Ls / Rs / Lb / Rb / LFE 7.1.2 L / C / R / Ls / Rs / Lb / Rb / Hl7 / Hr7 / LFE 7.1.4 L / C / R / Ls / Rs / Lb / Rb / Hfl / Hfr / Hbl / Hbr / LFE 3.1.2 L3 / C / R3 / Hfl3 / Hfr3 / LFE

[0080] "Transmission channel" is a channel for transmitting compressed audio signals, and a part of the "transmission channel" may be the same as the "presentation channel", but is not limited thereto, and another part of the "transmission channel" may be a channel (mixed channel) of an audio signal in which the audio signal of the presentation channel is mixed. In other words, the "transmission channel" may be a channel containing the audio signal of the "presentation channel", but may be a channel of a mixed channel in which a part of the channel is the same as the presentation channel and the remainder is different from the presentation channel. The "transmission channel" may be named to be distinguished from the "presentation channel". For example, when the transmission channel is an A / B channel, the A / B channel may contain an audio signal of an L2 / R2 channel. When the transmission channel is a T / P / Q channel, the T / P / Q channel may contain an audio signal of a C / IFE / Hfl3 and Hfr3 channel. When the transmission channel is an S / U / V channel, the S / U / V channel may contain an audio signal of an L and R / Ls and an Rs / Hfl and Hfr channel.

[0081] In this specification, a 'three-dimensional audio signal' may refer to an audio signal for detecting the distribution of sound and the position of a sound source in a three-dimensional space.

[0082] In the present specification, a “three-dimensional audio channel in front of the listener” may refer to a three-dimensional audio channel based on the layout of the audio channel in front of the listener. A “three-dimensional audio channel in front of the listener” may be referred to as a “front 3D audio channel”. In particular, a “3D audio channel in front of the listener” may be referred to as a “screen-centered 3D audio channel” because it is a 3D audio channel based on the layout of the audio channels arranged around a screen located in front of the listener.

[0083] In this specification, a “3D audio channel that omnidirectionally surrounds the listener” may mean a 3D audio channel based on the layout of the audio channel that is omnidirectionally arranged around the listener. A “3D audio channel that omnidirectionally surrounds the listener” may be referred to as a “full 3D audio channel”. In this document, omnidirectional may mean all directions including the front direction, the side direction, and the rear direction. In particular, a “3D audio channel that omnidirectionally surrounds the listener” may also be referred to as a “listener-centric 3D audio channel” because it is a 3D audio channel based on the layout of the audio channel that is omnidirectionally arranged around the listener.

[0084] In the present specification, a "channel group" as a data unit may include a (compressed) audio signal of at least one channel. More specifically, the channel group may include at least one of a basic channel group independent of another channel group or a subordinate channel group dependent on at least one channel group. In this case, the target channel group on which the subordinate channel group depends may be another subordinate channel group, and may be a subordinate channel group associated with a lower channel layout. Alternatively, the channel group on which the subordinate channel group depends may be a basic channel group. A "channel group" may be referred to as a "coding group" because it includes data of the channel group. A subordinate channel group used to further expand the number of channels from the channels included in the basic channel group may be referred to as a scalable channel group or an extended channel group.

[0085] The audio signal of the "basic channel group" may include an audio signal of a single channel or an audio signal of a stereo channel. Without being limited thereto, the audio signal of the "basic channel group" may include an audio signal of a 3D audio channel in front of the listener.

[0086] For example, the audio signal of the "slave channel group" may include audio signals of channels other than the audio signal of the "basic channel group" between the audio signal of the 3D audio channel in front of the listener and the audio signal of the 3D audio channel surrounding the listener omnidirectionally. In this case, a portion of the audio signal of another channel may be an audio signal in which the audio signal of at least one channel is mixed (i.e., an audio signal of a mixed channel).

[0087] For example, the audio signal of the "basic channel group" can be a single-channel audio signal or a stereo channel audio signal. The "multi-channel audio signal" reconstructed based on the audio signals of the "basic channel group" and the "slave channel group" can be an audio signal of a 3D audio channel in front of the listener or an audio signal of a 3D audio channel omnidirectionally surrounding the listener.

[0088] In the present specification, 'up-mixing' may mean an operation in which the number of presentation channels of an output audio signal is increased compared to the number of presentation channels of an input audio signal through de-mixing.

[0089] In the present specification, "demixing" may mean an operation of separating an audio signal of a specific channel from an audio signal in which audio signals of various channels are mixed (i.e., an audio signal of a mixed channel), and may mean one of the mixing operations. In this case, "demixing" may be implemented as a calculation using an "demixing matrix" (or a "downmixing matrix" corresponding thereto), and the "demixing" matrix may include at least one "demixing weight parameter" (or a "downmixing weight parameter" corresponding thereto) as a coefficient of the demixing matrix (or a "downmixing matrix" corresponding thereto). The "demixing weight parameter" may also be referred to as a "demixing parameter", and the "downmixing weight parameter" may also be referred to as a "downmixing parameter". Alternatively, "demixing" may be implemented as an arithmetic calculation based on a portion of the "demixing matrix" (or a "downmixing matrix" corresponding thereto), and may be implemented in various ways, but is not limited thereto. As described above, "demixing" may be related to "upmixing".

[0090] “Mixing” may mean any operation of generating an audio signal of a new channel (ie, mixed channel) by summing values ​​obtained by multiplying each of the audio signals of a plurality of channels by a corresponding weight (ie, by mixing the audio signals of the plurality of channels).

[0091] "Mixing" can be divided into "mixing" performed by an audio encoding device in a narrow sense and "demixing" performed by an audio decoding device.

[0092] The "mixing" performed in the audio encoding device can be implemented as a calculation using a "(down) mixing matrix", and the "(down) mixing matrix" may include at least one "(down) mixing weight parameter" as a coefficient of the (down) mixing matrix. Alternatively, the "(down) mixing" may be implemented as an arithmetic calculation based on a part of the "(down) mixing matrix", and may be implemented in various ways without limitation.

[0093] In the present specification, an "upmix channel group" may mean a group including at least one upmix channel, and an "upmix channel" may mean a demix channel separated by demixing with respect to an audio signal of a coding / decoding channel. An "upmix channel group" in a narrow sense may include an "upmix channel". However, an "upmix channel group" in a broad sense may also include an "coding / decoding channel" as well as an "upmix channel". Here, an "coding / decoding channel" may mean an independent channel of an audio signal that is encoded (compressed) and included in a bitstream or an independent channel of an audio signal obtained by decoding from a bitstream. In this case, in order to obtain the audio signal of the coding / decoding channel, a separate (de)mixing operation is not required.

[0094] The audio signal of the “upmixed channel group” in a broad sense may be a multi-channel audio signal, and the output multi-channel audio signal may be one of at least one multi-channel audio signal (i.e., the audio signal of at least one upmixed channel group) as an audio signal output through a device such as a speaker.

[0095] In the present specification, 'downmixing' may mean an operation in which the number of presentation channels of an output audio signal is reduced compared to the number of presentation channels of an input audio signal through mixing.

[0096] In the present specification, a 'factor for error removal' (or an error removal factor (ERF)) may be a factor for removing an error of an audio signal occurring due to lossy codec.

[0097] More specifically, the error of the audio signal occurring due to lossy coding may include an error caused by coding (quantization) based on psychoacoustic characteristics, etc. The “factor for error removal” may be referred to as a “coding error removal (CER) factor,” “error elimination ratio,” etc. In particular, the “error removal factor” may be referred to as a “scaling factor” because the error removal operation substantially corresponds to a scaling operation.

[0098] In the specification, "frame" may refer to a unit obtained by dividing an audio signal in the time domain. For example, frame t (t is a natural number) may be referred to as a "current frame". Here, t may be a natural number for identifying each of a series of frames. Frame tN (N is a natural number less than t) may be referred to as a "previous frame". The frame size (or interval) may be predefined by a manufacturer or user's setting.

[0099] Figure 1A is a block diagram of an audio scene classifier according to an embodiment of the present disclosure. Figure 1A , the audio scene classifier 100 may include a feature extractor 110 , a time correlation analyzer 120 , and an estimator 130 .

[0100] The audio scene classifier 100 may classify scenes corresponding to a series of frames of the audio signal. For example, the audio scene classifier 100 may classify each of the series of frames of the audio signal into a dialogue type, a music type, or a sound effect type. However, the present disclosure is not limited thereto, and the types and number of types of scenes that may be classified by the audio scene classifier 100 may be changed according to the settings of the manufacturer or the user.

[0101] The audio scene classifier 100 may use data corresponding to at least one previous frame to classify the scene of the current frame of the audio signal. For example, the audio scene classifier 100 may classify the scene of the current frame by using similarities between data corresponding to at least one previous frame and data corresponding to the current frame.

[0102] The feature extractor 110 may obtain (or receive) a first audio signal AS corresponding to a first frame (or may be referred to as a current frame) t Here, the first audio signal AS t It may be data obtained by visualizing corresponding speech data (eg, a spectrogram), but is not limited thereto.

[0103] The feature extractor 110 can extract the first audio signal AS t The feature extractor 110 can be based on the first audio signal AS t Extract the first eigenvector F t For example, the feature extractor 110 may extract the first audio signal AS by using t The first neural network is used as input to extract the first feature vector F t The feature extractor 110 can convert the first feature vector F t is transmitted to the time correlation analyzer 120. Figure 1B The detailed functions and operations of the first neural network are described in detail.

[0104] The temporal correlation analyzer 120 may obtain (or calculate) similarities between features corresponding to a plurality of frames. The temporal correlation analyzer 120 may receive a first feature vector F t and at least one second eigenvector F t-1 ,…,F t-N The time correlation analyzer 120 may be based on the first feature vector F t and at least one second eigenvector F t-1 ,…,F t-N To obtain the time correlation vector TC t At least one second feature vector F may be extracted from at least one second audio signal corresponding to at least one second frame before the first frame (or may also be referred to as a previous frame). t-1 ,…,F t-N In other words, the second frame may be at least one of a series of frames preceding the first frame. The number of second feature vectors may be N. Here, N may be a natural number. The time correlation vector TC t The first eigenvector F can be expressed as t and at least one second eigenvector F t-1 ,…,F t-N The similarities between.

[0105] The estimator 130 may analyze the first audio signal AS t The estimator 130 may receive a first feature vector F t , at least one second eigenvector F t-1 ,…,F t-M and the time correlation vector TC t Here, M may be a natural number. In an embodiment of the present disclosure, M may be less than or equal to N. When M is less than N, the first audio signal AS may be processed with a smaller amount of operations than when M is equal to N. t Classify the scenes.

[0106] The estimator 130 may use a second neural network to estimate the first audio signal AS t The second neural network classifies the first feature vector F t , at least one second eigenvector F t-1 ,…,F t-M and the time correlation vector TC t As input, the estimator 130 may output a signal corresponding to the first audio signal AS t The scene classification results SC t . refer to Figure 1D The detailed functions and operations of the second neural network are described in detail.

[0107] According to an embodiment of the present disclosure, the audio scene classifier 100 is configured in a single module including a feature extractor 110, a time correlation analyzer 120, and an estimator 130, regardless of the number of scene types, so that time delay due to an increase in the number of scene types does not occur.

[0108] Figure 1B yes Figure 1A FIG. 1 is a block diagram of the detailed operation of the feature extractor 110 in FIG. The feature extractor 110 may include a first neural network 112. The first neural network 112 may be based on the first audio signal AS t As input, output the first eigenvector F t . Omitted reference Figure 1A Details provided.

[0109] and Figure 1B Reference together Figure 1A , the feature extractor 110 can obtain the first audio signal AS t For example, the feature extractor 110 may receive a first audio signal AS in the form of a spectrogram. t .

[0110] In an embodiment of the present disclosure, the audio scene classifier 100 may further include a preprocessor 140. The preprocessor 140 may obtain (or receive) a multi-channel audio signal MCAS. The preprocessor 140 may obtain a first frame f from the multi-channel audio signal MCAS. t Corresponding signal. Frame (for example, the first frame f t ) can have a size of τ.

[0111] In an embodiment of the present disclosure, the preprocessor 140 may obtain t A first audio signal AS including at least one channel signal is extracted from the corresponding signal t For example, the multi-channel audio signal MCAS may include i channel signals. The preprocessor 140 may extract the first audio signal AS including k (k is less than i) channel signals from the i channel signals of the multi-channel audio signal MCAS. t (i and k are natural numbers).

[0112] In an embodiment of the present disclosure, the preprocessor 140 may transform the multi-channel audio signal into the time-frequency domain. For example, the preprocessor 140 may generate a spectrogram based on the multi-channel audio signal MCAS. The preprocessor 140 may generate a spectrogram for each of at least one channel signal in the multi-channel audio signal MCAS. In other words, the preprocessor 140 may generate a first audio signal AS expressed as a vector having time, frequency, and channel dimensions. t .

[0113] In an embodiment of the present disclosure, the multi-channel audio signal MCAS may have a 7.1.4 channel layout. In Table 1, the multi-channel audio signal MCAS may have L / C / R / Ls / Rs / Lb / Rb / Hfl / Hfr / Hbl / Hbr / LFE channel signals. In this case, i may be 12. For example, the preprocessor 140 may extract the L / C / R / Ls / Rs signal from the L / C / R / Ls / Rs / Lb / Rb / Hfl / Hfr / Hbl / Hbr / LFE channel signals. In this case, k may be 5. Therefore, the first audio signal AS t The sub-audio signal corresponding to each channel may be included (eg, AS t_1 ,AS t_2 ,AS t_3 ,AS t_4 and AS t_5 ).

[0114] In an embodiment of the present disclosure, the preprocessor 140 may downmix the multi-channel audio signal MCAS into five channel signals (e.g., L / C / R / Ls / Rs signals). In an embodiment of the present disclosure, the preprocessor 140 may downmix the multi-channel audio signal into a stereo signal. In an embodiment of the present disclosure, the preprocessor 140 may transform the downmixed signal into the time-frequency domain.

[0115] The first neural network 112 may include at least one convolution layer, a pooling layer, and a fully connected layer. The convolution layer obtains feature data by processing the input data with a filter of a predefined size. Before the feature data is transferred from the convolution layer to the next layer, an activation function may be used. For example, the activation function may be a sigmoid function, a hyperbolic tangent function, a softmax function, or a rectified linear unit (ReLU) function, but the present disclosure is not limited thereto. The parameters of the filter of the convolution layer may be optimized by the training process to be described below. The pooling layer, which is a layer for obtaining and outputting the feature values ​​of only some samples from the feature values ​​of all samples of the feature data to reduce the size of the input data, may include a maximum pooling layer and an average pooling layer. In an embodiment of the present disclosure, the pooling layer may be a maximum pooling layer. A fully connected layer in which each neuron of one layer is connected to each neuron of the next layer is a layer for classifying features. In a fully connected layer, an activation function may be used before outputting data.

[0116] exist Figure 1B , the first neural network 112 includes three convolutional layers, three pooling layers, and one fully connected layer. However, this is merely an example, and the present disclosure is not limited thereto. Therefore, the number of convolutional layers, the number of pooling layers, and the number of fully connected layers included in the first neural network 112 may be modified differently. Likewise, the number and size of filters used in each convolutional layer may be modified differently, and the connection and connection method between each layer may also be modified differently.

[0117] The first neural network 112 may receive a first audio signal AS t The first neural network 112 can use the first audio signal AS t As input, extract (generate or obtain) the first eigenvector F t In the embodiment of the present disclosure, the first eigenvector F t The first eigenvector F stored in the memory t In obtaining the frame corresponding to the next frame (eg, F t+1 ) of the time correlation vector (e.g. TC t+1 ) process.

[0118] In an embodiment of the present disclosure, the first neural network 112 may be trained to output a feature vector corresponding to each audio signal by receiving a data set including a vectorized audio signal as input. In other words, as the training is repeated, the parameters of the first neural network 112 for outputting the feature vector may be updated.

[0119] Figure 1C yes Figure 1A A block diagram of the detailed operation of the time correlation analyzer 120 in FIG. Figure 1A and Figure 1B The details provided are omitted. For example, in the following, it is assumed that M is 5, but the present disclosure is not limited thereto. Since M is 5, N may be greater than or equal to 5.

[0120] and Figure 1C Reference together Figure 1A , the time correlation analyzer 120 may receive the first feature vector F t and at least one second eigenvector F t-1 ,…,F t-5 The time correlation analyzer 120 may perform a first feature vector F t and at least one second eigenvector F t-1 ,…,F t-5 The time correlation analyzer 120 may perform the first feature vector F t and the first eigenvector F t The inner product of (i.e., the first eigenvector F t The time correlation analyzer 120 can obtain the time correlation vector TC t , the time correlation vector TC t Including the first eigenvector F t and at least one second eigenvector F t-1 ,…,F t-5 The inner product of each of the two and the first eigenvector F t With the first eigenvector F t The inner product of .

[0121] Figure 1D yes Figure 1A A block diagram of the detailed operation of the estimator 130 in FIG. Figures 1A to 1C The details provided are omitted.

[0122] and Figure 1D Reference together Figure 1A , the estimator 130 may include a cascade function 132 and a second neural network 134. The estimator 130 may estimator the first audio signal AS by using a feature vector of a current frame, at least one feature vector of at least one previous frame, and a time correlation vector corresponding to the current frame.t The estimator 130 may receive a first feature vector F t , at least one second eigenvector F t-1 ,…,F t-5 and the time correlation vector TC t .

[0123] The concatenation function 132 can convert the first eigenvector F t , at least one second eigenvector F t-1 ,…,F t-5 and the time correlation vector TC t The concatenation function 132 may output a concatenated vector.

[0124] The second neural network 134 may include at least one fully connected layer and an argmax function. Before outputting data from the fully connected layer, an activation function may be used. At least one fully connected layer may output a probability value of a predefined scene type by using a connected vector. In an embodiment of the present disclosure, an activation function (e.g., a softmax function) may be used to extract a probability value. The argmax function may output a scene type with the highest probability (i.e., a scene classification result SC ) among predefined scene types (e.g., a dialogue type, a music type, and a sound effect type). t ).

[0125] exist Figure 1D , the second neural network 134 includes two fully connected layers and an argmax function. However, this is only an example, and the present disclosure is not limited thereto. Therefore, the number of fully connected layers included in the second neural network 134 may be modified differently. Similarly, the connection order and method between each layer may also be modified differently.

[0126] The second neural network 134 may receive the connected vector. The second neural network 134 may output a scene classification result SC by taking the connected vector as input. t In the embodiment of the present disclosure, the first eigenvector F t , at least one second eigenvector F t-1 ,…,F t-5 and the time correlation vector TC t The second neural network 134 as input is used to process the first audio signal AS t Classify the scenes.

[0127] In an embodiment of the present disclosure, by taking a data set including a feature vector and a time correlation vector as input, the second neural network 134 can be trained to classify the scene corresponding to each audio signal. In other words, when the training is repeated, the parameters of the second neural network 134 for outputting the feature vector can be updated. In an embodiment of the present disclosure, the first neural network 112 and the second neural network 134 can be jointly trained. For example, the audio scene classifier 100 can be regarded as one neural network and trained.

[0128] Figure 2A is a block diagram of a multi-channel audio signal processor according to an embodiment of the present disclosure.

[0129] refer to Figure 2A , the multi-channel audio signal processor 200 may include an audio scene classifier 210 and a down-mix channel audio generator 220. The audio scene classifier 210 may include a feature extractor 212, a time correlation analyzer 214, and an estimator 216. The functions and operations of the audio scene classifier 210, the feature extractor 212, the time correlation analyzer 214, and the estimator 216 are similar to Figures 1A to 1D The functions and operations of the audio scene classifier 100, the feature extractor 110, the time correlation analyzer 120 and the estimator 130 are described herein, and their redundant descriptions are omitted.

[0130] The multi-channel audio signal processor 200 may obtain (or receive) a multi-channel audio signal corresponding to a plurality of frames including a first frame and at least one second frame. The multi-channel audio signal may include a plurality of channel signals. The multi-channel audio signal processor 200 may extract a first audio signal including at least one channel signal from the multi-channel audio signal.

[0131] The audio scene classifier 210 may classify the scene of the first audio signal.

[0132] The down-mix channel audio generator 220 may be based on the scene classification result SC t The multi-channel audio signal corresponding to the first frame is down-mixed. The down-mix channel audio generator 220 can obtain the corresponding scene classification result SC t The downmix profile 222 may include a downmix parameter for mixing a multi-channel audio signal into a multi-channel audio signal having a different channel layout. The downmix channel audio generator 220 may obtain a downmix parameter for mixing a multi-channel audio signal (hereinafter referred to as a "first multi-channel audio signal") into a multi-channel audio signal having a different channel layout (hereinafter referred to as a "second multi-channel audio signal") based on the downmix profile. For example, the downmix profile 222 may depend on the scene classification result SC t, the channel layout of the first multi-channel audio signal and the channel layout of the second multi-channel audio signal.

[0133] The downmix channel audio generator 220 can downmix the first multi-channel audio signal corresponding to the first frame according to the channel layout of the second multi-channel audio signal based on the downmix parameter. The downmix profile and / or the downmix parameter can be predetermined. As a result of the downmix, the downmix channel audio generator 220 can generate a downmix audio signal (i.e., the second multi-channel audio signal). In this specification, the downmix audio signal is also referred to as the second multi-channel audio signal, but can also be a single-channel signal including one channel.

[0134] Figure 2B is a diagram showing a scene classification result according to an embodiment of the present disclosure. Figure 1A , Figure 1B , Figure 1C and Figure 2A as well as Figure 2B The audio scene classifiers 100 and 210 can extract the first audio signal AS from the multi-channel audio signal MCAS t For example, the first audio signal AS t The first audio signal AS may include C / L / R / SL / SR channel signals of the multi-channel audio signal MCAS. t May correspond to the first frame.

[0135] The audio scene classifiers 100 and 210 can classify the first audio signal AS t For example, the first audio signal AS t The corresponding scene may be classified into one of a dialogue type, a sound effect type, and a music type. The classification accuracies of the audio scene classifiers 100 and 200 according to an embodiment of the present disclosure are summarized in Table 2 shown below.

[0136] Table 2

[0137]

[0138] Referring to Table 2, when the first audio signal AS t When the ground truth is a dialogue type, the audio scene classifiers 100 and 210 classify the first audio signal AS with an accuracy of 98.0%. t It is estimated to be a conversation type. When the first audio signal AS t When the basic fact is the sound effect type, the audio scene classifiers 100 and 210 classify the first audio signal AS with an accuracy of 85.1%. t It is estimated to be a sound effect type. When the first audio signal AS tWhen the basic fact is the music genre, the audio scene classifiers 100 and 210 classify the first audio signal AS with an accuracy of 89.9%. t The audio scene classifiers 100 and 210 have an average classification accuracy of 95.8%, and can classify scenes corresponding to specific frames of audio signals with improved accuracy.

[0139] Figure 3A is a diagram for describing a scalable channel layout structure according to an embodiment of the present disclosure.

[0140] The 3D audio decoding device of the related art receives compressed audio signals of independent channels of a specific channel layout from a bitstream. The 3D audio decoding device of the related art reconstructs audio signals of 3D audio channels that omnidirectionally surround a listener by using the compressed audio signals of the independent channels received from the bitstream. In this case, only the audio signals of the specific channel layout may be reconstructed.

[0141] Alternatively, the 3D audio decoding device of the related art receives compressed audio signals of independent channels (first independent channel group) of a specific channel layout from a bitstream. For example, the specific channel layout may be a 5.1 channel layout, and in this case, the compressed audio signals of the first independent channel group may be compressed audio signals of five surround channels and one subwoofer channel.

[0142] Herein, in order to increase the number of channels, the 3D audio decoding device of the related art also receives compressed audio signals of other channels (second independent channel group) independent of the first independent channel group. For example, the compressed audio signals of the second independent channel group can be compressed audio signals of two height channels.

[0143] That is, the 3D audio decoding device of the related art reconstructs the audio signal of the 3D audio channel surrounding the listener omnidirectionally by using the compressed audio signal of the second independent channel group received from the bitstream separated from the compressed audio signal of the first independent channel group received from the bitstream. Therefore, the audio signal with the increased number of channels is reconstructed. Here, the audio signal of the 3D audio channel surrounding the listener omnidirectionally may be an audio signal of 5.1.2 channels.

[0144] On the other hand, a conventional audio decoding apparatus supporting only reproduction of an audio signal of a stereo channel cannot correctly process a compressed audio signal included in a bitstream.

[0145] A 3D audio decoding device of the related art that supports the reproduction of a 3D audio signal first decompresses (decodes) the compressed audio signals of the first independent channel group and the second independent channel group to reproduce the audio signals of the stereo channels. Then, the 3D audio decoding device of the related art upmixes the audio signals generated by the decompression. However, in order to reproduce the audio signals of the stereo channels, an operation such as upmixing must be performed.

[0146] Therefore, a scalable channel layout structure capable of processing compressed audio signals in a conventional audio decoding device is needed. In addition, in the audio decoding device 500 and the audio decoding device 700 supporting the reproduction of 3D audio signals according to an embodiment of the present disclosure, a scalable channel layout structure capable of processing compressed audio signals according to the 3D audio channel layout supporting the reproduction is needed. Here, the scalable channel layout structure may mean a layout structure in which the number of channels can be freely increased from the basic channel layout.

[0147] The audio decoding device 500 and the audio decoding device 700 according to an embodiment of the present disclosure can reconstruct an audio signal of a scalable channel layout structure from a bitstream. Using the scalable channel layout structure according to an embodiment of the present disclosure, the number of channels can be increased from the stereo channel layout 300 to the 3D audio channel layout 310 in front of the listener. In addition, using the scalable channel layout structure, the number of channels can be increased from the 3D audio channel layout 310 in front of the listener to the 3D audio channel layout 320 positioned omnidirectionally around the listener (or the 3D audio channel layout 320 omnidirectionally around the listener). For example, the 3D audio channel layout 310 in front of the listener can be a 3.1.2 channel layout. The 3D audio channel layout 320 omnidirectionally surrounding the listener can be a 5.1.2 or 7.1.2 channel layout. However, the scalable channel layout that can be implemented in the present disclosure is not limited to this.

[0148] As a basic channel group, an audio signal of a stereo channel of the related art may be compressed. A conventional audio decoding device may decompress the compressed audio signal of the basic channel group from a bit stream, thereby smoothly reproducing the audio signal of the stereo channel of the related art.

[0149] In addition, as the slave channel group, audio signals of channels other than the audio signals of stereo channels of the related art in the multi-channel audio signal may be compressed.

[0150] However, in the process of increasing the number of channels, a part of the audio signal of the channel group may be an audio signal in which signals of some independent channels of the audio signal of a specific channel layout are mixed.

[0151] Therefore, in the audio decoding apparatus 500 and the audio decoding apparatus 700, a portion of the audio signal of the base channel group and a portion of the audio signal of the slave channel group may be demixed to generate an audio signal of an upmix channel included in a specific channel layout.

[0152] There may be one or more slave channel groups. For example, the audio signals of the channels other than the audio signals of the stereo channels in the audio signal of the 3D audio channel layout 310 in front of the listener may be compressed into the audio signals of the first slave channel group.

[0153] Audio signals of channels other than audio signals of channels reconstructed from the base channel group and the first slave channel group among audio signals of the 3D audio channel layout 320 omnidirectionally surrounding the listener may be compressed into audio signals of the second slave channel group.

[0154] The audio decoding apparatus 500 and the audio decoding apparatus 700 according to an embodiment of the present disclosure may support reproduction of an audio signal of a 3D audio channel layout 320 that omnidirectionally surrounds a listener.

[0155] Therefore, the audio decoding apparatus 500 and the audio decoding apparatus 700 according to an embodiment of the present disclosure may reconstruct an audio signal of the 3D audio channel layout 320 omnidirectionally surrounding a listener based on the audio signal of the basic channel group and the audio signals of the first and second slave channel groups.

[0156] The conventional audio signal processing apparatus may ignore the compressed audio signals of the subordinate channel group that cannot be reconstructed from the bitstream, and reproduce the audio signals of the stereo channels reconstructed from the bitstream.

[0157] Similarly, the audio decoding device 500 and the audio decoding device 700 can process the compressed audio signals of the basic channel group and the slave channel group to reconstruct the audio signals of the supportable channel layout in the scalable channel layout. The audio decoding device 500 and the audio decoding device 700 may not reconstruct the compressed audio signals about the unsupported higher channel layout from the bitstream. Therefore, the audio signals of the supportable channel layout can be reconstructed from the bitstream while ignoring the compressed audio signals associated with the higher channel layout that the audio decoding device 500 and the audio decoding device 700 do not support.

[0158] In particular, the encoding and decoding apparatus of the related art compresses and decompresses the audio signals of the independent channels of a specific channel layout. Therefore, compression and decompression of the audio signals of the limited channel layout are possible.

[0159] However, through the audio encoding device 400 and the audio encoding device 600 and the audio decoding device 500 and the audio decoding device 700 supporting a scalable channel layout according to an embodiment of the present disclosure, transmission and reconstruction of audio signals of stereo channels may be possible. Using the audio encoding device 400 and the audio encoding device 600 and the audio decoding device 500 and the audio decoding device 700 according to an embodiment of the present disclosure, transmission and reconstruction of audio signals of a 3D channel layout in front of a listener may be possible. Using the audio encoding device 400 and the audio encoding device 600 and the audio decoding device 500 and the audio decoding device 700 according to an embodiment of the present disclosure, transmission and reconstruction of audio signals of a 3D channel layout in front of a listener may be possible.

[0160] In other words, the audio encoding device 400 and the audio encoding device 600 and the audio decoding device 500 and the audio decoding device 700 can send and reconstruct audio signals according to the layout of the stereo channel. In addition, the audio encoding device 400 and the audio encoding device 600 and the audio decoding device 500 and the audio decoding device 700 can freely transform the audio signal of the current channel layout into the audio signal of the different channel layout. By mixing / demixing between the audio signals of the channels included in the different channel layouts, the conversion between the channel layouts may be possible. The audio encoding device 400 and the audio encoding device 600 and the audio decoding device 500 and the audio decoding device 700 according to the embodiment of the present disclosure can support the conversion between various channel layouts, thereby sending and reproducing audio signals of various 3D channel layouts. That is, between the channel layout in front of the listener and the channel layout that omnidirectionally surrounds the listener or between the stereo channel layout and the stereo front channel layout, channel dependency cannot be guaranteed, but free conversion by mixing / demixing of audio signals may be possible.

[0161] According to an embodiment of the present disclosure, the audio encoding device 400 and the audio encoding device 600 and the audio decoding device 500 and the audio decoding device 700 support processing of audio signals of a channel layout in front of a listener, and thereby send and reconstruct audio signals corresponding to speakers arranged around a screen, thereby improving the listener's sense of immersion.

[0162] Reference later Figure 4A , Figure 4B , Figure 4C , Figure 4D , Figure 5A , Figure 5B , Figure 5C , Figure 5D , Fig. 6A , Figure 6B , Fig. 7A and Figure 7BDetailed operations of the audio encoding apparatus 400 and the audio encoding apparatus 600 and the audio decoding apparatus 500 and the audio decoding apparatus 700 are described.

[0163] Figure 3B is a diagram for describing an example of a scalable audio channel layout structure.

[0164] refer to Figure 3B In order to transmit the audio signal of the stereo channel layout 360, the audio encoding apparatus 400 and the audio encoding apparatus 600 may generate a compressed audio signal (A / B signal) of a basic channel group by compressing the L2 / R2 signal.

[0165] In this case, the audio encoding apparatus 400 and the audio encoding apparatus 600 may generate an audio signal of a basic channel group by compressing an L2 / R2 signal.

[0166] In addition, in order to transmit the audio signal of the layout 370 of 3.1.2 channels as one of the 3D audio channels in front of the listener, the audio encoding device 400 and the audio encoding device 600 can generate a compressed audio signal of the slave channel group by compressing the C, LFE, Hfl3 and Hfr3 signals. The audio decoding device 500 and the audio decoding device 700 can reconstruct the L2 / R2 signal by decompressing the compressed audio signal of the basic channel group. The audio decoding device 500 and the audio decoding device 700 can reconstruct the C, LFE, Hfl3 and Hfr3 signals by decompressing the compressed audio signal of the slave channel group.

[0167] The audio decoding apparatus 500 and the audio decoding apparatus 700 can reconstruct the L3 signal of the 3.1.2 channel layout 370 by demixing the L2 signal and the C signal (1). The audio decoding apparatus 500 and the audio decoding apparatus 700 can reconstruct the R3 signal of the 3.1.2 channel layout 170 by demixing the R2 signal and the C signal (2).

[0168] As a result, the audio decoding device 500 and the audio decoding device 700 can output L3, R3, C, Lfe, Hfl3 and Hfr3 signals as audio signals of the 3.1.2 channel layout 370.

[0169] In order to transmit the audio signal of the 5.1.2 channel layout 380 omnidirectionally surrounding the listener, the audio encoding device 400 and the audio encoding device 600 may also compress the L5 and R5 signals to generate a compressed audio signal of the second slave channel group.

[0170] As described above, the audio decoding device 500 and the audio decoding device 700 can reconstruct the L2 / R2 signal by decompressing the compressed audio signal of the base channel group, and reconstruct the C, LFE, Hfl3 and Hfr3 signals by decompressing the compressed audio signal of the first slave channel group. In addition, the audio decoding device 500 and the audio decoding device 700 can reconstruct the L5 and R5 signals by decompressing the compressed audio signal of the second slave channel group. In addition, as described above, the audio decoding device 500 and the audio decoding device 700 can reconstruct the L3 and R3 signals by demixing some of the decompressed audio signals.

[0171] In addition, the audio decoding apparatus 500 and the audio decoding apparatus 700 may reconstruct the Ls5 signal by demixing the L3 and L5 signals (3). The audio decoding apparatus 500 and the audio decoding apparatus 700 may reconstruct the Rs5 signal by demixing the R3 and R5 signals (4).

[0172] The audio decoding device 500 and the audio decoding device 700 can reconstruct the Hl5 signal by demixing the Hfl3 and Ls5 signals (5).

[0173] The audio decoding apparatus 500 and the audio decoding apparatus 700 can reconstruct the Hr5 signal by demixing the Hfr3 and Rs5 signals (6). Hfr3 and Hr5 are right front channels among height channels.

[0174] As a result, the audio decoding device 500 and the audio decoding device 700 can output Hl5, Hr5, LFE, L, R, C, Ls5 and Rs5 signals as audio signals of the 5.1.2 channel layout 380.

[0175] In order to transmit the audio signal of the 7.1.4 channel layout 390, the audio encoding apparatus 400 and the audio encoding apparatus 600 may also compress the Hfl, Hfr, Ls, and Rs signals into the audio signal of the third slave channel group.

[0176] As described above, the audio decoding device 500 and the audio decoding device 700 can decompress the compressed audio signal of the basic channel group, the compressed audio signal of the first subordinate channel group and the compressed audio signal of the second subordinate channel group, and reconstruct the Hl5, Hr5, LFE, L, R, C, Ls5 and Rs5 signals by demixing (1), (2), (3), (4), (5) and (6).

[0177] In addition, the audio decoding device 500 and the audio decoding device 700 can reconstruct the Hfl, Hfr, Ls and Rs signals by decompressing the compressed audio signal of the third subordinate channel group. The audio decoding device 500 and the audio decoding device 700 can reconstruct the Lb signal (7) of the 7.1.4 channel layout 390 by demixing the Ls5 signal and the Ls signal.

[0178] The audio decoding apparatus 500 and the audio decoding apparatus 700 may reconstruct the Rb signal of the 7.1.4 channel layout 190 by demixing the Rs5 signal and the Rs signal (8).

[0179] The audio decoding device 500 and the audio decoding device 700 can reconstruct the Hbl signal (9) of the 7.1.4 channel layout 190 by demixing the Hfl signal and the Hl5 signal.

[0180] The audio decoding apparatus 500 and the audio decoding apparatus 700 may reconstruct the Hbr signal (10) of the 7.1.4 channel layout 190 by demixing the Hfr signal and the Hr5 signal.

[0181] As a result, the audio decoding device 500 and the audio decoding device 700 can output Hfl, Hfr, LFE, C, L, R, Ls, Rs, Lb, Rb, Hbl and Hbr signals as audio signals of the 7.1.4 channel layout 390.

[0182] Therefore, the audio decoding apparatus 500 and the audio decoding apparatus 700 can reconstruct an audio signal of a 3D audio channel in front of a listener and an audio signal of a 3D audio channel omnidirectionally surrounding the listener as well as an audio signal of a stereo channel layout of the related art by supporting a scalable channel layout in which the number of channels is increased through a demixing operation.

[0183] Reference above Figure 3B The scalable channel layout structure described in detail is merely an example, and the channel layout structure may be implemented to be scalable to include various channel layouts.

[0184] Figure 4A is a block diagram of an audio encoding device according to an embodiment of the present disclosure.

[0185] The audio encoding apparatus 400 may include a memory 410 and a processor 430. The audio encoding apparatus 400 may be implemented as a device capable of performing audio processing, such as a server, a TV, a camera, a cellular phone, a tablet personal computer (PC), a laptop computer, and the like.

[0186] Although in Figure 4AThe memory 410 and the processor 430 are separately shown in FIG. 4 , but the memory 410 and the processor 430 may be implemented by one hardware module (eg, a chip).

[0187] The processor 430 may be implemented as a dedicated processor for audio processing based on a neural network. Alternatively, the processor 430 may be implemented by a combination of a general-purpose processor (such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU)) and software. The dedicated processor may include a memory for implementing an embodiment of the present disclosure or a memory processor for using an external memory.

[0188] The processor 430 may include a plurality of processors. In this case, the processor 330 may be implemented as a combination of dedicated processors, or may be implemented by a combination of software and a plurality of general-purpose processors (such as an AP, a CPU, or a GPU).

[0189] The memory 410 may store one or more instructions for audio processing. In an embodiment of the present disclosure, the memory 410 may store a neural network. When the neural network is implemented in the form of a dedicated hardware chip for artificial intelligence or as part of an existing general-purpose processor (e.g., CPU or AP) or a graphics-specific processor (e.g., GPU), the neural network may not be stored in the memory 410. The neural network may be implemented by an external device (e.g., a server), and in this case, the audio encoding device 400 may request and receive result information based on the neural network from the external device.

[0190] The processor 430 may sequentially process the continuous frames according to the instructions stored in the memory 410 and obtain continuous encoded (compressed) frames. The continuous frames may refer to frames constituting the audio.

[0191] The processor 430 may perform an audio processing operation with the original audio signal as input and output a bitstream including a compressed audio signal. In this case, the original audio signal may be a multi-channel audio signal. The compressed audio signal may be a multi-channel audio signal whose number of channels is less than or equal to the number of channels of the original audio signal.

[0192] Processor 430 may execute Figure 1A , Figure 1B , Figure 1C , Figure 1D and Figure 2A The audio scene classifiers 100 and 210 and at least some functions of the multi-channel audio signal processor 200 in FIG.

[0193] In this case, the bit stream may include a basic channel group and further include n subordinate channel groups (n is an integer greater than or equal to 1). Therefore, the number of channels can be freely increased according to the number of subordinate channel groups.

[0194] Figure 4B is a block diagram of an audio encoding device according to an embodiment of the present disclosure.

[0195] refer to Figure 4B , the audio encoding apparatus 400 may include a multi-channel audio encoder 450, a bitstream generator 480, and an additional information generator 485. The multi-channel audio encoder 450 may include a multi-channel audio signal processor 460 and a compressor 470.

[0196] Return to reference Figure 4A As described above, the audio encoding device 400 may include a memory 410 and a processor 430, and is used to implement Figure 4B Instructions of the components of the multi-channel audio encoder 450, the multi-channel audio signal processor 460, the compressor 470, the bitstream generator 480, and the additional information generator 485 may be stored in Figure 4A The processor 430 may execute instructions stored in the memory 410. Figure 4B The elements such as the multi-channel audio encoder 450, the multi-channel audio signal processor 460, the compressor 470, the bitstream generator 480, and the additional information generator 485 shown in the figure can be obtained by classifying the operations performed by the processor 430 executing the program (or instruction) stored in the memory 410 according to the function. Figure 4B The operations performed by the illustrated elements such as the multi-channel audio encoder 450 , the multi-channel audio signal processor 460 , the compressor 470 , the bitstream generator 480 , and the additional information generator 485 may be considered to be actually performed by the processor 430 .

[0197] The multi-channel audio signal processor 460 may obtain at least one audio signal of a basic channel group and at least one audio signal of at least one subordinate channel group from the original audio signal. For example, when the original audio signal is an audio signal of a 7.1.4 channel layout, the multi-channel audio signal processor 460 may obtain an audio signal of 2 channels (stereo channels) as an audio signal of a basic channel group in the audio signal of the 7.1.4 channel layout.

[0198] The multi-channel audio signal processor 460 may obtain audio signals of channels other than the audio signals of the 2 channels from the audio signals of the 3.1.2 channel layout as audio signals of the first slave channel group to reconstruct the audio signals of the 3.1.2 channel layout as one of the 3D audio channels in front of the listener. In this case, the audio signals of some channels of the first slave channel group may be demixed to generate audio signals of demixed channels.

[0199] The multi-channel audio signal processor 460 may obtain an audio signal of a channel other than the audio signal of the basic channel group and the audio signal of the first subordinate channel group from the audio signal of the 5.1.2 channel layout as an audio signal of the second subordinate channel group to reconstruct the audio signal of the 5.1.2 channel layout as one of the 3D audio channels in front and behind the listener. In this case, the audio signals of some channels of the second subordinate channel group may be demixed to generate audio signals of demixed channels.

[0200] The multi-channel audio signal processor 460 can obtain the audio signals of the channels other than the audio signals of the first slave channel group and the audio signals of the second slave channel group from the audio signals of the 7.1.4 channel layout as the audio signals of the third slave channel group to reconstruct the audio signals of the 7.1.4 channel layout as one of the 3D audio channels surrounding the listener in all directions. Likewise, the audio signals of some channels of the third slave channel group can be demixed to obtain audio signals of demixed channels.

[0201] Will refer to it later Figure 4C The detailed operation of the multi-channel audio signal processor 460 is described.

[0202] The compressor 470 may compress the audio signal of the basic channel group and the audio signal of the subordinate channel group. That is, the compressor 470 may compress at least one audio signal of the basic channel group to obtain at least one compressed audio signal of the basic channel group. Herein, compression may mean compression based on various audio codecs. For example, compression may include transformation and quantization processes.

[0203] Herein, the audio signal of the basic channel group may be a single channel or a stereo signal. Alternatively, the audio signal of the basic channel group may include an audio signal of a first channel generated by mixing the audio signal L of the left stereo channel with C_1. Here, C_1 may be an audio signal of a center channel in front of the listener decompressed after compression. In the name of the audio signal ("X_Y"), "X" may represent the name of the channel, and "Y" may represent an error removal factor being decoded, being upmixed, being applied (i.e., being scaled), or an LFE gain being applied. For example, a decoded signal may be expressed as "X_1", and a signal (upmixed signal) generated by upmixing the decoded signal may be expressed as "X_2". Alternatively, a signal to which an LFE gain is applied to a decoded LFE signal may also be expressed as "X_2". A signal to which an error removal factor is applied to an upmixed signal (i.e., a scaled signal) may be expressed as "X_3".

[0204] The audio signal of the basic channel group may include an audio signal of a second channel generated by mixing the audio signal R of the right stereo channel with C_1.

[0205] The compressor 470 may obtain at least one compressed audio signal of at least one slave channel group by compressing at least one audio signal of at least one slave channel group.

[0206] The additional information generator 485 may generate additional information based on at least one of the original audio signal, the compressed audio signal of the base channel group, or the compressed audio signal of the slave channel group. In this case, the additional information may be information related to the multi-channel audio signal and include pieces of information for reconstructing the multi-channel audio signal.

[0207] For example, the additional information may include an audio object signal of a 3D audio channel in front of the listener, the audio object signal indicating at least one of the position, shape, area or direction of an audio signal, an audio object (sound source). Alternatively, the additional information may include information about the total number of audio streams including a basic channel audio stream and a slave channel audio stream. The additional information may include downmix gain information. The additional information may include channel mapping table information. The additional information may include volume information. The additional information may include LFE gain information. The additional information may include dynamic range control (DRC) information. The additional information may include channel layout rendering information. The additional information may also include information about the number of coupled audio streams, information indicating a multi-channel layout, information about whether there is a dialogue and a dialogue level in the audio signal, information indicating whether to output LFE, information about whether there is an audio object on the screen, information about the presence or absence of an audio signal of a continuous audio channel (or a scene-based audio signal or an ambisonic audio signal), and information about the presence or absence of an audio signal of a discrete audio channel (or an object-based audio signal or a spatial multi-channel audio signal). The additional information may include information about unmixing, the information including at least one unmixing weight parameter of an unmixing matrix for reconstructing the multi-channel audio signal. Unmixing and (down) mixing correspond to each other, so that the information about unmixing may correspond to the information about (down) mixing, and the information about unmixing may include information about (down) mixing. For example, the information about unmixing may include at least one (down) mixing weight parameter of a (down) mixing matrix. The unmixing weight parameter may be obtained based on the (down) mixing weight parameter.

[0208] The additional information may be various combinations of the above-mentioned multiple pieces of information. In other words, the additional information may include at least one of the above-mentioned multiple pieces of information.

[0209] When there is an audio signal of a slave channel corresponding to at least one audio signal of the basic channel group, the additional information generator 485 may generate slave channel audio signal identification information indicating that there is the audio signal of the slave channel.

[0210] The bitstream generator 480 may generate a bitstream including the compressed audio signal of the base channel group and the compressed audio signal of the dependent channel group. The bitstream generator 480 may generate a bitstream further including the additional information generated by the additional information generator 485.

[0211] More specifically, the bitstream generator 480 may generate a base channel audio stream and a slave channel audio stream. The base channel audio stream may include a compressed audio signal of a base channel group, and the slave channel audio stream may include a compressed audio signal of a slave channel group.

[0212] The bitstream generator 480 may generate a bitstream including a base channel audio stream and a plurality of subordinate channel audio streams. The plurality of subordinate channel audio streams may include n subordinate channel audio streams (n is an integer greater than 1). In this case, the base channel audio stream may include a single channel audio signal or a compressed audio signal of a stereo channel.

[0213] For example, in the channels of the first multi-channel layout reconstructed from the base channel audio stream and the first slave channel audio stream, the number of surround channels may be S n-1 , the number of subwoofer channels can be W n-1 , and the number of height channels can be H n-1 In the second multi-channel layout reconstructed from the base channel audio stream, the first slave channel audio stream, and the second slave channel audio stream, the number of surround channels may be S n , the number of subwoofer channels can be W n , and the number of height channels can be H n .

[0214] In this case, S n-1 Can be less than or equal to S n , W n-1 Can be less than or equal to W n , and H n-1 Can be less than or equal to H n In this paper, S n-1 Equal to S n , W n-1 Equal to W n And H n-1 Equal to H n situation.

[0215] That is, the number of surround channels of the second multi-channel layout needs to be greater than the number of surround channels of the first multi-channel layout. Alternatively, the number of subwoofer channels of the second multi-channel layout needs to be greater than the number of subwoofer channels of the first multi-channel layout. Alternatively, the number of height channels of the second multi-channel layout needs to be greater than the number of height channels of the first multi-channel layout.

[0216] In addition, the number of surround channels of the second multi-channel layout may not be less than the number of surround channels of the first multi-channel layout. Similarly, the number of subwoofer channels of the second multi-channel layout may not be less than the number of subwoofer channels of the first multi-channel layout. The number of height channels of the second multi-channel layout may not be less than the number of height channels of the first multi-channel layout.

[0217] In addition, there is no situation where the number of surround channels of the second multi-channel layout is equal to the number of surround channels of the first multi-channel layout, the number of subwoofer channels of the second multi-channel layout is equal to the number of subwoofer channels of the first multi-channel layout, and the number of height channels of the second multi-channel layout is equal to the number of height channels of the first multi-channel layout. That is, all channels of the second multi-channel layout may not be the same as all channels of the first multi-channel layout.

[0218] More specifically, for example, when the first multi-channel layout is a 5.1.2 channel layout, the second multi-channel layout may be a 7.1.4 channel layout.

[0219] In addition, the bitstream generator 480 may generate metadata including additional information.

[0220] As a result, the bitstream generator 480 may generate a bitstream including a base channel audio stream, a dependent channel audio stream, and metadata.

[0221] The bitstream generator 480 may generate a bitstream in a form in which the number of channels may be freely increased from a basic channel group.

[0222] That is, an audio signal of a basic channel group can be reconstructed from a basic channel audio stream, and a multi-channel audio signal in which the number of channels is increased from the basic channel group can be reconstructed from the basic channel audio stream and the dependent channel audio stream.

[0223] The bitstream generator 480 may generate a file stream having a plurality of audio tracks. The bitstream generator 480 may generate an audio stream of a first audio track including at least one compressed audio signal of a basic channel group. The bitstream generator 480 may generate an audio stream of a second audio track including identification information of a slave channel audio signal. In this case, the second audio track following the first audio track may be adjacent to the first audio track.

[0224] When there is a slave channel audio signal corresponding to at least one audio signal of a basic channel group, the bitstream generator 480 may generate an audio stream of a second audio track including at least one compressed audio signal of at least one slave channel group.

[0225] When there is no slave channel audio signal corresponding to at least one audio signal of the basic channel group, the bitstream generator 480 may generate an audio stream of a second track including a next audio signal of the basic channel group relative to the audio signal of the first track of the basic channel group.

[0226] Figure 4C is a block diagram of a multi-channel audio signal processor 460 according to an embodiment of the present disclosure.

[0227] refer to Figure 4C, the multi-channel audio signal processor 460 may include a channel layout identifier 461 , a down-mix channel audio generator 462 , and an audio signal classifier 466 .

[0228] The channel layout identifier 461 may identify at least one channel layout from the original audio signal. In this case, the at least one channel layout may include a plurality of hierarchical channel layouts. The channel layout identifier 461 may identify the channel layout of the original audio signal. The channel layout identifier 461 may identify a channel layout lower than the channel layout of the original audio signal. For example, when the original audio signal is an audio signal of a 7.1.4 channel layout, the channel layout identifier 461 may identify the 7.1.4 channel layout and identify a 5.1.2 channel layout, a 3.1.2 channel layout, a 2-channel layout, etc. lower than the 7.1.4 channel layout. A higher channel layout may mean a layout in which the number of at least one of the surround channels / subwoofer channels / height channels is greater than the number of the lower channel layout. Depending on whether the number of surround channels is large or small, a higher / lower channel layout may be determined, and for the same number of surround channels, a higher / lower channel layout may be determined depending on whether the number of subwoofer channels is large or small. For the same number of surround channels and subwoofer channels, a higher / lower channel layout may be determined depending on whether the number of height channels is large or small.

[0229] In addition, the identified channel layout may include a target channel layout. The target channel layout may mean the highest channel layout of the audio signal included in the final output bitstream. The target channel layout may be the channel layout of the original audio signal or a channel layout lower than the channel layout of the original audio signal.

[0230] More specifically, the channel layout identified from the original audio signal can be determined hierarchically according to the channel layout of the original audio signal. In this case, the channel layout identifier 461 can identify at least one of the predetermined channel layouts. For example, the channel layout identifier 461 can identify some of the predetermined channel layouts, 7.1.4 channel layout, 5.1.4 channel layout, 5.1.2 channel layout, 3.1.2 channel layout, and 2 channel layout from the layout of the original audio signal.

[0231] The channel layout identifier 461 may transmit a control signal to a downmix channel audio generator corresponding to at least one identified channel layout among the first downmix channel audio generator 463, the second downmix channel audio generator 464, ..., and the Nth downmix channel audio generator 465 based on the identified channel layout, and the downmix channel audio generator 462 may generate a downmix channel audio from the original audio signal based on the at least one channel layout identified in the channel layout identifier 461. The downmix channel audio generator 462 may generate the downmix channel audio from the original audio signal by using a downmix matrix including at least one downmix weight parameter.

[0232] For example, when the channel layout of the original audio signal is the nth channel layout in ascending order in the predetermined channel layout, the downmix channel audio generator 462 can generate a downmix channel audio of the (n-1)th channel layout immediately lower than the channel layout of the original audio signal from the original audio signal. By repeating this process, the downmix channel audio generator 462 can generate a downmix channel audio of a channel layout lower than the current channel layout.

[0233] For example, the downmix channel audio generator 462 may include a first downmix channel audio generator 463, a second downmix channel audio generator 464, and an (n-1)th downmix channel audio generator. (n-1) may be less than or equal to N.

[0234] In this case, the (n-1)th down-mix channel audio generator can generate an audio signal of the (n-1)th channel layout from the original audio signal. In addition, the (n-2)th down-mix channel audio generator can generate an audio signal of the (n-2)th channel layout from the original audio signal. In this way, the first down-mix channel audio generator 463 can generate an audio signal of the first channel layout from the original audio signal. In this case, the audio signal of the first channel layout can be an audio signal of a basic channel group.

[0235] Each of the first down-mix channel audio generator 463, the second down-mix channel audio generator 464 and the (n-1) down-mix channel audio generator 465 can be connected in a cascade manner. That is, down-mix channel audio generators 463, 464 and 465 can be connected so that the output of the higher down-mix channel audio generator becomes the input of the lower down-mix channel audio generator. For example, the audio signal of the (n-1) channel layout can be output from the (n-1) down-mix channel audio generator with the original audio signal as input, and the audio signal of the (n-1) channel layout can be input to the (n-2) down-mix channel audio generator, and the (n-2) down-mix channel audio can be generated from the (n-2) down-mix channel audio generator. In this way, the first down-mix channel audio generator 463, the second down-mix channel audio generator 464 and the (n-1) down-mix channel audio generator 465 can be connected to output the audio signal of each channel layout.

[0236] The audio signal classifier 466 may obtain an audio signal of a basic channel group and an audio signal of a subordinate channel group based on an audio signal of at least one channel layout. In this case, the audio signal classifier 466 may mix an audio signal of at least one channel included in the audio signal of at least one channel layout through the mixer 467. The audio signal classifier 466 may classify the mixed audio signal into at least one of an audio signal of a basic channel group or an audio signal of a subordinate channel group.

[0237] Figure 4D is a diagram for describing a detailed operation of the audio signal classifier 466.

[0238] refer to Figure 4D , Figure 4C The downmix channel audio generator 462 can obtain an audio signal of a 5.1.2 channel layout 491, an audio signal of a 3.1.2 channel layout 492, an audio signal of a 2-channel layout 493, and an audio signal of a single channel layout 494 as audio signals of lower channel layouts from an original audio signal of a 7.1.4 channel layout 490. A first downmix channel audio generator 463, a second downmix channel audio generator 464, and an (n-1)th downmix channel audio generator 465 of the downmix channel audio generator 462 are connected in a cascade manner so that audio signals can be sequentially obtained from a current channel layout to a lower channel layout.

[0239] Figure 4C The audio signal classifier 466 may classify the audio signal of the single channel layout 494 into audio signals of a basic channel group 495 .

[0240] The audio signal classifier 466 may classify the audio signal of the L2 channel as a part of the audio signal of the 2-channel layout 493 as an audio signal of the subordinate channel group #1 496. The audio signal of the L2 channel and the audio signal of the R2 channel are mixed to generate the audio signal of the single channel layout 494, so that conversely, the audio decoding device 500 and the audio decoding device 700 may demix the audio signal of the single channel layout 494 and the audio signal of the L2 channel to reconstruct the audio signal of the R2 channel. Therefore, the audio signal of the R2 channel may not be classified as an audio signal of a separate channel group.

[0241] The audio signal classifier 466 may classify the audio signal of the Hfl3 channel, the audio signal of the C channel, the audio signal of the LFE channel, and the audio signal of the Hfr3 channel among the audio signals of the 3.1.2 channel layout 492 as the audio signal of the slave channel group #2 497. The audio signal of the L2 channel is generated by mixing the audio signal of the L3 channel and the audio signal of the Hfl3 channel, so that conversely, the audio decoding apparatus 500 and the audio decoding apparatus 700 may reconstruct the audio signal of the L2 channel of the slave channel group #1 496 and the audio signal of the Hfl3 channel of the slave channel group #2 497.

[0242] Therefore, the audio signal of the L3 channel among the audio signals of the 3.1.2 channel layout 492 may not be classified as an audio signal of a specific channel group.

[0243] For the same reason, the R3 channel may not be classified as an audio signal of a specific channel group.

[0244] The audio signal classifier 466 may transmit the audio signal of the L channel and the audio signal of the R channel as the audio signal of some channels of the 5.1.2 channel layout 491 as the audio signal of the subordinate channel group #3 498 so as to transmit the audio signal of the 5.1.2 channel layout 491. The audio signal of one of the Ls5, Hl5, Rs5, and Hr5 channels may be one of the audio signals of the 5.1.2 channel layout 491, but may not be classified as an audio signal of a separate subordinate channel group. This is because the signals of the Ls5, Hl5, Rs5, and Hr5 channels may not be channel audio signals in front of the listener, and may be signals in which the audio signals of at least one of the audio channels in front of, beside, and behind the listener in the audio signal of the 7.1.4 channel layout 490 may be mixed. By compressing the audio signal of the audio channel in front of the listener in the original audio signal, rather than classifying the mixed signal as an audio signal of the subordinate channel group and compressing it, the sound quality of the audio signal of the audio channel in front of the listener may be improved. As a result, the listener may perceive that the sound quality of the reproduced audio signal is improved.

[0245] However, depending on circumstances, Ls5 or Hl5 instead of L may be classified as the audio signal of the slave channel group #3 498, and Rs5 or Hr5 instead of R may be classified as the audio signal of the slave channel group #3 498.

[0246] The audio signal classifier 466 may classify an audio signal of an Ls, Hfl, Rs, or Hfr channel in an audio signal of a 7.1.4 channel layout 490 as an audio signal of a subordinate channel group #4 499. In this case, Lb instead of Ls, Hbl instead of Hfl, Rb instead of Rs, and Hbr instead of Hfr may not be classified as an audio signal of a subordinate channel group #4 499. By compressing an audio signal of a side audio channel near the front of a listener, rather than classifying an audio signal of an audio channel behind a listener in an audio signal of a 7.1.4 channel layout 490 as an audio signal of a channel group and compressing it, the sound quality of the audio signal of the side audio channel near the front of a listener may be improved. Therefore, the listener may feel that the sound quality of the reproduced audio signal is improved. However, depending on the situation, Lb instead of Ls, Hbl instead of Hfl, Rb instead of Rs, and Hbr instead of Hfr may be classified as an audio signal of a subordinate channel group #4 499.

[0247] result, Figure 4C The downmix channel audio generator 462 may generate a plurality of lower layout audio signals (downmix channel audios) based on a plurality of lower channel layouts identified from the original audio signal layout. Figure 4C The audio signal classifier 466 can classify the audio signals of the basic channel group 495 and the audio signals of the subordinate channel group #1 496, the subordinate channel group #2 497, the subordinate channel group #3 498, and the subordinate channel group #4 499. The classified audio signals of the channels can classify a part of the audio signal of the independent channel in the audio signal of each channel as the audio signal of the channel group according to each channel layout. The audio decoding device 500 and the audio decoding device 700 can reconstruct the audio signal not classified by the audio signal classifier 466 by demixing. When the audio signal of the left channel relative to the listener is classified as the audio signal of a specific channel group, the audio signal of the right channel corresponding to the left channel can be classified as the audio signal of the corresponding channel group. That is, the audio signal of the coupled channel can be classified as the audio signal of one channel group.

[0248] When the audio signals of the stereo channel layout are classified as audio signals of the basic channel group 495, the audio signals of the coupled channels may all be classified as audio signals of one channel group. Figure 4DAs described above, when the audio signal of the mono channel layout is classified as the audio signal of the basic channel group 495, one of the audio signals of the stereo channel may be classified as the audio signal of the subordinate channel group #1 496, as an exception. However, the method of classifying the audio signals of the channel group may be various, without being limited to the reference Figure 4D That is, when classified audio signals of a channel group are demixed and audio signals of channels not classified as audio signals of the channel group can be reconstructed from the demixed audio signals, the audio signals of the channel group can be classified in various forms.

[0249] Figure 5A is a block diagram of a multi-channel audio decoding device 500 according to an embodiment of the present disclosure.

[0250] The audio decoding apparatus 500 may include a memory 510 and a processor 530. The audio decoding apparatus 500 may be implemented as a device capable of audio processing, such as a server, a TV, a camera, a mobile phone, a tablet PC, a laptop computer, and the like.

[0251] Although in Figure 5A The memory 510 and the processor 530 are separately shown in FIG. 5 , but the memory 510 and the processor 530 may be implemented by one hardware module (eg, a chip).

[0252] The processor 530 may be implemented as a dedicated processor for audio processing based on a neural network. Alternatively, the processor 530 may be implemented by a combination of a general-purpose processor (such as an AP, a CPU, or a GPU) and software. The dedicated processor may include a memory for implementing an embodiment of the present disclosure or a memory processor for using an external memory.

[0253] The processor 530 may include a plurality of processors. In this case, the processor 530 may be implemented as a combination of dedicated processors, or may be implemented by a combination of software and a plurality of general-purpose processors (such as an AP, a CPU, or a GPU).

[0254] The memory 510 may store one or more instructions for audio processing. In an embodiment of the present disclosure, the memory 510 may store a neural network. When the neural network is implemented in the form of a dedicated hardware chip for artificial intelligence or as part of an existing general-purpose processor (e.g., CPU or AP) or a graphics-specific processor (e.g., GPU), the neural network may not be stored in the memory 510. The neural network may be implemented by an external device (e.g., a server), and in this case, the audio decoding device 500 may request and receive result information based on the neural network from the external device.

[0255] The processor 530 may sequentially process the consecutive frames according to the instructions stored in the memory 510 to obtain consecutive reconstructed frames. The consecutive frames may refer to frames constituting the audio.

[0256] Processor 530 can output multi-channel audio signal by performing audio processing operation to input bit stream.Bit stream can be realized in scalable form to increase the number of channels from basic channel group.For example, processor 530 can obtain compressed audio signal of basic channel group from bit stream, and can reconstruct audio signal (for example, stereo channel audio signal) of basic channel group by decompressing compressed audio signal of basic channel group.In addition, processor 530 can reconstruct audio signal of subordinate channel group by decompressing compressed audio signal of subordinate channel group from bit stream.Processor 530 can reconstruct multi-channel audio signal based on audio signal of basic channel group and audio signal of subordinate channel group.

[0257] The processor 530 may reconstruct the audio signal of the first subordinate channel group by decompressing the compressed audio signal of the first subordinate channel group from the bitstream.The processor 530 may reconstruct the audio signal of the second subordinate channel group by decompressing the compressed audio signal of the second subordinate channel group.

[0258] The processor 530 may reconstruct a multi-channel audio signal with an increased number of channels based on the audio signal of the basic channel group and the corresponding audio signals of the first and second subordinate channel groups. Similarly, the processor 330 may decompress the compressed audio signals of n subordinate channel groups (where n is an integer greater than 2), and may reconstruct a multi-channel audio signal with a further increased number of channels based on the audio signal of the basic channel group and the corresponding audio signals of the basic channel group and the n subordinate channel groups.

[0259] Figure 5B is a block diagram of a multi-channel audio decoding device 500 according to an embodiment of the present disclosure.

[0260] refer to Figure 5B , the audio decoding apparatus 500 may include an information obtainer 550 and a multi-channel audio decoder 560. The multi-channel audio decoder 560 may include a decompressor 570 and a multi-channel audio signal reconstructor 580.

[0261] The audio decoding device 500 may include Figure 5A The memory 510 and the processor 530 are used to implement Figure 5BInstructions of components of the multi-channel audio signal reconstructor 580 such as the information obtainer 550, the multi-channel audio decoder 560, the decompressor 570, and the multi-channel audio signal reconstructor 580 may be stored in the memory 510. The processor 530 may execute the instructions stored in the memory 510. The operations performed by the processor 530 executing the program (or instructions) stored in the memory 510 may be obtained by classifying the operations according to the functions. Figure 5B The components shown in FIG. 5 are information obtainer 550, multi-channel audio decoder 560, decompressor 570, and multi-channel audio signal reconstructor 580. Figure 5B The operations performed by the illustrated elements such as the information obtainer 550 , the multi-channel audio decoder 560 , the decompressor 570 , and the multi-channel audio signal reconstructor 580 may be considered to be actually performed by the processor 530 .

[0262] The information obtainer 550 may obtain a compressed audio signal of a basic channel group from the bitstream. That is, the information obtainer 550 may classify a basic channel audio stream including at least one compressed audio signal of a basic channel group from the bitstream.

[0263] The information obtainer 550 may also obtain at least one compressed audio signal of at least one subordinate channel group from the bitstream. That is, the information obtainer 550 may classify at least one subordinate channel audio stream including at least one compressed audio signal of the subordinate channel group from the bitstream.

[0264] The bitstream may include a primary channel audio stream and a plurality of subordinate channel audio streams. The plurality of subordinate channel audio streams may include a first subordinate channel audio stream and a second subordinate channel audio stream.

[0265] In this case, limitation of channels of a multi-channel first audio signal reconstructed by a base channel audio stream and a first slave channel audio stream and a multi-channel second audio signal reconstructed by a base channel audio stream, a first slave channel audio stream, and a second slave channel audio stream will be described.

[0266] For example, in the channels of the first multi-channel layout reconstructed from the base channel audio stream and the first slave channel audio stream, the number of surround channels may be S n-1 , the number of subwoofer channels can be W n-1 , and the number of height channels can be H n-1 In the second multi-channel layout reconstructed from the base channel audio stream, the first slave channel audio stream, and the second slave channel audio stream, the number of surround channels may be S n , the number of subwoofer channels can be W n , and the number of height channels can be H n In this case, S n-1Can be less than or equal to S n , W n-1 Can be less than or equal to W n , and H n-1 Can be less than or equal to H n In this paper, S n-1 Equal to S n , W n-1 Equal to W n And H n-1 Equal to H n situation.

[0267] That is, the number of surround channels of the second multi-channel layout needs to be greater than the number of surround channels of the first multi-channel layout. Alternatively, the number of sub-bass channels of the second multi-channel layout needs to be greater than the number of sub-bass channels of the first multi-channel layout. Alternatively, the number of height channels of the second multi-channel layout needs to be greater than the number of height channels of the first multi-channel layout.

[0268] In addition, the number of surround channels of the second multi-channel layout may not be less than the number of surround channels of the first multi-channel layout. Similarly, the number of subwoofer channels of the second multi-channel layout may not be less than the number of subwoofer channels of the first multi-channel layout. The number of height channels of the second multi-channel layout may not be less than the number of height channels of the first multi-channel layout.

[0269] In addition, there is no situation where the number of surround channels of the second multi-channel layout is equal to the number of surround channels of the first multi-channel layout, the number of subwoofer channels of the second multi-channel layout is equal to the number of subwoofer channels of the first multi-channel layout, and the number of height channels of the second multi-channel layout is equal to the number of height channels of the first multi-channel layout. That is, all channels of the second multi-channel layout may not be the same as all channels of the first multi-channel layout.

[0270] More specifically, for example, when the first multi-channel layout is a 5.1.2 channel layout, the second multi-channel layout may be a 7.1.4 channel layout.

[0271] The bitstream may include a file stream having a plurality of audio tracks including a first audio track and a second audio track. A process in which the information obtainer 550 obtains at least one compressed audio signal of at least one subordinate channel group according to additional information included in the audio track will be described below.

[0272] The information obtainer 550 may obtain at least one compressed audio signal of a basic channel group from the first audio track.

[0273] The information obtainer 550 may obtain the slave channel audio signal identification information from a second audio track adjacent to the first audio track.

[0274] When the slave channel audio signal identification information indicates that the slave channel audio signal exists in the second audio track, the information obtainer 550 may obtain at least one audio signal of at least one slave channel group from the second audio track.

[0275] When the slave channel audio signal identification information indicates that the slave channel audio signal does not exist in the second audio track, the information obtainer 550 may obtain the next audio signal of the basic channel group from the second audio track.

[0276] The information obtainer 550 may obtain additional information related to reconstruction of the multi-channel audio from the bitstream. That is, the information obtainer 550 may classify metadata including the additional information from the bitstream and obtain the additional information from the classified metadata.

[0277] The decompressor 570 may reconstruct an audio signal of a basic channel group by decompressing at least one compressed audio signal of the basic channel group.

[0278] The decompressor 570 may reconstruct at least one audio signal of at least one slave channel group by decompressing at least one compressed audio signal of at least one slave channel group.

[0279] In this case, the decompressor 570 may include additional first, ..., nth decompressors for decoding the decompressed audio signal of each channel group (n channel groups). In this case, the first, ..., nth decompressors may operate in parallel with each other.

[0280] The multi-channel audio signal reconstructor 580 may reconstruct the multi-channel audio signal based on at least one audio signal of the basic channel group and at least one audio signal of the at least one slave channel group.

[0281] For example, when the audio signal of the basic channel group is an audio signal of a stereo channel, the multi-channel audio signal reconstructor 580 may reconstruct an audio signal of a 3D audio channel in front of the listener based on the audio signal of the basic channel group and the audio signal of the first subordinate channel group. For example, the 3D audio channel in front of the listener may be a 3.1.2 channel.

[0282] Alternatively, the multi-channel audio signal reconstructor 580 may reconstruct the audio signals of the audio channels omnidirectionally surrounding the listener based on the audio signals of the basic channel group, the audio signals of the first slave channel group, and the audio signals of the second slave channel group. For example, the 3D audio channels omnidirectionally surrounding the listener may be 5.1.2 channels or 7.1.4 channels.

[0283] The multi-channel audio signal reconstructor 580 may reconstruct the multi-channel audio signal based not only on the audio signal of the basic channel group and the audio signal of the subordinate channel group but also on the additional information. In this case, the additional information may be additional information for reconstructing the multi-channel audio signal. The multi-channel audio signal reconstructor 580 may output at least one reconstructed multi-channel audio signal.

[0284] The multi-channel audio signal reconstructor 580 according to an embodiment of the present disclosure may generate a first audio signal of a 3D audio channel in front of a listener from at least one audio signal of a basic channel group and at least one audio signal of at least one subordinate channel group. The multi-channel audio signal reconstructor 580 may reconstruct a multi-channel audio signal including a second audio signal of a 3D audio channel in front of a listener based on the first audio signal and an audio object signal of the 3D audio channel in front of the listener. In this case, the audio object signal may indicate at least one of a shape, region, position, or direction of an audio signal, an audio object (sound source), and may be obtained from the information obtainer 550.

[0285] Will refer to it later Figure 5C The detailed operation of the multi-channel audio signal reconstructor 580 is described.

[0286] Figure 5C is a block diagram of a multi-channel audio signal reconstructor 580 according to an embodiment of the present disclosure.

[0287] refer to Figure 5C , the multi-channel audio signal reconstructor 580 may include an up-mix channel group audio generator 581 and a renderer 586 .

[0288] The upmix channel group audio generator 581 may generate an audio signal of the upmix channel group based on the audio signal of the base channel group and the audio signal of the slave channel group. In this case, the audio signal of the upmix channel group may be a multi-channel audio signal. In this case, in addition, further based on additional information (e.g., information about dynamic demixing weight parameters), a multi-channel audio signal may be generated.

[0289] The upmix channel group audio generator 581 can generate an audio signal of an upmix channel by demixing the audio signal of the basic channel group and some audio signals of the slave channel group. For example, by demixing the audio signals L and R of the basic channel group and a portion C of the audio signal of the slave channel group, audio signals L3 and R3 of the demix channel (or upmix channel) can be generated.

[0290] The upmix channel group audio generator 581 may generate audio signals of some channels in the multi-channel audio signal by bypassing the demixing operation on some audio signals of the slave channel group. For example, the upmix channel group audio generator 581 may generate audio signals of C, LFE, Hfl3, and Hfr3 channels of the multi-channel audio signal by bypassing the demixing operation on audio signals of C, LFE, Hfl3, and Hfr3 channels as some audio signals of the slave channel group.

[0291] As a result, the upmix channel group audio generator 581 can generate an audio signal of the upmix channel group based on the audio signal of the upmix channel generated by demixing and the audio signal of the slave channel group bypassing the demixing operation. For example, the upmix channel group audio generator 581 can generate audio signals of L3, R3, C, LFE, Hfl3, and Hfr3 channels as audio signals of 3.1.2 channels based on the audio signals of L3 and R3 channels as the audio signals of the demix channel and the audio signals of C, LFE, Hfl3, and Hfr3 channels as the audio signals of the slave channel group.

[0292] Will refer to it later Figure 5D The detailed operation of the upmix channel group audio generator 581 is described.

[0293] The renderer 586 may include a volume controller 588 and a limiter 589. The multi-channel audio signal input to the renderer 586 may be a multi-channel audio signal of at least one channel layout. The multi-channel audio signal input to the renderer 586 may be a pulse code modulation (PCM) signal.

[0294] The volume (loudness) of the audio signal of each channel may be measured based on ITU-R BS.1770, and the volume (loudness) may be signaled through additional information of a bitstream.

[0295] The volume controller 588 may control the volume of the audio signal of each channel to a target volume (eg, ˜24 LKFS) based on volume information signaled through a bitstream.

[0296] The true peak value can be measured based on ITU-R BS.1770.

[0297] The limiter 589 may limit the true peak level of the audio signal after volume control (eg, to -1 dBTP).

[0298] Although the post-processing components included in the renderer 586, such as the volume controller 588 and the limiter 589, have been described so far, at least one component may be omitted and the order of each component may be changed according to circumstances, but is not limited thereto.

[0299] The multi-channel audio signal output unit 590 can output at least one multi-channel audio signal after post-processing. For example, the multi-channel audio signal output unit 590 can output the audio signal of each channel in the multi-channel audio signal to the audio output device corresponding to each channel according to the target channel layout, taking the post-processed multi-channel audio signal as input. The audio output device may include various types of speakers.

[0300] Figure 5D is a block diagram of an upmix channel audio generator 581 according to an embodiment of the present disclosure.

[0301] refer to Figure 5D , the upmix channel group audio generator 581 may include a demixer 582. The demixer 582 may include a first demixer 583 and second demixers 584 to 585.

[0302] The demixer 582 can obtain an audio signal of a new channel (upmixed channel or demixed channel) from the audio signals of some channels (decoded channels) in the audio signals of the basic channel group and the audio signals of the subordinate channel group. That is, the demixer 582 can obtain an audio signal of an upmixed channel from mixing at least one audio signal of several channels. The demixer 582 can output an audio signal of a specific layout including the audio signals of the upmixed channel and the audio signals of the decoded channel.

[0303] For example, the demixing operation may be bypassed in the demixer 582 so that the audio signals of the basic channel group may be output as the audio signals of the first channel layout.

[0304] The first demixer 583 can demix the audio signals of some channels with the audio signals of the basic channel group and the audio signals of the first subordinate channel group as input. In this case, the audio signals of the demixed channels (or upmixed channels) can be generated. The first demixer 583 can generate the audio signals of the independent channels by bypassing the mixing operation of the audio signals of other channels. The first demixer 583 can output the audio signals of the second channel layout, which are signals including the audio signals of the upmixed channels and the audio signals of the independent channels.

[0305] The second demixer 584 can generate an audio signal of a demixed channel (or upmixed channel) by demixing the audio signals of some channels in the audio signal of the second channel layout and the audio signal of the second slave channel. The second demixer 584 can generate an audio signal of an independent channel by bypassing the mixing operation relative to the audio signal of other channels. The second demixer 584 can output an audio signal of a third channel layout, which includes the audio signal of the upmixed channel and the audio signal of the independent channel.

[0306] Similar to the operation of the second demixer 584, the nth demixer may output an audio signal of an nth channel layout based on an audio signal of an (n-1)th channel layout and an audio signal of an (n-1)th slave channel group.

[0307] The Nth demixer 585 may output an audio signal of an Nth channel layout based on the audio signal of the (N-1)th channel layout and the audio signal of the (N-1)th slave channel group.

[0308] Although the audio signals of the lower channel layout are shown to be directly input to the corresponding demixers (such as demixer 583, demixer 584 and up to the Nth demixer 585), Figure 5C The audio signal of the channel layout output by the renderer 586 of the Nth demixer 584 can be input to each demixer (such as the demixer 583, the demixer 584 and up to the Nth demixer 585). That is, the post-processed audio signal of the lower channel layout can be input to each of the demixers, such as the demixer 583, the demixer 584 and the Nth demixer 585.

[0309] refer to Figure 5D , describing that the demixer 583, the demixer 584 and the Nth demixer 585 can be connected in a cascade manner to output the audio signal of each channel layout.

[0310] However, without connecting the demixer 583, the demixer 584, and the Nth demixer 585 in cascade, an audio signal of a specific layout may be output from an audio signal of a basic channel group and an audio signal of at least one subordinate channel group.

[0311] The audio signal generated by mixing signals of several channels in the audio encoding apparatus 400 and the audio encoding apparatus 600 may have a reduced level by using a downmix gain for preventing clipping. The audio decoding apparatus 500 and the audio decoding apparatus 700 may match the level of the audio signal to the level of the original audio signal based on the corresponding downmix gain of the signal generated by mixing.

[0312] The operation based on the above-mentioned downmix gain can be performed for each channel or channel group. The audio encoding device 400 and the audio encoding device 600 can signal the information about the downmix gain through the additional information of the bitstream for each channel or each channel group. Therefore, the audio decoding device 500 and the audio decoding device 700 can obtain the information about the downmix gain from the additional information of the bitstream for each channel or each channel group, and perform the above-mentioned operation based on the downmix gain.

[0313] The demixer 582 may perform a demixing operation based on the dynamic demixing weight parameters of the demixing matrix (corresponding to the downmixing weight parameters of the downmixing matrix). In this case, the audio encoding device 400 and the audio encoding device 600 may signal the dynamic demixing weight parameters or the dynamic downmixing weight parameters corresponding thereto through the additional information of the bitstream. Some demixing weight parameters may not be signaled and have fixed values.

[0314] Therefore, the audio decoding device 500 and the audio decoding device 700 can obtain information about dynamic demixing weight parameters (or information about dynamic downmixing weight parameters) from the additional information of the bitstream, and perform a demixing operation based on the obtained information about the dynamic demixing weight parameters (or information about dynamic downmixing weight parameters).

[0315] Fig. 6A is a block diagram of an audio encoding device 600 according to an embodiment of the present disclosure.

[0316] refer to Fig. 6A , the audio encoding apparatus 600 may include a multi-channel audio encoder 650, a bitstream generator 680, and an error removal related information generator 690. The multi-channel audio encoder 650 may include a multi-channel audio signal processor 660 and a compressor 670.

[0317] Reference again Figure 4A , Fig. 6A Each of the multi-channel audio encoder 650, the multi-channel audio signal processor 660, the compressor 670, the bitstream generator 680, and the error removal related information generator 690 may be composed of Figure 4A As described above, the audio encoding device 400 may include a memory 410 and a processor 430, and is used to implement Fig. 6A The instructions of the multi-channel audio encoder 650, the multi-channel audio signal processor 660, the compressor 670, the bit stream generator 680 and the error removal related information generator 690 may be stored in Figure 4AThe processor 430 may execute the instructions stored in the memory 410. The operations performed by the processor 430 executing the program (or instruction) stored in the memory 410 may be obtained by classifying the operations performed by the processor 430 according to the functions. Fig. 6A The multi-channel audio encoder 650, the multi-channel audio signal processor 660, the compressor 670, the bit stream generator 680 and the error removal related information generator 690 are shown. Fig. 6A The operations performed by the multi-channel audio encoder 650 , the multi-channel audio signal processor 660 , the compressor 670 , the bitstream generator 680 , and the error removal related information generator 690 shown in FIG. 6 may be considered to be actually performed by the processor 430 .

[0318] Fig. 6A The operations of the multi-channel audio encoder 650, the multi-channel audio signal processor 660, the compressor 670, and the bitstream generator 680 correspond to the operations of the multi-channel audio encoder 450, the multi-channel audio signal processor 460, the compressor 470, and the bitstream generator 480, respectively, and thus will be used Figure 4B replaces its detailed description.

[0319] The error removal related information generator 690 may include Figure 4B The additional information generator 485 may exist alone, but is not limited to this.

[0320] The error removal related information generator 690 may determine an error removal factor (e.g., a scaling factor) based on the first power value and the second power value. In this case, the first power value may be an energy value of one channel of the original audio signal or an energy value of an audio signal of one channel obtained by down-mixing the original audio signal. The second power value may be a power value of an audio signal of an upmixed channel as one of the audio signals of the upmixed channel group. The audio signal of the upmixed channel group may be an audio signal obtained by demixing the base channel reconstruction signal and the slave channel reconstruction signal.

[0321] The error removal related information generator 690 may determine an error removal factor for each channel.

[0322] The error removal related information generator 690 may generate information related to error removal (or error removal related information), including information about the determined error removal factor. The bitstream generator 680 may generate a bitstream further including the error removal related information. Figure 6B The detailed operation of the error removal related information generator 690 is described.

[0323] Figure 6Bis a block diagram of a reconstructor according to an embodiment of the present disclosure.

[0324] refer to Figure 6B The error removal related information generator 690 may include a decompressor 692 , a demixer 694 , a root mean square (RMS) value determination unit 696 , and an error removal factor determination unit 698 .

[0325] The decompressor 692 may generate a base channel reconstruction signal by decompressing the compressed audio signal of the base channel group. In addition, the decompressor 692 may generate a slave channel reconstruction signal by decompressing the compressed audio signal of the slave channel group.

[0326] The demixer 694 may demix the base channel reconstruction signal and the slave channel reconstruction signal to generate an audio signal of the upmixed channel group. More specifically, the demixer 694 may generate an audio signal of the upmixed channel (or demixed channel) by demixing the audio signals of some channels in the audio signals of the base channel group and the slave channel group. The demixer 694 may bypass the demixing operation on some audio signals in the audio signals of the base channel group and the slave channel group.

[0327] The demixer 694 may obtain an audio signal of an upmixed channel group including an audio signal of an upmixed channel and an audio signal bypassing a demixing operation.

[0328] The root mean square (RMS) value determination unit 696 may determine an RMS value of a first audio signal of one upmix channel in the upmix channel group. The RMS value determination unit 696 may determine an RMS value of a second audio signal of one channel of the original audio signal or an RMS value of a second audio signal of one channel of the audio signal downmixed from the original audio signal. In this case, the channel of the first audio signal and the channel of the second audio signal may indicate the same channel in the channel layout.

[0329] The error removal factor determination unit 698 may determine the error removal factor based on the RMS value of the first audio signal and the RMS value of the second audio signal. For example, a value generated by dividing the RMS value of the first audio signal by the RMS value of the second audio signal may be obtained as the value of the error removal factor. The error removal factor determination unit 698 may generate information about the determined error removal factor. The error removal factor determination unit 698 may output error removal related information including information about the error removal factor.

[0330] Fig. 7A is a block diagram of an audio decoding device according to an embodiment of the present disclosure.

[0331] refer to Fig. 7A, the audio decoding device 700 may include an information obtainer 750, a multi-channel audio decoder 760, a decompressor 770, a multi-channel audio signal reconstructor 780 and an error removal related information obtainer 755.

[0332] Reference again Figure 5A , Fig. 7A Each of the information obtainer 750, the error removal related information obtainer 755, the multi-channel audio decoder 760, the decompressor 770 and the multi-channel audio signal reconstructor 780 can be composed of Figure 5A As described above, the audio decoding device 500 may include a memory 510 and a processor 530, and is used to implement Figure 7B The instructions of the information obtainer 750, the error removal related information obtainer 755, the multi-channel audio decoder 760, the decompressor 770 and the multi-channel audio signal reconstructor 780 may be stored in Figure 5A The processor 530 may execute instructions stored in the memory 510. Fig. 7A The information obtainer 750, the error removal related information obtainer 755, the multi-channel audio decoder 760, the decompressor 770, and the multi-channel audio signal reconstructor 780 shown can be obtained by classifying the operations performed by the processor 530 executing the program (or instruction) stored in the memory 510 according to the function. Therefore, the following description is obtained by Fig. 7A The operations performed by the information obtainer 750 , the error removal related information obtainer 755 , the multi-channel audio decoder 760 , the decompressor 770 , and the multi-channel audio signal reconstructor 780 shown in FIG. 7 may be regarded as being performed by the processor 530 .

[0333] Fig. 7A The operations of the information obtainer 750, the decompressor 770 and the multi-channel audio signal reconstructor 780 respectively include Figure 5B The operations of the information obtainer 550, the decompressor 570 and the multi-channel audio signal reconstructor 580 are described below, and therefore the redundant description will be replaced with reference to Figure 5B In the following, a description will be provided for Figure 5B The description is not redundant.

[0334] The information obtainer 750 may obtain metadata from the bitstream.

[0335] The error removal related information obtainer 755 may obtain the error removal related information from the metadata included in the bitstream. Here, the information about the error removal factor included in the error removal related information may be the error removal factor of the audio signal of one upmix channel of the upmix channel group. The error removal related information obtainer 755 may be included in the information obtainer 750.

[0336] The multi-channel audio signal reconstructor 780 may generate an audio signal of an upmix channel group based on at least one audio signal of a base channel and at least one audio signal of at least one slave channel group. The audio signal of the upmix channel group may be a multi-channel audio signal. The multi-channel audio signal reconstructor 780 may reconstruct an audio signal of one upmix channel by applying an error removal factor to an audio signal of one upmix channel included in the upmix channel group.

[0337] The multi-channel audio signal reconstructor 780 may output a multi-channel audio signal including a reconstructed audio signal of one upmixed channel.

[0338] Figure 7B is a block diagram of a multi-channel audio signal reconstructor according to an embodiment of the present disclosure.

[0339] The multi-channel audio signal reconstructor 780 may include an upmix channel group audio generator 781 and a renderer 783. The renderer 783 may include an error remover 784, a volume controller 785, a limiter 786, and a multi-channel audio signal output unit 787.

[0340] Figure 7B The up-mix channel group audio generator 781, the error remover 784, the volume controller 785, the limiter 786 and the multi-channel audio signal output unit 787 may include Figure 5C The operations of the upmix channel group audio generator 581, the volume controller 588, the limiter 589 and the multi-channel audio signal output unit 590 are described above, and therefore the redundant description will be replaced with reference to Figure 5C In the following, a description will be provided for Figure 5C The description is not redundant.

[0341] The error remover 784 can reconstruct the error-removed audio signal of the first channel based on the audio signal of the first upmixed channel in the upmixed channel group of the multi-channel audio signal and the error removal factor of the first upmixed channel. In this case, the error removal factor can be a value based on the RMS value of the audio signal of the first channel of the original audio signal or the audio signal downmixed from the original audio signal and the RMS value of the audio signal of the first upmixed channel of the upmixed channel group. The first channel and the first upmixed channel can indicate the same channel of the channel layout. The error remover 784 can remove the error caused by encoding by making the RMS value of the audio signal of the first upmixed channel of the current upmixed channel group the RMS value of the audio signal of the first channel of the original audio signal or the audio signal downmixed from the original audio signal.

[0342] The error removal factor may be different between adjacent audio frames. In this case, in the end segment of a frame and the initial segment of the next frame, the audio signal may bounce due to the discontinuity factor used for error removal.

[0343] Therefore, the error remover 784 can determine the error removal factor used in the frame boundary adjacent segment by performing smoothing on the error removal factor. The frame boundary adjacent segment can mean the end segment of the previous frame relative to the boundary and the first segment of the next frame relative to the boundary. Each segment can include a certain number of samples.

[0344] Here, smoothing may refer to an operation of converting a discontinuous error removal factor between adjacent audio frames into a continuous error removal factor in a frame boundary section.

[0345] The multi-channel audio signal output unit 787 may output a multi-channel audio signal including an error-removed audio signal of one channel.

[0346] Depending on circumstances, at least one of the post-processing components 785 and 786 included in the renderer 783 may be omitted, and the order of the post-processing components (such as the error remover 784, the volume controller 785, and the limiter 786) may be changed.

[0347] As described above, the audio encoding apparatus 400 and the audio encoding apparatus 600 may generate a bitstream. The audio encoding apparatus 400 and the audio encoding apparatus 600 may transmit the generated bitstream.

[0348] In this case, a bitstream can be generated in the form of a file stream. The audio decoding device 500 and the audio decoding device 700 can receive the bitstream. The audio decoding device 500 and the audio decoding device 700 can reconstruct the multi-channel audio signal based on the information obtained from the received bitstream. In this case, the bitstream can be included in a certain file container. For example, the file container can be a moving picture expert group (MPEG)-4 media container for compressing each piece of multimedia digital data, such as MPEG-4 part 14 (MP4) etc.

[0349] Fig. 8A This is a view for describing the transmission order and rule of the audio stream in each channel group by the audio encoding device according to an embodiment of the present disclosure.

[0350] In the scalable format, the transmission order and rules of the audio streams in each channel group can be described as follows.

[0351] The audio encoding apparatus 400 and the audio encoding apparatus 600 may first transmit a coupled stream, and then transmit a non-coupled stream.

[0352] The audio encoding apparatus 400 and the audio encoding apparatus 600 may first transmit a coupled stream for a surround channel, and then transmit a coupled stream for a height channel.

[0353] The audio encoding apparatus 400 and the audio encoding apparatus 600 may first transmit a coupled stream of a front channel, and then transmit a coupled stream of a side channel or a rear channel.

[0354] For uncoupled stream transmission, audio encoding device 400 and audio encoding device 600 can first send the stream for the center channel, and then send the stream for the LFE channel and other channels. Herein, when the basic channel group includes a single channel signal, other channels may exist. In this case, the other channels may be one of the left channel L2 or the right channel R2 of the stereo channel.

[0355] The audio encoding device 400 and the audio encoding device 600 may compress the audio signals of the coupled channels into a pair. The audio encoding device 400 and the audio encoding device 600 may first send a coupled stream including the audio signals compressed into a pair. For example, the coupled channels may refer to left-right symmetrical channels such as L / R, Ls / Rs, Lb / Rb, Hfl / Hfr, Hbl / Hbr channels, etc.

[0356] Hereinafter, according to the above-mentioned transmission order and rule of the streams in each channel group, the stream configuration of each channel group in the bit stream 810 of Case 1 will be described.

[0357] refer to Fig. 8A, for example, the audio encoding apparatus 400 and the audio encoding apparatus 600 may compress L1 and R1 signals as 2-channel audio signals, and the compressed L1 and R1 signals may be included in a C1 bitstream of a basic channel group (BCG).

[0358] Next to the basic channel group, the audio encoding apparatus 400 and the audio encoding apparatus 600 may compress the 4-channel audio signal into an audio signal of a subordinate channel group #1.

[0359] The audio encoding apparatus 400 and the audio encoding apparatus 600 may compress the Hfl3 signal and the Hfr3 signal, and the compressed Hfl3 signal and the Hfr3 signal may be included in the C2 bitstream of the bitstream of the slave channel group #1.

[0360] The audio encoding apparatus 400 and the audio encoding apparatus 600 may compress the C signal, and the compressed C signal may be included in the M1 bitstream of the bitstream of the slave channel group #1.

[0361] The audio encoding apparatus 400 and the audio encoding apparatus 600 may compress the LFE signal, and the compressed LFE signal may be included in the M2 bitstream of the bitstream of the slave channel group #1.

[0362] The audio decoding apparatus 500 and the audio decoding apparatus 700 may reconstruct an audio signal of a 3.1.2 channel layout based on compressed audio signals of a basic channel group and a slave channel group #1.

[0363] Next to the slave channel group #2, the audio encoding apparatus 400 and the audio encoding apparatus 600 may compress the 6-channel audio signal into the audio signal of the slave channel group #2.

[0364] The audio encoding apparatus 400 and the audio encoding apparatus 600 may first compress the L signal and the R signal, and the compressed L signal and the R signal may be included in the C3 bitstream of the bitstream of the slave channel group #2.

[0365] Next to the C3 bitstream, the audio encoding apparatus 400 and the audio encoding apparatus 600 may compress the Ls signal and the Rs signal, and the compressed Ls and Rs signals may be included in the C4 bitstream of the bitstream of the slave channel group #2.

[0366] Next to the C4 bitstream, the audio encoding apparatus 400 and the audio encoding apparatus 600 may compress the Hfl signal and the Hfr signal, and the compressed Hfl and Hfr signals may be included in the C5 bitstream of the bitstream of the slave channel group #2.

[0367] The audio decoding apparatus 500 and the audio decoding apparatus 700 may reconstruct an audio signal of a 7.1.4 channel layout based on the compressed audio signals of the basic channel group, the slave channel group #1, and the slave channel group #2.

[0368] Hereinafter, according to the transmission order and rule of the streams in each channel group described above, the stream configuration of each channel group in the bit stream 820 of Case 2 will be described.

[0369] The audio encoding apparatus 400 and the audio encoding apparatus 600 may compress the L2 signal and the R2 signal which are 2-channel audio signals, and the compressed L2 signal and the R2 signal may be included in the C1 bitstream of the bitstream of the basic channel group.

[0370] Next to the basic channel group, the audio encoding apparatus 400 and the audio encoding apparatus 600 may compress the 6-channel audio signal into an audio signal of a subordinate channel group #1.

[0371] The audio encoding apparatus 400 and the audio encoding apparatus 600 may first compress the L signal and the R signal, and the compressed L signal and the R signal may be included in the C2 bitstream of the bitstream of the slave channel group #1.

[0372] The audio encoding apparatus 400 and the audio encoding apparatus 600 may compress the Ls signal and the Rs signal, and the compressed Ls signal and the Rs signal may be included in the C3 bitstream of the bitstream of the slave channel group #1.

[0373] The audio encoding apparatus 400 and the audio encoding apparatus 600 may compress the C signal, and the compressed C signal may be included in the M1 bitstream of the bitstream of the slave channel group #1.

[0374] The audio encoding apparatus 400 and the audio encoding apparatus 600 may compress the LFE signal, and the compressed LFE signal may be included in the M2 bitstream of the bitstream of the slave channel group #1.

[0375] The audio encoding apparatus 500 and the audio decoding apparatus 700 may reconstruct an audio signal of a 7.1.0 channel layout based on compressed audio signals of a base channel group and a dependent channel group #1.

[0376] Next to the slave channel group #1, the audio encoding apparatus 400 and the audio encoding apparatus 600 may compress the 4-channel audio signal into an audio signal of the slave channel group #2.

[0377] The audio encoding apparatus 400 and the audio encoding apparatus 600 may compress the Hfl signal and the Hfr signal, and the compressed Hfl signal and the Hfr signal may be included in the C4 bitstream of the bitstream of the slave channel group #2.

[0378] The audio encoding apparatus 400 and the audio encoding apparatus 600 may compress the Hbl signal and the Hbr signal, and the compressed Hfl signal and the Hfr signal may be included in the C5 bitstream of the bitstream of the slave channel group #2.

[0379] The audio decoding apparatus 500 and the audio decoding apparatus 700 may reconstruct an audio signal of a 7.1.4 channel layout based on the compressed audio signals of the basic channel group, the slave channel group #1, and the slave channel group #2.

[0380] Hereinafter, according to the transmission order and rule of the streams in each channel group described above, the stream configuration of each channel group in the bit stream 830 of Case 3 will be described.

[0381] The audio encoding apparatus 400 and the audio encoding apparatus 600 may compress the L2 signal and the R2 signal which are 2-channel audio signals, and the compressed L2 signal and the R2 signal may be included in the C1 bitstream of the bitstream of the basic channel group.

[0382] Next to the basic channel group, the audio encoding apparatus 400 and the audio encoding apparatus 600 may compress the 10-channel audio signal into an audio signal of a subordinate channel group #1.

[0383] The audio encoding apparatus 400 and the audio encoding apparatus 600 may first compress the L signal and the R signal, and the compressed L signal and the R signal may be included in the C2 bitstream of the bitstream of the slave channel group #1.

[0384] The audio encoding apparatus 400 and the audio encoding apparatus 600 may compress the Ls signal and the Rs signal, and the compressed Ls signal and the Rs signal may be included in the C3 bitstream of the bitstream of the slave channel group #1.

[0385] The audio encoding apparatus 400 and the audio encoding apparatus 600 may compress the Hfl signal and the Hfr signal, and the compressed Hfl signal and the Hfr signal may be included in the C4 bitstream of the bitstream of the slave channel group #1.

[0386] The audio encoding apparatus 400 and the audio encoding apparatus 600 may compress the Hbl signal and the Hbr signal, and the compressed Hfl signal and the Hfr signal may be included in the C5 bitstream of the bitstream of the slave channel group #1.

[0387] The audio encoding apparatus 400 and the audio encoding apparatus 600 may compress the C signal, and the compressed C signal may be included in the M1 bitstream of the bitstream of the slave channel group #1.

[0388] The audio encoding apparatus 400 and the audio encoding apparatus 600 may compress the LFE signal, and the compressed LFE signal may be included in the M2 bitstream of the bitstream of the slave channel group #1.

[0389] The audio encoding apparatus 500 and the audio decoding apparatus 700 may reconstruct an audio signal of a 7.1.4 channel layout based on the compressed audio signals of the basic channel group and the dependent channel group #1.

[0390] The audio decoding apparatus 500 and the audio decoding apparatus 700 may perform demixing in a step-by-step manner by using at least one upmixing unit. The demixing may be performed based on audio signals of channels included in at least one channel group.

[0391] For example, the 1.x to 2.x upmixing unit (first upmixing unit) may demix the audio signal of the right channel from the audio signal of the single channel as the mixed right channel.

[0392] Alternatively, the 2.x to 3.x upmixing unit (second upmixing unit) may demix the audio signal of the center channel from the audio signals of the L2 and R2 channels corresponding to the mixed center channel. Alternatively, the 2.x to 3.x upmixing unit (second upmixing unit) may demix the audio signal of the L3 channel and the audio signal of the R3 channel from the audio signals of the L2 channel and the R2 channel and the audio signal of the C channel of the mixed L3 channel and the R3 channel.

[0393] The 3.x to 5.x upmixing unit (third upmixing unit) may demix the audio signals of the Ls5 channel and the Rs5 channel from the audio signals of the L3, R3, L(5), and R(5) channels corresponding to the Ls5 / Rs5 mixed channel.

[0394] The 5.x to 7.x upmixing units (fourth upmixing units) may demix the audio signal of the Lb channel and the audio signal of the Rb channel from the audio signals of the Ls5, Ls7, and Rs7 channels corresponding to the mixed Lb / Rb channels.

[0395] The xx2(FH) to xx2(H) upmixing unit (fourth upmixing unit) may demix the audio signals of the Hl channel and the Hr channel from the audio signals of the Hfl3, Hfr3, L3, L5, R3, and R5 channels corresponding to the mixed Ls / Rs channels.

[0396] The xx2(H) to xx4 upmixing units (fifth upmixing units) may demix the audio signals of the Hbl channel and the Hbr channel from the audio signals of the Hl, Hr, Hfl, and Hfr channels corresponding to the mixed Hbl / Hbr channels.

[0397] For example, the audio decoding apparatus 500 and the audio decoding apparatus 700 may perform demixing to a 3.2.1 channel layout by using the first upmixing unit.

[0398] The audio decoding apparatus 500 and the audio decoding apparatus 700 may perform demixing to a 7.1.4 channel layout by using the second and third upmixing units for the surround channels and the fourth and fifth upmixing units for the height channels.

[0399] Alternatively, the audio decoding apparatus 500 and the audio decoding apparatus 700 may perform demixing to a 7.1.0 channel layout by using the first mixing unit, the second mixing unit, and the third mixing unit.The audio decoding apparatus 500 and the audio decoding apparatus 700 may not perform demixing from a 7.1.0 channel layout to a 7.1.4 channel layout.

[0400] Alternatively, the audio decoding apparatus 500 and the audio decoding apparatus 700 may perform demixing to a 7.1.4 channel layout by using a first mixing unit, a second mixing unit, and a third mixing unit. The audio decoding apparatus 500 and the audio decoding apparatus 700 may not perform demixing on a height channel.

[0401] Hereinafter, a rule for generating a channel group by the audio encoding apparatus 400 and the audio encoding apparatus 600 will be described. For a channel layout CLi of a scalable format (i is an integer from 0 to n, and CLi indicates Si, Wi, and Hi), Si+Wi+Hi may mean the number of channels of channel group #i. The number of channels of channel group #i may be greater than the number of channels of channel group #i-1.

[0402] Channel group #i can include as many original channels (display channels) of Cli as possible. The original channels can follow the priority described below.

[0403] When H i-1 When it is 0, the height channel can be prioritized over other channels. The center channel and LFE channel can be prioritized before other channels.

[0404] The priority of the highly front channel can be before the priority of the side channel and the highly rear channel.

[0405] The priority of the side channel may be before the priority of the back channel. In addition, the priority of the left channel may be before the priority of the right channel.

[0406] For example, when n is 4, CL0 is a stereo channel, CL1 is a 3.1.2 channel, CL2 is a 5.1.2 channel, and CL3 is a 7.1.4 channel, the channel group may be generated as described below.

[0407] The audio encoding apparatus 400 and the audio encoding apparatus 600 may generate a basic channel group including A (L2) and B (R2) signals. The audio encoding apparatus 400 and the audio encoding apparatus 600 may generate a slave channel group #1 including Q1 (Hfl3), Q2 (Hfr3), T (=C) and P (=LFE) signals. The audio encoding apparatus 400 and the audio encoding apparatus 600 may generate a slave channel group #2 including S1 (=L) and S2 (=R) signals.

[0408] The audio encoding device 400 and the audio encoding device 600 may generate a slave channel group #3 including V1 (Hfl), V2 (Hfr), U1 (Ls), and U2 (Rs) signals.

[0409] The audio decoding apparatus 500 and the audio decoding apparatus 700 may reconstruct a 7.1.4-channel audio signal from a decompressed audio signal by using a downmix matrix. In this case, the downmix matrix may include, for example, downmix weight parameters as in Table 3 provided below.

[0410] Table 3

[0411] L R C LFE LS Rs Lb R Hf HkDJ Hb Hb A(L2 / L3) 1 cw δ*α δ*β B(L2 / L3) 1 c2 δ*α δ*β T(C) 1 P(LFE) 1 Q1(Hfl3) w*δ*α w*δ*β 1 γ Q2(Hfr3) w*δ*α w*δ*β 1 γ S1(L) 1 S2(R) 1 U1(Ls7) 1 U2(Rs7) 1 V1(Hfl3) 1

[0412] Here, cw indicates a center weight, which may be 0 when the channel layout of the basic channel group is a 3.1.2 channel layout, and may be 1 when the channel layout of the basic channel group is a 2-channel layout. In addition, w may indicate a surround to height mixing weight. α, β, γ, and δ may indicate a downmix weight parameter and may be variable. The audio encoding apparatus 400 and the audio encoding apparatus 600 may generate a bitstream including downmix weight parameter information such as α, β, γ, δ, and w, and the audio decoding apparatus 500 and the audio decoding apparatus 700 may obtain the downmix weight parameter information from the bitstream.

[0413] On the other hand, the weight parameter information of the down-mix matrix (or demix matrix) may be in the form of an index. For example, the weight parameter information of the down-mix matrix (or demix matrix) may be index information indicating one down-mix (or demix) weight parameter set among multiple down-mix (or demix) weight parameter sets, and at least one down-mix (or demix) weight parameter corresponding to one down-mix (or demix) weight parameter set may exist in the form of a lookup table (LUT). For example, the weight parameter information of the down-mix (or demix) matrix may be information indicating one down-mix (or demix) weight parameter set among multiple down-mix (or demix) weight parameter sets, and at least one of α, β, γ, δ or w may be predefined in the LUT corresponding to the one down-mix (or demix) weight parameter set. Therefore, the audio decoding device 500 and the audio decoding device 700 may obtain α, β, γ, δ and w corresponding to one down-mix (demix) weight parameter set.

[0414] The matrix for downmixing from the first channel layout to the second channel layout may include multiple matrices. For example, the matrix may include a first matrix for downmixing from the first channel layout to the third channel layout and a second matrix for downmixing from the third channel layout to the second channel layout.

[0415] More specifically, for example, a matrix for downmixing an audio signal of a 7.1.4 channel layout to an audio signal of a 3.1.2 channel layout may include a first matrix for downmixing an audio signal of a 7.1.4 channel layout to an audio signal of a 5.1.4 channel layout and a second matrix for downmixing an audio signal of a 5.1.4 channel layout to an audio signal of a 3.1.2 channel layout.

[0416] Tables 4 and 5 show first and second matrices for downmixing from an audio signal of a 7.1.4 channel layout to an audio signal of a 3.1.2 channel layout based on content-based downmix parameters and surround-to-height-based weights.

[0417] Table 4

[0418] L R C LFE LS Rs Lb R Ls5 α β Rs5 α β

[0419] First matrix (7.1 to 5.1 down-mixing matrix)

[0420] Table 5

[0421] L R C LFE Ls5 Rs5 Hf HkDJ Hb Hb L3 1 0 0 0 γ 0 0 0 0 0 R3 0 1 0 0 0 γ 0 0 0 0 C 0 0 1 0 0 0 0 0 0 0 LFE 0 0 0 1 0 0 0 0 0 0 Hfl3 0 0 0 0 γ*w 0 0 0 δ 0 Hfr3 0 0 0 0 0 γ*w 0 0 0 δ

[0422] Second matrix (5.1.4 to 3.1.2 downmix matrix)

[0423] Here, α, β, γ, or δ indicates one of the downmixing parameters, and w indicates the surround-to-height weight. For upmixing (or demixing) from 5.x channels to 7.x channels, demixing weight parameters α and β may be used.

[0424] For upmixing from xx2(H) channels to xx4 channels, the demixing weight parameter γ may be used.

[0425] For upmixing from 3.x channels to 5.x channels, the demixing weight parameter δ may be used.

[0426] For upmixing from xx2(H) channels to xx2(H) channels, the demixing weight parameters w and δ may be used.

[0427] For up-mixing from 2.x channels to 3.x channels, a demixing weight parameter of -3dB may be used. That is, the demixing weight parameter may be a fixed value and may not be signaled.

[0428] Furthermore, for upmixing to 1.x channels and 2.x channels, a demixing weight parameter of -6dB may be used. That is, the demixing weight parameter may be a fixed value and may not be signaled.

[0429] The demixing weight parameters for demixing may be parameters included in one type among a plurality of types. For example, the demixing weight parameters α, β, γ, and δ of type 1 may be 0 dB, 0 dB, -3 dB, and -3 dB. The demixing weight parameters α, β, γ, and δ of type 2 may be -3 dB, -3 dB, -3 dB, and -3 dB. The demixing weight parameters α, β, γ, and δ of type 3 may be 0 dB, -1.25 dB, -1.25 dB, and -1.25 dB. Type 1 may be a type indicating a case where an audio signal is a general audio signal, Type 2 may be a type indicating a case where a dialogue is included in an audio signal (a dialogue type), and Type 3 may be a type indicating a case where a sound effect exists in an audio signal (a sound effect type).

[0430] The audio encoding device 400 and the audio encoding device 600 can analyze the audio signal and determine one of multiple types according to the analyzed audio signal. The audio encoding device 400 and the audio encoding device 600 can perform down-mixing with respect to the original audio by using the determined type of demixing weight parameter to generate an audio signal of a lower channel layout.

[0431] The audio encoding device 400 and the audio encoding device 600 may generate a bitstream including index information indicating one type of a plurality of types. The audio decoding device 500 and the audio decoding device 700 may obtain the index information from the bitstream and identify one type of the plurality of types based on the obtained index information. The audio decoding device 500 and the audio decoding device 700 may up-mix the audio signal of the decompressed channel group by using the demixing weight parameter of the identified type to reconstruct the audio signal of the specific channel layout.

[0432] Alternatively, the audio signal generated according to the downmix may be expressed as Equation 1 provided below. That is, the downmix may be performed based on an operation using an equation in a first-order polynomial form, and each downmixed audio signal may be generated.

[0433] Equation 1

[0434] Ls5=α×Ls7+β×Lb7

[0435] Rs5=α×R.s7+β×Rb7

[0436] L3=L5+δ×Ls5

[0437] R3=R5+δ×Rs5

[0438] L2=L3+p 2 ×C

[0439] R2=R3+p 2 ×C

[0440] Mono=p 1 ×(L2+R2)

[0441] Hl=Hfl+γ×Hbl

[0442] Hr=Hfr+γ×Hbr

[0443] Hfl3=Hl×w′×δ×Ls5

[0444] Hfr3=Hr×w′×δ×Rs5

[0445] Here, p 1 may be about 0.5 (ie, -6 dB), and p 2α and β may be values ​​for mixing down the number of surround channels from 7 channels to 5 channels. For example, α or β may be 1 (i.e., 0 dB), 0.866 (i.e., -1.25 dB), and 0.707 (i.e., -3 dB). In addition, Y may be a value for mixing down the number of height channels from 4 channels to 5 channels. For example, γ may be one of 0.866 or 0.707. δ may be a value for mixing down the number of surround channels from 5 channels to 3 channels. δ may be one of 0.866 or 0.707. w′ may be a value for mixing down from H2 (e.g., the height channels of a 5.1.2 channel layout or a 7.1.2 channel layout) to Hf2 (the height channels of a 3.1.2 channel layout).

[0446] Likewise, the audio signal generated by demixing can be expressed as Equation 2. That is, demixing can be performed in a step-by-step manner based on operations using equations in a first-order polynomial form (each operation process of the equation corresponds to one demixing process), without being limited to operations using a demixing matrix, and each demixed audio signal can be generated.

[0447] Equation 2

[0448]

[0449] L3=L2-p 2 ×C

[0450] R3=R2-p 2 ×C

[0451]

[0452] Hl=Hfl3-w′×(L3-L5)

[0453] Hr=Hfr3-w′×(R3-R5)

[0454]

[0455] Furthermore, w′ may be a value used for downmixing from H2 (e.g., the height channel of a 5.1.2 channel layout or a 7.1.2 channel layout) to Hf2 (the height channel of a 3.1.2 channel layout) or for demixing from Hf2 (the height channel of a 3.1.2 channel layout) to H2 (e.g., the height channel of a 5.1.2 channel layout or a 7.1.2 channel layout).

[0456] sum can be updated according to w w and the value of w' corresponding thereto. w can be about -1 or 1 and can be sent for each frame.

[0457] For example, sum w The initial value of can be 0, and when w is 1 for each frame, sum w The value of can be increased by 1, and when w is -1 for each frame, sum w The value of can be reduced by 1. w When the value of increases or decreases by 1, w When the value is outside the range of 0-10, sum w The value of can be kept as 0 or 10. w Table 5 showing the relationship between may be as follows. That is, w may be gradually updated for each frame, and thus may be used for demixing from Hf2 to H2.

[0458] [Table 6]

[0459] <![CDATA[sum w ]]> 0 1 2 3 4 5 W’ 0 0.0179 0.0391 0.0658 0.1038 0.25 <![CDATA[sum w ]]> 6 7 8 9 10 W’ 0.3962 w’ 0.4609 0.4821 0.5

[0460] Without being limited thereto, demixing may be performed by integrating a plurality of demixing processes. For example, a signal of the Ls5 channel or the Rs5 channel demixed from the two surround channels of L2 and R2 may be expressed as in Equation 3 which arranges the second to fifth equations of Equation 2.

[0461] Equation 3

[0462]

[0463] The signal of the H1 channel or Hr channel demixed from the two surround channels of L2 and R2 can be expressed as Equation 4, which arranges the second and third equations and the eighth and ninth equations of Equation 2.

[0464] Equation 4

[0465] Hl=Hfl3-w×(L2-p 2 ×C-L5)

[0466] Hr=Hfr3-w×(R2-p 2 ×C-R5)

[0467] Figure 8B and Figure 8C is a diagram for describing the mechanism of gradual downmixing according to an embodiment of the present disclosure. The gradual downmixing of the surround channel and the height channel may have, for example, Figure 8B and Figure 8C The mechanism shown.

[0468] The downmix related information (or demix related information) may be index information indicating one of a plurality of modes based on a combination of 5 preset downmix weight parameters (or demix weight parameters). For example, as shown in Table 7, downmix weight parameters corresponding to a plurality of modes may be predetermined.

[0469] Table 7

[0470] model Down-mixing weight parameters (α, β, γ, δ, w) (or demixing weight parameters) 1 (1,1,0.707,0.707,-1) 2 (0.707,0.707,0.707,0.707,-1) 3 (1,0.866,0.866,0.866,-1) 4 (1,1,0.707,0.707,1) 5 (0.707,0.707,0.707,0.707,1) 6 (1,0.866,0.866,0.866,1)

[0471] In the following, reference will be made to Fig.9A , Fig. 9B , Fig. 10A , Fig. 10B An audio encoding process and an audio decoding process for performing downmixing or demixing based on a scene classification result of an audio signal are described. Hereinafter, embodiments of the present disclosure according to the technical spirit of the present disclosure will be described in detail in sequence.

[0472] Fig.9A is a block diagram of an audio encoding device according to an embodiment of the present disclosure.

[0473] The audio encoding device 900 may include a memory 910 and a processor 930. The audio encoding device 900 may be implemented as a device capable of performing audio processing, such as a server, a TV, a camera, a cellular phone, a tablet PC, a laptop computer, and the like.

[0474] Although in Fig.9A The memory 910 and the processor 930 are separately shown in FIG. 1 , but the memory 910 and the processor 930 may be implemented by one hardware module (eg, a chip).

[0475] The processor 930 may be implemented as a dedicated processor for audio processing based on a neural network. Alternatively, the processor 930 may be implemented by a combination of a general-purpose processor such as an AP, a CPU, or a GPU and software. The dedicated processor may include a memory for implementing an embodiment of the present disclosure or a memory processor for using an external memory.

[0476] The processor 930 may include a plurality of processors. In this case, the processor 330 may be implemented as a combination of dedicated processors, or may be implemented by a combination of software and a plurality of general-purpose processors (such as an AP, a CPU, or a GPU).

[0477] The memory 910 may store one or more instructions for audio processing. In an embodiment of the present disclosure, the memory 910 may store a neural network. When the neural network is implemented in the form of a dedicated hardware chip for artificial intelligence or as part of an existing general-purpose processor (e.g., a CPU or AP) or a graphics-specific processor (e.g., a GPU), the neural network may not be stored in the memory 910. The neural network may be implemented by an external device (e.g., a server), and in this case, the audio encoding device 900 may request and receive result information based on the neural network from the external device.

[0478] The processor 930 may sequentially process the continuous frames according to the instructions stored in the memory 910 and obtain continuous encoded (compressed) frames. The continuous frames may refer to frames constituting the audio.

[0479] Processor 930 can perform audio processing operations with the original audio signal as input, and output a bitstream including a compressed audio signal. In this case, the original audio signal can be a multi-channel audio signal. The compressed audio signal can be a multi-channel audio signal with a number of channels less than or equal to the number of channels of the original audio signal. In this case, the bitstream can include a compressed audio signal of a basic channel group, and in addition include compressed audio signals of n subordinate channel groups (n is an integer greater than or equal to 1). Therefore, according to the number of subordinate channel groups, the number of channels can be freely increased.

[0480] Fig. 9B is a block diagram of an audio encoding device 900 according to an embodiment of the present disclosure.

[0481] The audio encoding apparatus 900 may include a multi-channel audio encoder 950, a bitstream generator 980, and an additional information generator 985. The multi-channel audio encoder 950 may include a multi-channel audio signal processor 960 and a compressor 970.

[0482] Return to reference Fig.9A As described above, the audio encoding device 900 may include a memory 910 and a processor 930, and is used to implement Fig. 9B Instructions of components such as a multi-channel audio encoder 950, a multi-channel audio signal processor 960, a down-mix channel audio generator 965, a compressor 970, a bitstream generator 980, and an additional information generator 985 may be stored in Fig.9A The processor 930 may execute instructions stored in the memory 910. Fig. 9B The multi-channel audio encoder 950, the multi-channel audio signal processor 960, the down-mix channel audio generator 965, the compressor 970, the bitstream generator 980, and the additional information generator 985 shown can be obtained by classifying the operations performed by the processor 930 executing the program (or instruction) stored in the memory 910 according to the function. Therefore, the following description is obtained by Fig. 9B The operations shown to be performed by the multi-channel audio encoder 950 , the multi-channel audio signal processor 960 , the down-mix channel audio generator 965 , the compressor 970 , the bitstream generator 980 , and the additional information generator 985 may be considered to be actually performed by the processor 930 .

[0483] The multi-channel audio signal processor 960 may obtain (eg, generate) at least one audio signal of a basic channel group and at least one audio signal of at least one slave channel group from a multi-channel audio signal (ie, an original audio signal).

[0484] The multi-channel audio signal processor 960 may include an audio scene classifier 961, a downmix channel audio generator 965, and an audio signal classifier 967. The operations of the audio scene classifier 961 and the downmix channel audio generator 965 correspond to Figure 1A , Figure 1B , Figure 1C , Figure 1D and Figure 2A The operations of the audio scene classifier 100 and the audio scene classifier 210 and the downmix channel audio generator 220 are described in detail, and therefore, the detailed description thereof is replaced by Figure 1A , Figure 1B , Figure 1C , Figure 1D and Figure 2A Description.

[0485] The audio scene classifier 961 may include a feature extractor 962, a time correlation analyzer 963, and an estimator 964. The operations of the feature extractor 962, the time correlation analyzer 963, and the estimator 964 correspond to Figure 1A , Figure 1B , Figure 1C , Figure 1D and Figure 2A The operations of the feature extractors 110 and 212, the time correlation analyzers 120 and 214, and the estimators 130 and 216 are replaced by the following descriptions: Figure 1A , Figure 1B , Figure 1C , Figure 1D and Figure 2A Description.

[0486] The audio scene classifier 961 may classify scenes corresponding to the multi-channel audio signal. The audio scene classifier 961 may classify scenes for each frame of the multi-channel audio signal.

[0487] The audio scene classifier 961 may downsample the multi-channel audio signal and classify a scene corresponding to the multi-channel audio signal based on the downsampled multi-channel audio signal.

[0488] In an embodiment of the present disclosure, the audio scene classifier 961 may obtain an audio signal of a center channel (e.g., a C channel signal), an audio signal of a front channel (e.g., an L channel signal and an R channel signal), and an audio signal of a side channel (e.g., an SL channel signal and an SR channel signal) from a multi-channel audio signal. The audio scene classifier 961 may classify the multi-channel audio signal into a dialogue type, an effect type, or a music type based on the obtained audio signal of the center channel, the audio signal of the front channel, and the audio signal of the side channel. However, the present disclosure is not limited thereto, and the multi-channel audio signal may be classified into various scene types having different features (e.g., frequency, intensity, spectrum, etc.) from each other. In an embodiment of the present disclosure, the audio scene classifier 961 may classify the multi-channel audio signal into a default type. Here, the default type indicates a type other than a predetermined scene type.

[0489] The audio scene classifier 961 may determine the scene type of the multi-channel audio signal by using a neural network (e.g., a first neural network and a second neural network) for classifying scenes. Specifically, the audio scene classifier 961 may obtain a probability value corresponding to a predetermined scene type by using a neural network. The audio scene classifier 961 may determine the scene type of the multi-channel audio signal as the scene type having the highest probability value. The audio scene classifier 961 may output the determined scene type (i.e., the scene classification result SC t ).

[0490] The down-mix channel audio generator 965 can be based on the scene classification result SC t The multi-channel audio signal is down-mixed according to the predetermined channel layout. As a result of the down-mixing, the down-mix channel audio generator 965 may generate an audio signal of the predetermined channel layout (i.e., a down-mix audio signal). Specifically, the down-mix channel audio generator 965 may receive the scene classification result SC t The down-mix channel audio generator 965 can obtain the audio corresponding to the scene classification result SC t The downmix profile 966 of the present invention. The downmix channel audio generator 965 can obtain downmix parameters for mixing the multi-channel audio signal into a multi-channel audio signal with different channel layouts based on the downmix profile. The downmix channel audio generator 965 can downmix the multi-channel audio signal based on the downmix parameters.

[0491] Table 8

[0492]

[0493] Table 9

[0494]

[0495] With reference to Table 8, for example, the downmix channel audio generator 965 can downmix the audio signal of 7.1.4 channels into the audio signal of 3.1.2 channels by using the downmix matrix. With reference to Table 9 together with Table 8, the downmix parameters (e.g., α, β, γ) included in the downmix matrix can be configured differently according to the scene type. The downmix configuration file 966 may include data corresponding to the downmix matrix and the downmix parameter settings. The specific operation for downmixing performed by the downmix channel audio generator 965 is similar to the operation described by using equation 1, and is therefore omitted.

[0496] The audio signal classifier 967 may generate at least one audio signal of a base channel group and at least one audio signal of a slave channel group based on the downmix audio signal.

[0497] The compressor 970 can compress the audio signal of the basic channel group and the audio signal of the slave channel group. That is, the compressor 970 can compress at least one audio signal of the basic channel group to obtain at least one compressed audio signal of the basic channel group. Herein, compression can mean compression based on various audio codecs. For example, compression can include transformation and quantization processes.

[0498] The compressor 970 may obtain at least one compressed audio signal of at least one slave channel group by compressing at least one audio signal of at least one slave channel group.

[0499] The additional information generator 985 may generate a scene classification result SC t Additional information.

[0500] The bitstream generator 980 may generate a bitstream including a compressed audio signal of a basic channel group and a compressed audio signal of a dependent channel group.

[0501] The bitstream generator 980 may generate a bitstream further including the additional information generated by the additional information generator 985 .

[0502] More specifically, the bitstream generator 980 may generate a basic audio stream and a subsidiary audio stream. The basic audio stream may include a compressed audio signal of a basic channel group, and the subsidiary audio stream may include a compressed audio signal of a subordinate channel group.

[0503] In addition, the bitstream generator 980 may generate metadata including additional information. As a result, the bitstream generator 980 may generate a bitstream including an elementary audio stream, an auxiliary audio stream, and metadata.

[0504] Although not referenced Fig.9A and Fig. 9B Description, but the audio encoding device 900 can perform reference Figure 4A , Figure 4B , Figure 4C , Figure 4D , Fig. 6A and Figure 6B The operations and functions of the audio encoding device 400 and the audio encoding device 600 are described.

[0505] Fig. 10A is a block diagram of an audio decoding device 1000 according to an embodiment of the present disclosure.

[0506] The audio decoding apparatus 1000 may include a memory 1010 and a processor 1030. The audio decoding apparatus 1000 may be implemented as a device capable of audio processing, such as a server, a TV, a camera, a mobile phone, a tablet PC, a laptop computer, and the like.

[0507] Although in Fig. 10A The memory 1010 and the processor 1030 are separately shown in FIG. 1 , but the memory 1010 and the processor 1030 may be implemented by one hardware module (eg, a chip).

[0508] The processor 1030 may be implemented as a dedicated processor for audio processing based on a neural network. Alternatively, the processor 1030 may be implemented by a combination of a general-purpose processor (such as an AP, a CPU, or a GPU) and software. The dedicated processor may include a memory for implementing an embodiment of the present disclosure or a memory processor for using an external memory.

[0509] The processor 1030 may include a plurality of processors. In this case, the processor 1030 may be implemented as a combination of dedicated processors, or may be implemented by a combination of software and a plurality of general-purpose processors (such as an AP, a CPU, or a GPU).

[0510] The memory 1010 may store one or more instructions for audio processing. In an embodiment of the present disclosure, the memory 1010 may store a neural network. When the neural network is implemented in the form of a dedicated hardware chip for artificial intelligence or as part of an existing general-purpose processor (e.g., a CPU or AP) or a graphics-specific processor (e.g., a GPU), the neural network may not be stored in the memory 1010. The neural network may be implemented by an external device (e.g., a server), and in this case, the audio decoding device 1000 may request and receive result information based on the neural network from the external device.

[0511] The processor 1030 may sequentially process the consecutive frames according to the instructions stored in the memory 1010 to obtain consecutive reconstructed frames. The consecutive frames may refer to frames constituting the audio.

[0512] Processor 1030 can output multi-channel audio signal by performing audio processing operation to input bit stream.Bitstream can be realized in scalable form, to increase the quantity of the channel from basic channel group.For example, processor 1030 can obtain the compressed audio signal of basic channel group from bitstream, and can reconstruct the audio signal (for example, stereo channel audio signal) of basic channel group by decompressing the compressed audio signal of basic channel group.In addition, processor 1030 can reconstruct the audio signal of subordinate channel group by decompressing the compressed audio signal of subordinate channel group from bitstream.Processor 1030 can reconstruct multi-channel audio signal based on the audio signal of basic channel group and the audio signal of subordinate channel group.

[0513] The processor 1030 may reconstruct the audio signal of the first subordinate channel group by decompressing the compressed audio signal of the first subordinate channel group from the bitstream. The processor 1030 may reconstruct the audio signal of the second subordinate channel group by decompressing the compressed audio signal of the second subordinate channel group.

[0514] The processor 1030 may reconstruct a multi-channel audio signal with an increased number of channels based on the audio signal of the basic channel group and the corresponding audio signals of the first and second subordinate channel groups. Similarly, the processor 330 may decompress the compressed audio signals of n subordinate channel groups (where n is an integer greater than 2), and may reconstruct a multi-channel audio signal with a further increased number of channels based on the audio signal of the basic channel group and the corresponding audio signals of the basic channel group and the n subordinate channel groups.

[0515] Fig. 10B is a block diagram of an audio decoding device 1000 according to an embodiment of the present disclosure.

[0516] refer to Fig. 10B , the audio decoding device 1000 may include an information obtainer 1050 and a multi-channel audio decoder 1060. The multi-channel audio decoder 1060 may include a decompressor 1070 and a multi-channel audio signal reconstructor 1080. The multi-channel audio signal reconstructor 1080 may include an up-mix channel group audio generator 1081, a demixing parameter identifier 1082, and a multi-channel audio signal output unit 1083.

[0517] The audio decoding device 1000 may include Fig. 10A The memory 1010 and the processor 1030 are used to implement Fig. 10BInstructions for each of the information obtainer 1050, the multi-channel audio decoder 1060, the decompressor 1070, the multi-channel audio signal reconstructor 1080, the upmix channel group audio generator 1081, the demixing parameter identifier 1082, and the multi-channel audio signal output unit 1083 may be stored in the memory 1010. The processor 1030 may execute the instructions stored in the memory 1010. The processor 1030 may execute the instructions stored in the memory 1010. Fig. 10B The information obtainer 1050, the multi-channel audio decoder 1060, the decompressor 1070, the multi-channel audio signal reconstructor 1080, the up-mixed channel group audio generator 1081, the demixing parameter identifier 1082, and the multi-channel audio signal output unit 1083 shown can be obtained by classifying the operations performed by the processor 1030 executing the program (or instruction) stored in the memory 1010 according to the function. Therefore, the following is described as being obtained by Fig. 10B The operations performed by the illustrated element information obtainer 1050 , the multi-channel audio decoder 1060 , the decompressor 1070 , the multi-channel audio signal reconstructor 1080 , the upmix channel group audio generator 1081 , the demixing parameter identifier 1082 , and the multi-channel audio signal output unit 1083 may be considered to be actually performed by the processor 1030 .

[0518] The information obtainer 1050 may obtain downmix related information corresponding to a downmix audio signal corresponding to the original audio signal and a scene classification result. The scene classification result may be obtained based on a time correlation vector between a first feature vector corresponding to a first frame of the original audio signal and at least one second feature vector corresponding to at least one second frame before the first frame of the original audio signal. Figure 1A , Figure 1B , Figure 1C , Figure 1D , Figure 2A and Figure 2B A description of the specific process of obtaining the scene classification result is therefore omitted below.

[0519] Specifically, the information obtainer 1050 may obtain a basic audio stream and at least one auxiliary audio stream from the bitstream. The basic audio stream may include at least one compressed audio signal of a basic channel group. The auxiliary audio stream may obtain at least one compressed audio signal of at least one subordinate channel group.

[0520] The information obtainer 1050 may obtain metadata from the bitstream. The metadata may include additional information. For example, the metadata may be downmix related information including a scene classification result of a multi-channel audio signal (i.e., an original audio signal). The downmix related information may include index information indicating one of a plurality of scene types. The downmix related information may be obtained for each frame, but the downmix related information may be periodically obtained for various data units. Alternatively, the downmix related information may be obtained aperiodically at each scene change.

[0521] The decompressor 1070 may obtain an audio signal of a basic channel group included in a basic audio stream by decompressing at least one compressed audio signal of the basic channel group. The decompressor 1070 may obtain at least one audio signal of at least one subordinate channel group included in a secondary audio stream from at least one compressed audio signal of at least one subordinate channel group.

[0522] The demixing parameter identifier 1082 may identify the demixing parameters based on the downmixing related information. That is, the demixing parameter identifier 1082 may identify the demixing parameters corresponding to the scene classification result. In other words, the demixing parameter identifier 1082 may identify a scene type from a plurality of scene types based on the index information related to the scene classification result, and identify the demixing parameters corresponding to the identified scene type. The demixing parameters corresponding to the plurality of scene types may be stored in a memory.

[0523] The upmixed channel group audio generator 1081 may generate (or reconstruct) an upmixed channel group audio signal by demixing at least one audio signal of a base channel group and at least one audio signal of at least one subordinate channel group according to a demixing parameter. In an embodiment of the present disclosure, the upmixed channel group audio signal may be an audio signal including at least one channel. In an embodiment of the present disclosure, the upmixed channel group audio signal may be a multi-channel audio signal.

[0524] The multi-channel audio signal output unit 1083 may output at least one upmixed channel group audio signal (ie, output a multi-channel audio signal).

[0525] Although not referenced Fig. 10A and Fig. 10B Description, but the audio decoding device 1000 can perform reference Figure 5A , Figure 5B , Figure 5C , Figure 5D , Fig. 7A and Figure 7B The operations and functions of the audio decoding device 500 and the audio decoding device 700 are described.

[0526] In the above, the audio decoding device 1000 performs the operation of demixing the downmixed audio signal by using the downmix related information generated in units of frames. However, the audio signal in the channel layout (e.g., 7.1.4 channel layout) higher than the audio signal in the output channel layout can be reconstructed. That is, the audio signal in the output layout can be reconstructed without demixing.

[0527] In this case, the audio decoding device 1000 can reconstruct the audio signal in the output channel layout by downmixing the reconstructed audio signal in the higher channel layout using the downmix related information generated in units of frames. As a result, the downmix related information received from the audio decoding device 900 is not limited to being used in the demixing operation by the audio decoding device 1000, and can also be used in the downmixing operation according to the situation.

[0528] However, the flag information is not limited to being transmitted in units of frames, and the downmix related information may be signaled for an upper audio data unit (e.g., a parameter sampling unit) including k frames (k is an integer greater than 1). In this case, information about the size of the upper audio data unit and the downmix related information received from the upper audio data unit may be signaled through a bitstream. The information about the size of the upper audio data unit may be information about the value of k.

[0529] When the downmix related information is received from the higher audio data unit, the downmix related information may not be obtained in units of frames included in the higher data unit. For example, the downmix related information may be obtained in the first frame included in the higher audio data unit, and the downmix related information may not be obtained in frames after the first frame of the higher audio data unit.

[0530] The flag may be obtained in a frame after the first frame of the higher audio data unit.

[0531] Based on the flag, when it is identified that the scene type of the previous frame is different from the scene type of the current frame, downmix related information can be additionally obtained. The downmix related information updated by the flag can be used in frames after the frame in which the flag is obtained in a higher audio data unit.

[0532] When the scene type of the previous frame is the same as the scene type of the current frame, the flag of the current frame is not obtained, but the previously obtained down-mix related information can be used.

[0533] According to an embodiment of the present disclosure, the original sound effect can be maintained through appropriate down-mixing or up-mixing processing according to the scene classification result.

[0534] According to an embodiment of the present disclosure, audio signals can be dynamically mixed so that the audio of the surround channels and the audio of the height channels can be well represented on a large screen. That is, when the audio being reproduced is focused on the surround, the audio signals of the surround channels Ls and Rs can be distributed not only to the L / R channels but also to the height channels, thereby maximizing the surround effect. Alternatively, by mixing the audio signals of the surround channels Ls and Rs to the L / R channels instead of the height channels, horizontal sounds and vertical sounds can be distinguished, so that the surround effect and the height effect can be expressed in a balanced manner at the same time.

[0535] Fig.11 is a block diagram of an audio processing device according to an embodiment of the present disclosure.

[0536] The audio processing apparatus 1100 may include an audio scene classifier 1110, an audio noise cancellation (ANC) mode selector 1120, and a noise canceller 1130. The audio processing apparatus 1100 may cancel ambient sound while outputting an input audio signal.

[0537] The audio scene classifier 1110 can obtain the ambient sound AS t The audio scene classifier 1110 can classify the ambient sound AS t For example, the audio scene classifier 1110 can classify the scene with the ambient sound AS t The configuration, operation and function of the audio scene classifier 1110 correspond to those of reference Figure 1A , Figure 1B , Figure 1C and Figure 1D The configuration, operation, and functions of the audio scene classifier 100 are not described, and thus a detailed description thereof is omitted.

[0538] Although not shown, the audio processing device 1100 may further include a preprocessor. The preprocessor may receive the ambient sound AS t The preprocessor can convert ambient sound into AS t In the embodiment of the present disclosure, the preprocessor can generate a signal corresponding to the ambient sound AS t The preprocessor may transmit the transformed ambient sound (eg, the spectrogram) to the audio scene classifier 1110, and the audio scene classifier 1110 may classify the ambient sound AS based on the transformed ambient sound. t Classify the scenes.

[0539] The active noise cancellation (ANC) mode selector 1120 may select a scene classification result SC based on the scene classification result SC. tTo select a mode of active noise cancellation. For example, the ANC mode selector 1120 may select a first mode of active noise cancellation based on a first danger level, and select a second mode of active noise cancellation based on a second danger level. For example, when the danger level is large, the ANC mode selector 1120 may select a first mode of performing noise cancellation with a stronger intensity, and when the danger level is less than the danger level corresponding to the first mode, select a second mode of performing noise cancellation with a weaker intensity. The ANC mode selector 1120 may transmit data MD corresponding to the selected mode to the noise canceller 1130.

[0540] The noise canceller 1130 can cancel the ambient sound AS t The noise canceller 1130 can determine the ambient sound AS according to the selected mode. t The noise canceller 1130 can generate the noise canceller AS according to the selected mode. t opposite signal.

[0541] Although not shown, the audio processing apparatus 1100 may further include an audio output unit. The audio output unit may output an input audio signal and an opposite signal at the same time.

[0542] According to an embodiment of the present disclosure, as the audio scene classifier 1110, the ambient sound AS t The scene has a small delay time and noise removal according to the scene can be performed quickly.

[0543] Fig.12 is a block diagram of an audio processing device according to an embodiment of the present disclosure.

[0544] The audio processing apparatus 1200 may include an audio scene classifier 1210 and a sound field controller 1220 .

[0545] The audio scene classifier 1210 may classify the first audio signal AS t For example, the first audio signal AS t The audio scene classifier 1110 may be an audio signal corresponding to a current frame (first frame) in an audio signal (e.g., a multi-channel audio signal and a stereo audio signal) provided through a streaming service (e.g., an over-the-top (OTT) service). The configuration, operation, and function of the audio scene classifier 1110 correspond to those of the reference Figure 1A , Figure 1B , Figure 1C and Figure 1D The configuration, operation, and functions of the audio scene classifier 100 are not described, and thus a detailed description thereof is omitted.

[0546] The sound field controller 1220 may classify the scene based on the result SCt Adjust the first audio signal AS t For example, the sound field controller 1220 may obtain a sound field control parameter corresponding to the scene classification result. The sound field control parameter may be set by a user or a manufacturer. The sound field control parameter may vary for each scene type. The sound field controller 1220 may adjust the first audio signal AS based on the sound field control parameter. t frequency, intensity or spectral deformation to adapt the sound field to the scene type.

[0547] Although not shown, the audio processing apparatus 1200 may further include an audio output unit. The audio output unit may output the first audio signal AS t The adaptive audio signal is obtained by adjusting the sound field.

[0548] According to an embodiment of the present disclosure, since scenes of audio signals are classified with a small delay time, a sound field can be adjusted by quickly responding to a scene change.

[0549] Fig.13 is a flowchart of an audio processing method according to an embodiment of the present disclosure. Figure 1A , Figure 1B , Figure 1C and Figure 1D An audio processing method is described. The audio scene classifier 100 or the multi-channel audio signal processor 200 may be referred to as an audio processing device.

[0550] In operation S1310, the audio processing apparatus may obtain a first audio signal corresponding to a first frame.

[0551] In operation S1320, the audio processing apparatus may extract a first feature vector by using a first neural network having the first audio signal as an input.

[0552] In operation S1330, the audio processing apparatus may obtain a time correlation vector representing similarity between the first feature vector and at least one second feature vector extracted from at least one second audio signal corresponding to at least one second frame before the first frame.

[0553] In operation S1340, a scene of the first audio signal may be classified by using the first feature vector, at least one second feature vector, and a second neural network having the time correlation vector as an input.

[0554] Fig.14A is a flowchart of an audio processing method according to an embodiment of the present disclosure. Figure 1A , Figure 1B , Figure 1C , Figure 1D, Figure 2A and Fig.13 An audio processing method is described. The audio scene classifier 100 or the multi-channel audio signal processor 200 may be referred to as an audio processing device.

[0555] Fig.14A Operation S1410, operation S1420, operation S1430, and operation S1440 may correspond to Fig.13 Therefore, the above reference is omitted. Fig.13 Provide a description.

[0556] Fig.14A Operation S1410 may include operation S1412 and operation S1414.

[0557] In operation S1412, the audio processing apparatus may obtain a multi-channel audio signal corresponding to a plurality of frames including a first frame and at least one second frame.

[0558] In operation S1414, the audio processing apparatus may extract a first audio signal including at least one channel signal from among the multi-channel audio signals corresponding to the first frame. In an embodiment of the present disclosure, the audio processing apparatus may down-mix the multi-channel audio signal to the first audio signal.

[0559] In operation S1450, the audio processing apparatus may downmix the multi-channel audio signal corresponding to the first frame based on the scene classification result.

[0560] Fig. 14B is a flowchart of an audio processing method according to an embodiment of the present disclosure. Figure 1A , Figure 1B , Figure 1C , Figure 1D , Figure 2A and Fig.14A An audio processing method is described. The audio scene classifier 100 or the multi-channel audio signal processor 200 may be referred to as an audio processing device.

[0561] In the embodiments of the present disclosure, Fig.14A Operation S1450 may include operation S1452, operation S1454, and operation S1456.

[0562] In operation S1452, the audio processing apparatus may obtain a downmix profile corresponding to the scene classification result.

[0563] In operation S1454, the audio processing apparatus may obtain a downmix parameter for mixing the multi-channel audio signal into a multi-channel audio signal having a different channel layout based on the downmix profile.

[0564] In operation S1456, the audio processing apparatus may downmix the multi-channel audio signal corresponding to the first frame according to the channel layout based on the downmix parameter.

[0565] Fig.15 is a flowchart of an audio processing method according to an embodiment of the present disclosure. Fig. 10A and Fig. 10B An audio processing method is described. The audio decoding apparatus 1000 may be referred to as an audio processing apparatus.

[0566] In operation S1510, the audio processing apparatus may obtain downmix related information including a downmixed audio signal corresponding to an original audio signal and a scene classification result from a bitstream.

[0567] In operation S1520, the audio processing apparatus may demix the downmix audio signal based on the downmix related information.

[0568] In operation S1530, an audio signal including at least one channel may be reconstructed based on the demixed audio signal.

[0569] Fig.16 is a flowchart of an audio processing method according to an embodiment of the present disclosure. Figure 1A , Figure 1B , Figure 1C , Figure 1D , Fig.11 and Fig.13 Describes audio processing methods.

[0570] Fig.16 Operation S1610, operation S1620, operation S1630, and operation S1640 may correspond to Fig.13 Therefore, the above reference is omitted. Fig.13 Provide a description.

[0571] In operation S1650, the audio processing device may select one of the modes of active noise cancellation based on the scene classification result. In an embodiment of the present disclosure, the audio processing device may generate an opposite signal for canceling the first audio signal based on the selected mode.

[0572] Fig.17 is a flowchart of an audio processing method according to an embodiment of the present disclosure. Figure 1A , Figure 1B , Figure 1C , Figure 1D , Fig.12 and Fig.13Describes audio processing methods.

[0573] Fig.17 Operation S1710, operation S1720, operation S1730, and operation S1740 may correspond to Fig.13 Therefore, reference is omitted. Fig.13 Provide a description.

[0574] In operation S1750, the audio processing apparatus may adjust the sound field of the first audio signal based on the scene classification result. In an embodiment of the present disclosure, the audio processing apparatus may output the adjusted signal.

[0575] Fig.18 is a block diagram of an audio processing device according to an embodiment of the present disclosure. Fig.18 The audio processing device 1800 may be a speaker device for reproducing audio, or may be a separate server device. The audio processing method according to the embodiments described in the specification may be performed by the speaker device, may be performed by a separate server device, or may be performed jointly by the speaker device and the separate server device (including the division of the process in the audio processing method and execution by multiple devices).

[0576] In the following, for ease of explanation, a description is provided under the assumption that the audio processing device 1800 performs the audio processing method. However, as described above, the present disclosure is not limited thereto, and a separate server device may exist, and the additional server device may perform part or all of the process. Therefore, in the embodiments described below, it should be understood that the operations performed by the audio processing device 1800 may also be performed by a separate computing device (such as a server device) even if not specifically explained.

[0577] refer to Fig.18 , the audio processing device 1800 according to an embodiment of the present disclosure may include a memory 1810, a processor 1830, an input / output interface 1850, and a communication interface 1870. However, the elements of the audio processing device 1800 are not limited to the above examples, and the audio processing device 1800 may include more or less elements than the above elements. In an embodiment of the present disclosure, at least some of the memory 1810, the processor 1830, the input / output interface 1850, and the communication interface 1870 may be implemented in the form of a single chip, and the processor 1830 may include one or more processors.

[0578] The memory 1810 is an element for storing various programs or data, and may include a storage medium such as a read-only memory (ROM), a random access memory (RAM), a hard disk, a compact disk read-only memory (CD-ROM), and a digital video disk (DVD) or a combination of storage media. The memory 1810 may not exist alone, but is integrated into the processor 1830. The memory 1810 may include a volatile memory, a nonvolatile memory, or a combination of a volatile memory and a nonvolatile memory. A program for performing operations according to an embodiment to be described later may be stored in the memory 1810. The memory 1810 may provide the stored data to the processor 1830 in response to a request of the processor 1830.

[0579] The processor 1830 is used to control a series of processes so that the audio processing device 1800 can process the audio according to the reference Figure 1A , Figure 1B , Figure 1C , Figure 1D , Figure 2A , Figure 2B , Figure 3A , Figure 3B , Figure 4A , Figure 4B , Figure 4C , Figure 4D , Figure 5A , Figure 5B , Figure 5C , Figure 5D , Fig. 6A , Figure 6B , Fig. 7A , Figure 7B , Fig. 8A , Figure 8B , Figure 8C , Fig.9A , Fig. 9B , Fig. 10A , Fig. 10B , Fig.11 , Fig.12 , Fig.13 , Fig.14A , Fig. 14B , Fig.15 , Fig.16 and Fig.17 The embodiments described herein are elements for operating, and may include one or more processors. The one or more processors may include general-purpose processors such as a central processing unit (CPU), an application processor (AP), a digital signal processor (DSP), etc., dedicated graphics processors such as a graphics processing unit (GP), a visual processing unit (VPU), etc., or dedicated AI processors such as a neural processing unit (NPU). For example, when one or more processors are dedicated AI processors, they can be designed with a hardware structure dedicated to processing with a specific AI model (e.g., a neural network model).

[0580] The processor 1830 may record data in the memory 1810 or read data stored in the memory 1810. In particular, the processor 1830 may process data according to a predefined rule or an AI model by executing a program stored in the memory 1810. Therefore, unless otherwise specifically explained, the processor 1830 may perform the operations described in the above embodiments, and the operations described in the above embodiments as being performed by the audio processing device 1800 may be considered to be performed by the processor 1830.

[0581] The input / output interface 1850 may include an input interface (e.g., a touch screen, a hard button, a microphone, etc.) for receiving an input of a control command or information from a user, and an output interface (e.g., a display panel, a speaker, etc.) for displaying an execution result of an operation according to a user's control or a state of the audio processing device 1800. According to an embodiment of the present disclosure, the input / output interface 1850 may display a moving image being reproduced, or may receive an input from a user for enlarging a partial area of ​​a moving image or selecting a specific object included in the moving image.

[0582] The communication interface 1870 is an element for transmitting and receiving signals (control commands, data, etc.) to and from an external device via a cable or wirelessly, and may be configured to include a communication chipset supporting various communication protocols. The communication interface 1870 may receive a signal from the outside and output the received signal to the processor 1830, or may transmit a signal output from the processor 1830 to the outside.

[0583] Figure 1A , Figure 1B , Figure 1C , Figure 1D , Figure 2A , Fig.11 and Fig.12 The elements shown in the embodiment (e.g., feature extractor 110, first neural network 112, time correlation analyzer 120, estimator 130, cascade function 132, second neural network 134, preprocessor 140, audio scene classifier 210, feature extractor 212, time correlation analyzer 214, estimator 216, downmix channel audio generator 220, audio scene classifier 1110, ANC mode selector 1120, noise canceller 1130, audio scene classifier 1210, and sound field controller 1220) can be executed by executing the program (or instructions) stored in the memory 1810 according to the function. Fig.18 The operations performed by the processor 1830 are classified to obtain. Therefore, it is described as being composed of Figure 1A , Figure 1B , Figure 1C, Figure 1D , Figure 2A , Fig.11 and Fig.12 The operations performed by the elements shown in FIG. 18 may be considered to be actually performed by the processor 1830 .

[0584] In the embodiments of the present disclosure, Fig.18 The audio processing device 1800 may correspond to Figure 4A and Fig.9A The audio encoding device 400 and the audio encoding device 900 or Figure 5A and Fig. 10A The audio decoding device 500 and the audio decoding device 1000 of the present invention may be executed with reference to Figure 3A , Figure 3B , Figure 4A , Figure 4B , Figure 4C , Figure 4D , Figure 5A , Figure 5B , Figure 5C , Figure 5D , Fig. 6A , Figure 6B , Fig. 7A , Figure 7B , Fig. 8A , Figure 8B , Figure 8C , Fig.9A , Fig. 9B , Fig. 10A and Fig. 10B At least some functions of the audio encoding device 400, the audio encoding device 600, and the audio encoding device 900 or the audio decoding device 500, the audio decoding device 700, and the audio decoding device 1000 are described.

[0585] In an embodiment of the present disclosure, the audio processing method may include an operation of obtaining a first audio signal corresponding to a first frame. The audio processing method may include an operation of extracting a first feature vector by using a first neural network having the first audio signal as an input. The audio processing method may include an operation of obtaining a time correlation vector representing a similarity between the first feature vector and at least one second feature vector extracted from at least one second audio signal corresponding to at least one second frame preceding the first frame. The audio processing method may include an operation of classifying a scene of the first audio signal by using a second neural network having the first feature vector, at least one second feature vector, and the time correlation vector as input.

[0586] In an embodiment of the present disclosure, the operation of obtaining the first audio signal may include the operation of obtaining a multi-channel audio signal corresponding to a plurality of frames including a first frame and at least one second frame. The operation of obtaining the first audio signal may include the operation of extracting a first audio signal including at least one channel signal from the multi-channel audio signal corresponding to the first frame. According to an embodiment of the present disclosure, for scene classification corresponding to the multi-channel audio signal, some channels of the multi-channel audio signal may be used, so that computing resources may be used efficiently.

[0587] In an embodiment of the present disclosure, the audio processing method may further include an operation of down-mixing the multi-channel audio signal corresponding to the first frame based on the scene classification result.

[0588] In an embodiment of the present disclosure, the down-mixing operation may include an operation of obtaining a down-mixing configuration file corresponding to the scene classification result. The down-mixing operation may include obtaining down-mixing parameters for mixing a multi-channel audio signal into a multi-channel audio signal with different channel layouts based on the down-mixing configuration file. The down-mixing operation may include an operation of down-mixing the multi-channel audio signal corresponding to the first frame according to the channel layout based on the down-mixing parameters. In an embodiment of the present disclosure, an audio signal with a small channel layout can be effectively generated by down-mixing the multi-channel audio signal according to the scene type.

[0589] In an embodiment of the present disclosure, the time correlation vector may include an inner product of a first eigenvector and at least one second eigenvector and an inner product of a first eigenvector and the first eigenvector.

[0590] In an embodiment of the present disclosure, the second neural network may receive as input a value obtained by concatenating the first feature vector, at least one second feature vector, and the time correlation vector.

[0591] In an embodiment of the present disclosure, the at least one second feature vector may include N second feature vectors. The second neural network may receive as input a value obtained by concatenating the first feature vector, M second feature vectors from the N second feature vectors, and a time correlation vector. Here, N is a natural number greater than M. According to an embodiment of the present disclosure, data of a relatively small size is input to the neural network, so that computing resources can be used efficiently.

[0592] In an embodiment of the present disclosure, the classification step may include an operation of classifying the scene of the first audio signal into a dialogue type, a music type, or a sound effect type.

[0593] In an embodiment of the present disclosure, the audio processing method may further include an operation of selecting one of the modes of active noise cancellation based on the scene classification result.

[0594] In an embodiment of the present disclosure, the classification step may include an operation of classifying a degree of danger corresponding to the first audio signal. The selection step may include an operation of selecting a first mode of active noise cancellation based on the first degree of danger, and selecting a second mode of active noise cancellation based on the second degree of danger. According to an embodiment of the present disclosure, whether to eliminate ambient sound is determined based on the scene classification result, so that the user can quickly identify dangerous situations through ambient sound.

[0595] In an embodiment of the present disclosure, the audio processing method may further include an operation of adjusting the sound field of the first audio signal based on the scene classification result. According to an embodiment of the present disclosure, when the sound field is adjusted according to the scene corresponding to the first audio signal, a richer listening experience can be provided to the user.

[0596] In an embodiment of the present disclosure, the audio processing device may include a memory storing one or more instructions. The audio processing device may include at least one processor that executes one or more instructions stored in the memory. The at least one processor may execute one or more instructions to obtain a first audio signal corresponding to a first frame. The at least one processor may execute one or more instructions to extract a first feature vector by using a first neural network with the first audio signal as input. The at least one processor may execute one or more instructions to obtain a time correlation vector representing the similarity between the first feature vector and at least one second feature vector extracted from at least one second audio signal corresponding to at least one second frame before the first frame. The at least one processor may execute one or more instructions to classify the scene of the first audio signal by using a second neural network with the first feature vector, at least one second feature vector, and the time correlation vector as input.

[0597] In an embodiment of the present disclosure, the audio processing method may include an operation of obtaining downmix related information including a downmix audio signal corresponding to an original audio signal and a scene classification result from a bitstream. The audio processing method may include an operation of demixing the downmix audio signal based on the downmix related information. The audio processing method may include an operation of reconstructing an audio signal including at least one channel based on the demixed audio signal. The scene classification result may be obtained based on a time correlation vector between a first feature vector corresponding to a first frame of the original audio signal and at least one second feature vector corresponding to at least one second frame before the first frame of the original audio signal.

[0598] In an embodiment of the present disclosure, the audio processing device may include a memory storing one or more instructions. The audio processing device may include at least one processor that executes one or more instructions stored in the memory. At least one processor may execute one or more instructions to obtain downmix related information including a downmixed audio signal and a scene classification result from an original audio signal. At least one processor may execute one or more instructions to demix the downmixed audio signal based on the downmix related information. At least one processor may execute one or more instructions to reconstruct an audio signal based on the demixed audio signal. The scene classification result may be obtained based on a time correlation vector between a first feature vector corresponding to a first frame of the original audio signal and at least one second feature vector corresponding to at least one second frame before the first frame of the original audio signal.

[0599] The above-described embodiments of the present disclosure may be written as a program or instructions executable on a computer, and the program or instructions may be stored in a storage medium.

[0600] The machine-readable storage medium may be provided in the form of a non-transitory storage medium. The term "non-transitory storage medium" simply means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but the term does not distinguish between data being semi-permanently stored in the storage medium and data being temporarily stored in the storage medium. For example, a non-transitory storage medium may include a buffer that temporarily stores data.

[0601] In an embodiment of the present disclosure, the above-mentioned method according to various embodiments of the present disclosure may be provided in a computer program product. The computer program product may be a commercial product that can be traded between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or distributed online (e.g., downloaded or uploaded) via an application store, or distributed (e.g., downloaded or uploaded) directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be at least temporarily stored or arbitrarily created in a storage medium that may be readable to, for example, a manufacturer's server, an application store's server, or a relay server.

[0602] The model associated with the above-mentioned neural network can be implemented as a software module. When implemented as a software module (eg, a program module including instructions), the neural network model can be stored on a computer-readable recording medium.

[0603] In addition, the neural network model can be integrated in the form of a hardware chip and can be part of the above-mentioned device. For example, the neural network model can be made in the form of a dedicated hardware chip for artificial intelligence, or as part of a general-purpose processor (e.g., CPU or AP) in a related field or a graphics-specific processor (e.g., GPU).

[0604] In addition, the neural network model can be provided in the form of downloadable software. The computer program product may include a product in the form of a software program that is electronically distributed electronically through a manufacturer or an electronic market (e.g., a downloadable application). For electronic distribution, at least a portion of the software program may be stored in a storage medium or temporarily generated. In this case, the storage medium may be a server of the manufacturer or electronic market, or a storage medium of a relay server.

[0605] The technical spirit of the present disclosure is described in detail with reference to the exemplary embodiments, but the technical spirit of the present disclosure is not limited to the above embodiments, and a person of ordinary skill in the art can make various changes and modifications to the technical spirit of the present disclosure within the technical spirit of the present disclosure without being limited to the aforementioned embodiments.

Claims

1. An audio processing method for scene classification, the audio processing method comprising: Obtaining a first audio signal corresponding to a first frame (S1310; S1410; S1610; S1710); Extracting a first feature vector by inputting the first audio signal into a first neural network (S1320; S1420; S1620; S1720); Obtaining a time correlation vector representing a similarity between the first feature vector and at least one second feature vector extracted from at least one second audio signal corresponding to at least one second frame temporally preceding the first frame (S1330; S1430; S1630; S1730); and The scene of the first audio signal is classified by inputting the first feature vector, the at least one second feature vector, and the time correlation vector into a second neural network (S1340; S1440; S1640; S1740).

2. The audio processing method according to claim 1, in, The obtaining of the first audio signal (S1410) includes: obtaining a multi-channel audio signal corresponding to a plurality of frames including the first frame and the at least one second frame (S1412); and The first audio signal including at least one channel signal is extracted from the multi-channel audio signal corresponding to the first frame (S1414). 3 . The audio processing method according to claim 2 , further comprising: based on a result of the classification of the scene, down-mixing the multi-channel audio signal corresponding to the first frame ( S1450 ).

4. The audio processing method according to claim 3, in, The downmix comprises: Obtaining a downmix configuration file corresponding to the result of classifying the scene (S1452); Based on the downmix configuration file, obtaining downmix parameters for mixing the multi-channel audio signal into a multi-channel audio signal with a different channel layout (S1454); as well as Based on the down-mixing parameters, the multi-channel audio signal corresponding to the first frame is down-mixed according to the channel layout (S1456).

5. The audio processing method according to any one of claims 1 to 4, in, The time correlation vector includes a first inner product of the first eigenvector and the at least one second eigenvector and a second inner product of the first eigenvector and the first eigenvector.

6. The audio processing method according to any one of claims 1 to 5, in, The second neural network receives as input a value obtained by concatenating the first feature vector, the at least one second feature vector, and the time correlation vector.

7. The audio processing method according to any one of claims 1 to 5, in, The at least one second eigenvector comprises N second eigenvectors, wherein the second neural network receives an input of a value obtained by cascading the first feature vector, M second feature vectors among the N second feature vectors, and the time correlation vector, and Here, each of N and M is a natural number, and N is greater than M.

8. The audio processing method according to any one of claims 1 to 7, in, The classifying the scene (S1340; S1440; S1640; S1740) includes: classifying the scene of the first audio signal into any one of a dialogue type, a music type, and a sound effect type.

9. The audio processing method according to any one of claims 1 and 5 to 8, further comprising: selecting a mode from modes of active noise cancellation based on a result of classifying the scene (S1650).

10. The audio processing method according to claim 9, in, The classifying of the scene comprises: classifying the danger level corresponding to the first audio signal, and The selecting includes selecting a first mode of the active noise cancellation based on a first risk level, and selecting a second mode of the active noise cancellation based on a second risk level.

11. The audio processing method according to any one of claims 1 and 5 to 8, further comprising: adjusting the sound field of the first audio signal based on a result of classifying the scene (S1750).

12. An audio processing device for scene classification, the audio processing device comprising: a memory storing at least one instruction (1810); and At least one processor (1830) configured to execute the at least one instruction (1810) to: Obtaining a first audio signal corresponding to a first frame; extracting a first feature vector by inputting the first audio signal into a first neural network; obtaining a time correlation vector, the time correlation vector representing a similarity between the first feature vector and at least one second feature vector extracted from at least one second audio signal corresponding to at least one second frame temporally preceding the first frame; as well as The scene of the first audio signal is classified by inputting the first feature vector, the at least one second feature vector and the time correlation vector into a second neural network.

13. The audio processing device according to claim 12, in, The at least one processor (1830) is further configured to execute the at least one instruction to select one of the modes of active noise cancellation based on a result of the scene classification.

14. The audio processing device according to claim 12, in, The at least one processor (1830) is also configured to execute the at least one instruction to adjust the sound field of the first audio signal based on a result of classifying the scene.

15. An audio processing method for scene classification, the audio processing method comprising: Obtaining downmix related information including a downmix audio signal corresponding to an original audio signal and a scene classification result from a bitstream (S1510); Demixing the downmix audio signal based on the downmix related information (S1520); and reconstructing an audio signal including at least one channel based on the demixed audio signal (S1530), in, The scene classification result is obtained based on a time correlation vector between a first feature vector and at least one second feature vector, wherein the first feature vector corresponds to a first frame of the original audio signal, and the at least one second feature vector corresponds to at least one second frame that temporally precedes the first frame of the original audio signal.

Images