Audio encoding apparatus and method, and audio decoding apparatus and method

Abstract

According to one embodiment, an audio signal processing apparatus is disclosed, comprising: a memory for storing instructions; and a processor for executing the instructions stored in the memory, wherein: the processor generates a first audio signal in a frequency domain by performing a frequency transform on a first audio signal comprising n channels, generates a frequency feature signal of a specific channel from the first audio signal in the frequency domain on the basis of a first DNN, generates a second audio signal comprising m (m < n) channels from the first audio signal on the basis of a second DNN, and generates an output audio signal by encoding the second audio signal and the frequency feature signal; the first audio signal is a higher order surround sound signal comprising a zero order signal and a plurality of first order signals, and the second audio signal comprises a mono signal or a stereo signal.

Description

Technical Field

[0001] The present disclosure relates to the field of audio encoding and decoding. More specifically, the present disclosure relates to the field of encoding and decoding audio including multiple channels based on artificial intelligence (AI). Background Art

[0002] Audio is encoded by an encoder-decoder conforming to a specific compression standard, for example, the Advanced Audio Coding (AAC) standard, the OPUS standard, etc., and then stored in a recording medium or transmitted through a communication channel in the form of a bitstream.

[0003] Generally, since general encoder-decoders do not support encoding / decoding multi-channel audio to provide a three-dimensional spatial effect to listeners, a method for encoding / decoding multi-channel audio at a low bitrate using a general encoder-decoder is required. Summary of the Invention

[0004] Technical Problem

[0005] The technical objective of an embodiment is to encode / decode multi-channel audio by using a general encoder-decoder that supports few-channel audio encoding / decoding.

[0006] In addition, the technical objective of an embodiment is to encode multi-channel audio at a low bitrate and reconstruct multi-channel audio with high quality.

[0007] Technical Solution

[0008] According to an embodiment, an audio signal processing apparatus may include: a memory storing one or more instructions; and a processor configured to execute the one or more instructions stored in the memory, wherein the processor is configured to: perform a frequency transform on a first audio signal including n channels to generate a first audio signal in the frequency domain, generate a frequency feature signal for each channel from the first audio signal in the frequency domain based on a first deep neural network (DNN), generate a second audio signal including m (where m < n) channels from the first audio signal based on a second DNN, and generate an output audio signal by encoding the second audio signal and the frequency feature signal, wherein the first audio signal is a high-order ambisonic signal including a zero-order signal and multiple first-order signals, and the second audio signal includes one of a mono signal and a stereo signal.

[0009] Advantageous Effects

[0010] According to an embodiment, multi-channel audio can be encoded / decoded by using a general encoder-decoder that supports few-channel audio encoding / decoding.

[0011] Furthermore, according to the embodiments, multi-channel audio can be encoded at a low bit rate and reconstructed with high quality.

[0012] However, the effects that can be obtained from the audio encoding apparatus and method and the audio decoding apparatus and method according to the embodiments are not limited to the effects described above, and other undescribed effects will be clearly understood by those skilled in the art from the following description. Attached Figure Description

[0013] Brief descriptions of the various figures are provided to provide a full understanding of the figures in this specification.

[0014] Figure 1 The process of encoding and decoding audio according to an embodiment is illustrated.

[0015] Figure 2 A block diagram illustrating the configuration of an audio encoding apparatus according to an embodiment is shown.

[0016] Figure 3 An example of a signal included in a high-order surround sound signal is shown.

[0017] Figure 4 A first deep neural network (DNN) according to an embodiment is shown.

[0018] Figure 5 The first audio signal in the frequency domain is shown. Figure 4 The comparison between the frequency characteristic signals shown.

[0019] Figure 6 A second DNN according to an embodiment is shown.

[0020] Figure 7 A method for combining audio feature signals and frequency feature signals is shown.

[0021] Figure 8 A method for combining audio feature signals and frequency feature signals is shown.

[0022] Figure 9 A block diagram illustrating the configuration of an audio decoding apparatus according to an embodiment is shown.

[0023] Figure 10 A third DNN according to an embodiment is shown.

[0024] Figure 11 A fourth DNN according to an embodiment is shown.

[0025] Figure 12 The methods for training the first DNN, second DNN, third DNN, and fourth DNN are shown.

[0026] Figure 13A flowchart is shown for describing the process of training a first DNN, a second DNN, a third DNN, and a fourth DNN by a training device.

[0027] Figure 14 A flowchart is shown for describing the process of training a first DNN, a second DNN, a third DNN, and a fourth DNN by a training device.

[0028] Figure 15 Another method of training a first DNN, a second DNN, a third DNN, and a fourth DNN is shown.

[0029] Figure 16 A flowchart is shown for describing another process of training a first DNN, a second DNN, a third DNN, and a fourth DNN by a training device.

[0030] Figure 17 A flowchart is shown for describing another process of training a first DNN, a second DNN, a third DNN, and a fourth DNN by a training device.

[0031] Figure 18 A flowchart is shown for describing an audio encoding method according to an embodiment.

[0032] Figure 19 A flowchart is shown for describing an audio decoding method according to an embodiment. Detailed Description of the Invention

[0033] According to an embodiment, an audio signal processing device may include: a memory storing one or more instructions; and a processor configured to execute the one or more instructions stored in the memory, wherein the processor is configured to: perform a frequency transform on a first audio signal including n channels to generate a first audio signal in the frequency domain, generate a frequency feature signal for each channel from the first audio signal in the frequency domain based on a first deep neural network (DNN), generate a second audio signal including m (where m < n) channels from the first audio signal based on a second DNN, and generate an output audio signal by encoding the second audio signal and the frequency feature signal, wherein the first audio signal is a high-order surround sound signal including a zero-order signal and a plurality of first-order signals, and the second audio signal includes one of a mono signal and a stereo signal.

[0034] The frequency feature signal may include a representative value for each channel, and the representative value for each channel may be values of a plurality of frequency bands corresponding to each channel of the first audio signal in the frequency domain.

[0035] The second DNN may obtain an audio feature signal from the first audio signal, and may output the second audio signal from an integrated feature signal in which the audio feature signal and the frequency feature signal are combined.

[0036] Integrated feature signals can be obtained by replacing samples of some channels in the audio feature signal with samples of the frequency feature signal.

[0037] The channels may include a predetermined number of consecutive channels starting from the first channel of the audio feature signal or a predetermined number of consecutive channels starting from the last channel of the audio feature signal.

[0038] The duration of an audio characteristic signal can be equal to the duration of a frequency characteristic signal.

[0039] In frequency characteristic signals, the number of samples for each channel within a predetermined time period can be 1.

[0040] The output audio signal can be represented as a bitstream, and frequency characteristic signals can be included in a supplementary region of the bitstream.

[0041] The processor can be configured to obtain a second audio signal by combining an intermediate audio signal output from a second DNN with a reduced-channel audio signal from a first audio signal.

[0042] The first DNN can be trained based on a comparison between a frequency domain training signal converted from a first training signal and a frequency domain training signal reconstructed from a frequency feature signal used for training by a DNN used for training, and the frequency feature signal used for training can be obtained from the frequency domain training signal based on the first DNN.

[0043] The second DNN can be trained based on at least one of the following results: a comparison of a second training signal obtained from the first training signal via the second DNN with a reduced few-channel training signal from the first training signal, a comparison of the first training signal with a fourth training signal reconstructed from the audio data used for training, and a comparison of a frequency feature signal used for training with a frequency feature signal used for training obtained from the audio data used for training.

[0044] The first DNN and the second DNN can be trained alternately.

[0045] According to another embodiment, the audio signal processing apparatus may include: a memory storing one or more instructions; and a processor configured to execute one or more instructions stored in the memory, wherein the processor is configured to: generate a third audio signal and a frequency feature signal comprising m channels by decoding an input audio signal; generate a weight signal comprising n (where n>m) channels from the frequency feature signal based on a third deep neural network (DNN); and generate a fourth audio signal comprising n channels by applying the weight signal to an intermediate audio signal comprising n channels generated from the third audio signal via a fourth DNN, wherein the third audio signal comprises one of a mono signal and a stereo signal, and the fourth audio signal is a high-order surround sound signal comprising a zero-order signal and a plurality of first-order signals.

[0046] The fourth DNN can obtain an integrated feature signal by processing the third audio signal, and output an intermediate audio signal from the audio feature signal included in the integrated feature signal. The frequency feature signal can be extracted from the integrated feature signal and then input into the third DNN.

[0047] The frequency characteristic signal may include a predetermined number of consecutive channels starting from the first channel in the integrated characteristic signal channel or a predetermined number of consecutive channels starting from the last channel.

[0048] The third and fourth DNNs can process frequency feature signals and audio feature signals respectively, thereby outputting weighted signals and intermediate audio signals with the same time length as the fourth audio signal.

[0049] The processor can be configured to obtain a fourth audio signal by multiplying samples of the intermediate audio signal with samples of the weighted signal.

[0050] The third and fourth DNNs can be trained based on at least one of the following results: the result of comparing a second training signal obtained from the first training signal via the second DNN with a reduced few-channel training signal from the first training signal; the result of comparing the first training signal with a fourth training signal reconstructed from the audio data used for training via the third and fourth DNNs; and the result of comparing a frequency feature signal used for training obtained via the first DNN with a frequency feature signal used for training obtained via the fourth DNN from the audio data used for training.

[0051] According to another embodiment, the audio signal processing method may include: performing frequency transformation on a first audio signal comprising n (where n is a natural number greater than 1) channels to generate a first audio signal in the frequency domain; generating a frequency feature signal for each channel from the first audio signal in the frequency domain based on a first DNN; generating a second audio signal comprising m (where m is a natural number less than n) channels from the first audio signal based on a second DNN; and generating an output audio signal by encoding the second audio signal and the frequency feature signals, wherein the first audio signal is a high-order surround sound signal comprising a zero-order signal and a plurality of first-order signals, and the second audio signal comprises one of a mono signal and a stereo signal.

[0052] According to another embodiment, the audio signal processing method may include: generating a third audio signal and a frequency feature signal comprising m channels by decoding an input audio signal; generating a weight signal comprising n (where n>m) channels from the frequency feature signal based on a third deep neural network (DNN); and generating a fourth audio signal comprising n channels by applying the weight signal to an intermediate audio signal comprising n channels generated from the third audio signal via a fourth DNN, wherein the third audio signal comprises one of a mono signal and a stereo signal, and the fourth audio signal is a high-order surround sound signal comprising a zero-order signal and a plurality of first-order signals.

[0053] Inventive Method

[0054] Because this disclosure allows for various variations and numerous embodiments, specific embodiments will be shown in the accompanying drawings and described in detail in the written description. However, this is not intended to limit this disclosure to the specific embodiments, and it is to be understood that all changes, equivalents, and substitutions that do not depart from the spirit and technical scope of this disclosure are included in the embodiments of this disclosure.

[0055] In the description of the embodiments, certain detailed explanations of the related technologies are omitted where it is believed that they may unnecessarily obscure the essence of this disclosure. Furthermore, the numbers used in the description of the embodiments (e.g., first, second, etc.) are merely identifier codes used to distinguish one element from another.

[0056] Furthermore, in this specification, it can be understood that when elements are “connected” or “coupled” to each other, elements can be directly connected or coupled to each other, but they can also be connected or coupled to each other through intermediate elements, unless otherwise stated.

[0057] In this specification, elements denoted as "-er(or)," "unit," or "module" may be combined into one element, or one element may be split into two or more elements, depending on the specific functions they perform. Furthermore, each element described below, in addition to its own primary function, may perform some or all of the functions performed by another element, and some of the primary functions of each element may be entirely performed by another element.

[0058] Furthermore, in this specification, "deep neural network (DNN)" is a representative example of an artificial neural network model that simulates brain nerves, and is not limited to artificial neural network models that use a specific algorithm.

[0059] Furthermore, in this specification, "parameters" are values ​​used in the computation of each layer of a neural network; for example, they may include weights used when input values ​​are applied to a specific computational expression. The parameters can be represented in matrix form. The parameters are values ​​set as a result of training and can be updated as needed using separate training data.

[0060] Furthermore, in this specification, "first audio signal" refers to the audio to be encoded, and "second audio signal" refers to the audio obtained as a result of performing artificial intelligence (AI) encoding on the first audio signal. Additionally, "third audio signal" refers to the audio obtained through first decoding during the audio decoding process, and "fourth audio signal" refers to the audio obtained as a result of performing AI encoding on the third audio signal.

[0061] Furthermore, in this specification, "first DNN" refers to a DNN used to obtain the frequency feature signal of the first audio signal, and "second DNN" refers to a DNN used to perform AI downscaling on the first audio signal. Additionally, "third DNN" refers to a DNN used to obtain the weight signal from the frequency feature signal, and "fourth DNN" refers to a DNN used to perform AI upscaling on the third audio signal.

[0062] Furthermore, in this specification, "AI reduction" refers to AI-based processing for reducing the number of audio channels, and "first encoding" refers to encoding processing using an audio compression method based on frequency transformation. Additionally, "first decoding" refers to decoding processing using an audio reconstruction method based on frequency transformation, and "AI amplification" refers to AI-based processing for increasing the number of audio channels.

[0063] In the following sections, embodiments based on the technical concepts of this disclosure will be described in sequence.

[0064] Figure 1 The process of encoding and decoding audio according to an embodiment is illustrated.

[0065] As mentioned above, due to the increase in the number of audio channels, the amount of information processed for encoding / decoding increases, thus requiring solutions to improve the encoding and decoding efficiency of audio signals.

[0066] like Figure 1 As shown, during the audio encoding process, a first audio signal 105, which includes multiple channels, is AI encoded (110) to obtain a second audio signal 115, which includes a small number of channels. The first audio signal 105 may be surround sound audio including W, X, Y, and Z channels, and the second audio signal 115 may be stereo audio including left (L) and right (R) channels or mono audio including one channel. In an embodiment, the first audio signal 105 may be audio with more than one channel, such as 5-channel audio, 6-channel audio, or 9-channel audio. In this disclosure, audio signals with a large number of channels, such as the first audio signal 105 and the fourth audio signal 145, may be referred to as multi-channel audio signals, and audio signals with a small number of channels, such as the second audio signal 115 and the third audio signal 135, may be referred to as few-channel audio signals. The number of channels in a few-channel audio signal may be less than the number of channels included in a multi-channel audio signal.

[0067] In this disclosure, a first encoding (120) and a first decoding (130) are performed on a second audio signal 115 that has fewer channels compared to the first audio signal 105, such that the first audio signal 105 can be encoded / decoded even by using a codec that does not support encoding / decoding of multi-channel audio signals.

[0068] Reference Figure 1 In a detailed description, during the audio encoding process, a first audio signal 105 comprising n channels is AI encoded (110) to obtain a second audio signal 115 comprising m channels, and the second audio signal 115 is then subjected to a first encoding (120). In one embodiment, n and m are natural numbers, where m is less than n. In another embodiment, n and m may be rational numbers.

[0069] During the audio decoding process, the audio data obtained as a result of AI encoding (110) is received, a third audio signal 135 with m channels is obtained through first decoding (130), and a fourth audio signal 145 with n channels is obtained by performing AI decoding (140) on the third audio signal 135.

[0070] During AI encoding (110), when the first audio signal 105 is input, it is AI-simplified to obtain a second audio signal 115 with fewer channels. During AI decoding (140), when the third audio signal 135 is input, it is AI-amplified to obtain a fourth audio signal 145. That is, since the number of channels of the first audio signal 105 is reduced by AI encoding (110), and the number of channels of the third audio signal 135 is increased by AI decoding (140), it is necessary to minimize the difference between the first audio signal 105 and the fourth audio signal 145 caused by the change in the number of channels.

[0071] In embodiments of this disclosure, frequency characteristic signals are used to compensate for changes in the number of channels that occur during the AI ​​encoding (110) and AI decoding (140) processes. The frequency characteristic signals represent the correlation between the channels of the first audio signal 105, and during the AI ​​decoding (140) process, a fourth audio signal 145 that is the same as or similar to the first audio signal 105 can be reconstructed based on the frequency characteristic signals.

[0072] The AI ​​used for AI encoding (110) and AI decoding (140) can be implemented as a DNN. This will be discussed in the following text. Figure 12 As described, the DNN used for AI encoding (110) and AI decoding (140) is jointly trained by sharing loss information, which can minimize the difference between the first audio signal 105 and the fourth audio signal 145.

[0073] Detailed description Figure 1 The first encoding (120) and first decoding (130) shown can reduce the amount of information in the second audio signal 115 reduced from the first audio signal 105AI through the first encoding (120). The first encoding (120) may include a process of transforming the second audio signal 115 to the frequency domain, a process of quantizing the signal that has been transformed to the frequency domain, a process of entropy encoding the quantized signal, etc. The process of the first encoding (120) can be implemented by using one of the audio signal compression methods based on frequency transformation using the Advanced Audio Coding (AAD) standard, the OPUS standard, etc.

[0074] The third audio signal 135 corresponding to the second audio signal 115 can be reconstructed by the first decoding (130) of the audio data. The first decoding (130) may include a process of generating a quantized signal by entropy decoding of the audio data, a process of inverse quantization of the quantized signal, and a process of transforming the frequency domain signal into a time domain signal. The process of the first decoding (130) can be implemented by using one of the audio signal reconstruction methods corresponding to audio signal compression methods based on frequency transformation using AAC standards, OPUS standards, etc., and is used in the process of the first encoding (120).

[0075] The audio data obtained through the audio encoding process may include frequency characteristic signals. As described above, a fourth audio signal 145 that is the same as or similar to the first audio signal 105 is reconstructed using the frequency characteristic signals.

[0076] Audio data can be transmitted in the form of a bitstream. The audio data may include data obtained based on sample values ​​in the second audio signal 115, such as the quantized sample values ​​of the second audio signal 115. Furthermore, the audio data may include multiple pieces of information used in the first encoding (120) process, such as prediction mode information, quantization parameter information, etc. Audio data can be generated according to the rules of an audio signal compression method, such as a syntax, which uses frequency-conversion-based audio signal compression methods such as the AAD standard and the OPUS standard.

[0077] Figure 2 A block diagram showing the configuration of the encoding device 200 (or audio encoding device) according to an embodiment is shown.

[0078] Reference Figure 2 The encoding apparatus 200 according to the embodiment may include an AI encoder 210 and a first encoder 230. The AI ​​encoder 210 may include a transformer 212, a feature extractor 214, and an AI reducer 216. Figure 2 As shown, the encoding device 200 according to the embodiment may further include a conventional reducer 250.

[0079] Although Figure 2 The AI ​​encoder 210, the first encoder 230, and the conventional reducer 250 are shown as separate components, but they can all be implemented by a single processor. In this case, they can be implemented as a dedicated processor or as a combination of software and a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU). The dedicated processor may include memory for implementing embodiments of this disclosure, or it may include a storage processor for using external memory.

[0080] The AI ​​encoder 210, the first encoder 230, and the conventional reducer 250 can be configured with multiple processors. In this case, they can be implemented as a combination of dedicated processors or a combination of software and multiple general-purpose processors such as an AP, CPU, or GPU. The transformer 212, the feature extractor 214, and the AI ​​reducer 216 can be implemented with different processors.

[0081] The AI ​​encoder 210 obtains frequency feature signals from a first audio signal 105 comprising n channels and a second audio signal 115 comprising m channels. In one embodiment, n and m are natural numbers, where m is less than n. In another embodiment, n and m may be rational numbers.

[0082] The first audio signal 105 can be a high-order surround sound signal comprising n channels. More specifically, the first audio signal 105 can be a high-order surround sound signal comprising a zero-order signal and multiple first-order signals. Reference will now be made to... Figure 3 Describes high-order surround sound signals.

[0083] Figure 3 An example of a signal included in a high-order surround sound signal is shown.

[0084] High-order surround sound signals can include zero-order signals corresponding to the W channel, first-order signals corresponding to the X, Y, and Z channels, and second-order signals corresponding to the R, S, and other channels. Although in Figure 3 It is not shown in the figure, but high-order surround sound signals can also include third-order signals, fourth-order signals, etc.

[0085] In an embodiment, the first audio signal 105 may include a zero-order signal corresponding to the W channel, and signals of higher orders than the zero-order signal (e.g., first-order signals corresponding to the X, Y, and Z channels). In an embodiment, the first audio signal 105 may include a first-order signal and signals of higher orders than the first-order signal.

[0086] In an embodiment, the second audio signal 115 may be one of a stereo signal and a mono signal.

[0087] Reference Figure 2 The second audio signal 115 can be output to the first encoder 230, and the frequency feature signal can be output from the feature extractor 214 to the AI ​​reducer 216 or the first encoder 230.

[0088] The AI ​​encoder 210 can obtain a frequency feature signal and a second audio signal 115 based on AI. Here, AI can instruct the processing performed by a DNN. Specifically, the AI ​​encoder 210 can obtain the frequency feature signal by using a first DNN and can obtain the second audio signal 115 by using a second DNN.

[0089] The AI ​​encoder 210 performs AI reduction to reduce the number of channels in the first audio signal 105 and obtains frequency characteristic signals indicating the characteristics of each channel of the first audio signal 105. The second audio signal 115 and the frequency characteristic signals can be sent to the decoding device 900 (or audio decoding device) through predetermined processing, and the decoding device 900 can reconstruct a fourth audio signal 145 that is the same as or similar to the first audio signal 105 by using the frequency characteristic signals.

[0090] The AI ​​encoder 210 is described in detail. The converter 212 transforms the first audio signal 105 from the time domain to the frequency domain, thereby obtaining a first audio signal in the frequency domain. The converter 212 can transform the first audio signal 105 into a first audio signal in the frequency domain according to various transformation methods, including Short Time Fourier Transform (STFT).

[0091] The first audio signal 105 includes samples identified based on the vocal tract and time, and the first audio signal in the frequency domain includes samples identified based on the vocal tract, time, and frequency bin. Here, the frequency bin indicates a frequency index that indicates the frequency (or frequency band) corresponding to the value of each sample.

[0092] Feature extractor 214 obtains frequency feature signals from the first audio signal in the frequency domain through the first DNN. As described above, the frequency feature signals indicate the correlation between the channels of the first audio signal 105, and the decoding device 900, which will be described below, can obtain a fourth audio signal 145 that is the same as or similar to the first audio signal 105 by using the frequency feature signals.

[0093] Feature extractor 214 obtains a frequency feature signal with fewer samples than the first audio signal in the frequency domain. The reason for obtaining the frequency feature signal is to compensate for signal loss due to the change in the number of channels reduced according to AI, so as to facilitate encoding by the first encoder 230 and reduce the number of bits in the audio data. The correlation between the channels of the first audio signal 105 can be detected from the first audio signal in the frequency domain; however, because the first audio signal in the frequency domain has n channels like the first audio signal 105, it is not encoded first, and its larger size increases the number of bits in the audio data. Therefore, according to the embodiment, feature extractor 214 can obtain a frequency feature signal with fewer samples than the first audio signal in the frequency domain, thereby simultaneously reducing the number of bits in the audio data and sending the correlation signal between the channels of the first audio signal 105 to the decoding device 900.

[0094] AI reducer 216 obtains second audio signal 115 by processing first audio signal 105 through second DNN. The number of channels in second audio signal 115 can be less than the number of channels in first audio signal 105. As described above, first encoder 230 does not support encoding of first audio signal 105, but can support encoding of second audio signal 115.

[0095] In an embodiment, the first audio signal 105 may be 4-channel surround sound audio, and the second audio signal 115 may be stereo audio, but the number of channels of the first audio signal 105 and the second audio signal 115 is not limited to 4 channels and 2 channels respectively.

[0096] When the frequency feature signal obtained by the feature extractor 214 is output to the AI ​​reducer 216, the AI ​​reducer 216 embeds the frequency feature signal during the processing of the first audio signal 105 by the second DNN. (Refer to the following...) Figures 6 to 8 Describe the process of embedding frequency characteristic signals.

[0097] The first encoder 230 can perform a first encoding on the second audio signal 115 output from the AI ​​reducer 216, thereby reducing the amount of information in the second audio signal 115. As a result of the first encoding by the first encoder 230, audio data can be obtained. The audio data can be represented in the form of a bitstream and can be transmitted to the decoding device 900 via a network. The audio data can serve as a reference for the output audio signal.

[0098] When the frequency feature signal is output from the feature extractor 214 to the first encoder 230, the first encoder 230 performs a first encoding on the second audio signal 115 and the frequency feature signal. In an embodiment, the frequency feature signal may have n channels, just like the first audio signal 105, and thus may be included in a supplementary region corresponding to the bitstream of audio data, rather than using a frequency-transformation-based encoding method. For example, the frequency feature signal may be included in the payload region or a user-defined region of the audio data.

[0099] like Figure 2 As shown, the encoding device 200 may also include a conventional reducer 250, which reduces the first audio signal 105 to obtain a multi-channel audio signal. For example, the multi-channel audio signal may have m channels, just like the second audio signal 115.

[0100] The few-channel audio signal can be combined into an audio signal output from the AI ​​reducer 216, and the resulting second audio signal 115 can be input to the first encoder 230.

[0101] In an embodiment, the conventional reducer 250 can obtain a low-channel audio signal by using at least one of a variety of algorithms for reducing the number of channels of the first audio signal 105.

[0102] For example, when the first audio signal 105 is a 4-channel audio signal including W-channel, X-channel, Y-channel, and Z-channel signals, two or more of these signals can be combined to obtain a few-channel audio signal. Here, the W-channel signal can indicate the sum of the intensities of sound sources in all directions, the X-channel signal can indicate the difference in intensities between front and rear sound sources, the Y-channel signal can indicate the difference in intensities between left and right sound sources, and the Z-channel signal can indicate the difference in intensities between upper and lower sound sources. When the second audio signal 115 is stereo audio, the conventional reducer 250 can obtain the signal obtained by subtracting the Y-channel signal from the W-channel signal as the left (L) signal, and can obtain the signal obtained by adding the W-channel and Y-channel signals as the right (R) signal. Another example is that the conventional reducer 250 can obtain a few-channel audio signal through UHJ encoding.

[0103] The few-channel audio signal corresponds to the predicted version of the second audio signal 115, and the audio signal output from the AI ​​reducer 216 corresponds to the residual version of the second audio signal 115. That is, the few-channel audio signal corresponding to the predicted version of the second audio signal 115 is combined with the audio signal output from the AI ​​reducer 216 in the form of skip connections, thereby reducing the number of layers in the second DNN.

[0104] In the following text, reference will be made to Figures 4 to 8 A first DNN for extracting frequency feature signals and a second DNN for performing AI reduction processing on the first audio signal 105 are described.

[0105] Figure 4 A first DNN 400 according to an embodiment is shown.

[0106] The first DNN 400 may include at least one convolutional layer and at least one shaping layer.

[0107] Convolutional layers process input data through filters of predetermined size to obtain feature data. The parameters of the convolutional layer filters can be optimized through the training process described below.

[0108] The shaping layer changes the size of the input data by altering the sample positions of the input data.

[0109] Reference Figure 4The first audio signal 107 in the frequency domain is input to the first DNN 400. The first audio signal 107 in the frequency domain includes samples identified based on the vocal tract, time, and frequency band. That is, the first audio signal 107 in the frequency domain can be three-dimensional data of the samples. Each sample of the first audio signal 107 in the frequency domain can be a frequency coefficient obtained as a result of frequency transformation.

[0110] Figure 4 The first audio signal 107 in the frequency domain is shown to have a size of (32, 4, 512), which means that the duration of the first audio signal 107 in the frequency domain is 32, the number of channels is 4, and the number of frequency bands is 512. The duration of 32 means that the number of frames is 32, and each frame corresponds to a predetermined time period (e.g., 5ms). The size of the first audio signal 107 in the frequency domain (32, 4, 512) is merely an example; according to embodiments, the size of the first audio signal 107 in the frequency domain or the size of the input / output signals for each layer can be varied.

[0111] The first convolutional layer 410 processes the first audio signal 107 in the frequency domain through filters, each filter being 3×1 in size. As a result of the processing of the first convolutional layer 410, a feature signal 415 of size (32, 4, a) can be obtained.

[0112] The second convolutional layer 420 processes the input signal through b filters, each filter being 3×1 in size. As the result of the processing of the second convolutional layer 420, a feature signal 425 of size (32, 4, b) can be obtained.

[0113] The third convolutional layer 430 processes the input signal through four filters, each with a size of 3×1. As the result of the processing of the third convolutional layer 430, a feature signal 435 with a size of (32, 4, 4) can be obtained.

[0114] The shaping layer 440 obtains a frequency feature signal 109 of size (128, 4) by changing a feature signal 435 of size (32, 4, 4). The shaping layer 440 can also obtain a frequency feature signal 109 of size (128, 4) by moving a sample identified by the second frequency band from the sample of the feature signal 435 of size (32, 4, 4) in the time axis direction to the fourth frequency band.

[0115] According to an embodiment of this disclosure, a first DNN 400 obtains a frequency feature signal 109 having the same number of channels as the first audio signal 107 in the frequency domain; however, within a predetermined time period, the number of samples per channel is less than that of the first audio signal 107 in the frequency domain. Although Figure 4The first DNN 400 is shown to include three convolutional layers and one shaping layer, but this is merely an example. The number of convolutional and shaping layers in the first DNN 400 can vary as long as a frequency feature signal 109 is available, where the number of channels is equal to the first audio signal 107 in the frequency domain, and the number of samples is less than that of the first audio signal 107 in the frequency domain. Similarly, the shaping layer can be replaced with a convolutional layer, and the number and size of filters used in each convolutional layer can differ.

[0116] Figure 5 The first audio signal 107 in the frequency domain is shown. Figure 4 The comparison between the frequency characteristic signals 109 shown.

[0117] Each sample of the first audio signal 107 in the frequency domain is identified based on frame (i.e., time), frequency band, and channel. (See reference...) Figure 5 During the first frame, there are k samples in the first channel.

[0118] Compared to the first audio signal 107 in the frequency domain, the frequency characteristic signal 109 has a small number of samples for each channel within a predetermined time period. For example, the number of samples for each channel can be 1 within the predetermined time period. Figure 5 As shown, the number of samples included in the first channel during the first frame can be 1.

[0119] The samples of the frequency characteristic signal 109 can be representative values ​​of multiple frequency bands of a specific channel within a predetermined time period. For example, the representative value of the fourth channel during the first frame, i.e., the sample value 0.5, can be the representative value of the frequency band corresponding to the first frequency band to the k-th frequency band during the first frame.

[0120] As described above, the frequency characteristic signal 109 can indicate the correlation between the channels of the first audio signal 105, particularly the correlation between the channels of the first audio signal 105 in the frequency domain. For example, a sample value of 0 for the third channel during the first frame of the frequency characteristic signal 109 may mean that the sample value, i.e., the frequency coefficient, of the third channel signal during the first frame of the first audio signal 107 in the frequency domain may be 0. Furthermore, if the sample value of the first channel is 0.5 and the sample value of the second channel is 0.2 during the first frame of the frequency characteristic signal 109, this may mean that the non-zero frequency components, i.e., the non-zero frequency coefficients, in the first channel signal of the first audio signal 107 in the frequency domain may be greater than those in the second channel signal.

[0121] According to embodiments of this disclosure, by using a frequency feature signal 109 with fewer samples than the first audio signal 107 in the frequency domain to send a correlation signal between channels to the decoding device 900, the number of bits of audio data can be reduced compared to the case where the first audio signal 107 in the frequency domain is used.

[0122] Figure 6 A second DNN 600 according to an embodiment is shown.

[0123] The second DNN 600 includes at least one convolutional layer and at least one shaping layer.

[0124] Unlike the two-dimensional convolutional layers in the first DNN 400, at least one convolutional layer in the second DNN 600 can be a one-dimensional convolutional layer. For convolution processing, the filters in a one-dimensional convolutional layer move only in the horizontal or vertical direction according to the stride, but the filters in a two-dimensional convolutional layer move in both the horizontal and vertical directions according to the stride.

[0125] Reference Figure 6 The first audio signal 105 is input to the second DNN 600. The samples of the first audio signal 105 are identified by time and channel. That is to say, the first audio signal 105 can be two-dimensional data.

[0126] The first convolutional layer 610 convolves the first audio signal 105 using filters, each with a size of 33. The filter size of the first convolutional layer 610 being 33 likely means that the filter's horizontal size is 33, and its vertical size is equal to the vertical size of the input signal, i.e., the vertical size (number of channels) of the first audio signal 105. As a result of the processing by the first convolutional layer 610, a feature signal 615 with a size of (128, a) is output.

[0127] The second convolutional layer 620 receives the output signal of the first convolutional layer 610 as input, and then processes the input signal through b filters, each filter having a size of 33. As a result of the processing, an audio feature signal 625 of size (128, b) can be obtained. According to the combination scheme of the frequency feature signal 109 described below, the size of the audio feature signal 625 can be (128, b-4).

[0128] The frequency feature signal 109 can be embedded in the processing of the second DNN 600 relative to the first audio signal 105, such as... Figure 6 As shown, the frequency feature signal 109 can be combined with the audio feature signal 625, and the integrated feature signal 628 obtained as a result of the combination can be input to the next layer.

[0129] Now refer to Figure 7 and Figure 8 A method for describing the combination of frequency characteristic signal 109 and audio characteristic signal 625.

[0130] Figure 7 and Figure 8 A method for combining audio feature signal 625 and frequency feature signal 109 is shown.

[0131] Reference Figure 7 The predetermined number of channels of the audio characteristic signal 625 ( Figure 7 The four samples of the audio feature signal 625 can be replaced by samples of the frequency feature signal 109. The channels of the audio feature signal 625 to be replaced may include a predetermined number of consecutive channels starting from the first channel of the audio feature signal 625 or a predetermined number of consecutive channels starting from the last channel. For example, when the frequency feature signal 109 has four channels, the samples of the first to fourth channels of the audio feature signal 625 are replaced by samples of the frequency feature signal 109, thereby obtaining the integrated feature signal 628.

[0132] Next, refer to Figure 8 The frequency characteristic signal 109 can be added to the audio characteristic signal 625. That is, when the audio characteristic signal 625 has b-4 channels and the frequency characteristic signal 109 has 4 channels, the frequency characteristic signal 109 can be added to the audio characteristic signal 625 to obtain an integrated characteristic signal 628 with b channels. The frequency characteristic signal 109 can be added before the first channel of the audio characteristic signal 625, or it can be added after the last channel of the audio characteristic signal 625.

[0133] exist Figure 7 and Figure 8 The reason for combining the frequency characteristic signal 109 with the audio characteristic signal 625 at the front or back end is to make it easier for the decoding device 900 to separate the frequency characteristic signal from the integrated characteristic signal.

[0134] Refer to the return Figure 6 The integrated feature signal 628 is input to the shaping layer 630. The integrated feature signal 628 of size (128, b) can be transformed into a feature signal 635 of size (16384, 2) through the shaping layer 630.

[0135] The output signal 635 of the shaping layer 630 is input to the third convolutional layer 640. The third convolutional layer 640 processes the signals input by two filters through convolution to obtain a second audio signal 115 with a size of (16384, 2), where each filter has a size of 1. The size of the second audio signal 115 being (16384, 2) means that the second audio signal 115 is a stereo signal with 16384 frames and two channels. According to an embodiment, when the second audio signal 115 is a mono signal, the size of the second audio signal 115 can be (16384, 1).

[0136] According to an embodiment, the second DNN 600 outputs a second audio signal 115, which has the same duration as the first audio signal 105 and has fewer channels than the first audio signal 105. Assuming the second DNN 600 can output such a second audio signal 115, then the second DNN 600 can have, in addition to... Figure 6 Various structures other than the one shown. In other words, although Figure 6 The second DNN 600 is shown to include three convolutional layers and one shaping layer, but this is merely an example. Therefore, the number of convolutional and shaping layers included in the second DNN 600 can vary, as long as a second audio signal 115 with the same duration as the first audio signal 105 and fewer channels than the first audio signal 105 is available. Similarly, the shaping layer can be replaced with convolutional layers, and the number and size of filters used in each convolutional layer can be different.

[0137] The encoding device 200 can transmit audio data obtained through AI encoding and first encoding to the decoding device 900 via a network. According to an embodiment, the audio data can be stored in a data storage medium, including magnetic media such as hard disks, floppy disks, or magnetic tapes; optical recording media such as optical disc read-only memory (CD-ROM) or digital multifunction disc (DVD); or magneto-optical media such as optical discs.

[0138] Figure 9 A block diagram showing the configuration of an audio decoding device 900 according to an embodiment is shown.

[0139] Reference Figure 9 The decoding device 900 includes a first decoder 910 and an AI decoder 930. The AI ​​decoder 930 may include a weighted signal acquirer 912, an AI amplifier 914, and a combiner 916.

[0140] Although Figure 9The first decoder 910 and the AI ​​decoder 930 are shown as separate elements, but both can be implemented by a single processor. In this case, they can be implemented as a dedicated processor or as a combination of software and a general-purpose processor such as an application processor (AP), CPU, or GPU. The dedicated processor may include memory for implementing embodiments of this disclosure or may include a storage processor for using external memory.

[0141] The first decoder 910 and the AI ​​decoder 930 can be configured with multiple processors. In this case, they can be implemented as a combination of dedicated processors or as a combination of software and multiple general-purpose processors such as an AP, CPU, or GPU. The weight signal acquirer 912, the AI ​​amplifier 914, and the combiner 916 can be implemented with different processors.

[0142] The first decoder 910 acquires audio data. The audio data acquired by the first decoder 910 can be used as a reference for the input audio signal. The audio data can be received via a network or obtained from a data storage medium, including magnetic media such as hard disks, floppy disks, or magnetic tapes; optical recording media such as CD-ROMs or DVDs; or magneto-optical media such as optical discs.

[0143] The first decoder 910 performs a first decoding of the audio data. The third audio signal 135 is obtained as a result of the first decoding of the audio data, and the third audio signal 135 is output to the AI ​​amplifier 914. The third audio signal 135 may include m channels as a second audio signal 115.

[0144] As described above, when the frequency characteristic signal is included in the supplementary region of the audio data, the frequency characteristic signal is reconstructed through a first decoding relative to the audio data. When the frequency characteristic signal is embedded in the third audio signal 135, the frequency characteristic signal can be obtained through processing by the fourth DNN of the AI ​​amplifier 914.

[0145] The AI ​​decoder 930 reconstructs a fourth audio signal 145, which includes n channels, based on the third audio signal 135 and frequency characteristic signals.

[0146] Since the signal loss caused by the change in the audio channel due to AI reduction cannot be compensated by the fourth audio signal 145 obtained by AI amplification of the third audio signal 135, the AI ​​decoder 930 obtains a weighted signal from the frequency feature signal to compensate for the signal loss, according to the embodiment.

[0147] In detail, the weighted signal acquisition unit 912 processes the frequency feature signal with n channels through a third DNN to obtain a weighted signal with n channels. The duration of the weighted signal can be equal to the duration of the intermediate audio signal obtained by the AI ​​amplifier 914, and can be greater than the duration of the frequency feature signal. The sample values ​​included in the weighted signal are the weights applied to the samples of the intermediate audio signal obtained by the AI ​​amplifier 914, and are used to reflect the correlation between the channels of the first audio signal 105 and the sample values ​​of each channel of the intermediate audio signal.

[0148] Now refer to Figure 10 Describe the third DNN of the weighted signal acquisition unit 912.

[0149] Figure 10 A third DNN 1000 according to an embodiment is shown.

[0150] Reference Figure 10 The third DNN 1000 may include at least one convolutional layer and at least one shaping layer. The convolutional layer included in the third DNN 1000 may be a two-dimensional convolutional layer.

[0151] The frequency feature signal 136 is input into the third DNN 1000, and the weight signal 137 is obtained through the processing in the third DNN 1000.

[0152] like Figure 10 As shown, the magnitude of the frequency characteristic signal 136 is (128, 4), which means that the frequency characteristic signal 136 has 128 frames of 4 channels.

[0153] The first convolutional layer 1010 processes the frequency feature signal 136 through filters to obtain a feature signal 1015 of size (128, 4, a), with each filter having a size of 3×1.

[0154] The second convolutional layer 1020 processes the input signal through b filters to obtain a feature signal 1025 of size (128, 4, b), with each filter having a size of 3×1.

[0155] The third convolutional layer 1030 processes the input signal through 128 filters to obtain a feature signal 1035 of size (128, 4, 128), with each filter having a size of 3×1.

[0156] The shaping layer 1040 obtains a weight signal 137 of size (16384, 4) by changing the position of samples in the feature signal 1035 of size (128, 4, 128). For example, the shaping layer 1040 can obtain a weight signal 137 of size (16384, 4) by moving samples from the second frequency band to the 128th frequency band on the time axis from the samples in the feature signal 1035 of size (128, 4, 128).

[0157] According to the embodiment, the third DNN 1000 obtains a weight signal 137, which has the same time length and channels as the intermediate audio signal output from the AI ​​amplifier 914. Therefore, assuming the third DNN 1000 can output such a weight signal 137, then the third DNN 1000 can have, in addition to... Figure 10 Various structures other than the one shown. In other words, although Figure 10 The example shown depicts a third DNN 1000 comprising three convolutional layers and one shaping layer. However, this is merely an example; therefore, the number of convolutional and shaping layers included in the third DNN 1000 can vary, as long as a weight signal 137 with the same time length and number of channels as the intermediate audio signal can be obtained. Similarly, the shaping layer can be replaced with convolutional layers, and the number and size of filters used in each convolutional layer can differ.

[0158] In the above scenario, the first DNN 400 obtains the frequency feature signal 109 relative to the frequency domain first audio signal 107 transformed from the first audio signal 105, while the weight signal acquirer 912 does not inversely transform the frequency feature signal 136 or the weight signal 137 to the time domain. This is to prevent latency caused by inverse transformation in the server-client architecture. In other words, for the rapid content consumption of client terminals that receive audio signals from the server in a streaming manner, the latency caused by inverse transformation is eliminated.

[0159] Next, we will now refer to Figure 11 Describe the fourth DNN of AI amplifier 914.

[0160] Figure 11 A fourth DNN 1100 according to an embodiment is shown.

[0161] Reference Figure 11 The fourth DNN 1100 may include at least one convolutional layer and at least one shaping layer. The convolutional layer included in the fourth DNN 1100 may be a one-dimensional convolutional layer.

[0162] The third audio signal 135 is input to the fourth DNN 1100 and amplified into an intermediate audio signal 138 through the processing AI in the fourth DNN 1100.

[0163] like Figure 11 As shown, the size of the third audio signal 135 is (16384, 2), which means that the third audio signal 135 has 16384 frames with 2 channels.

[0164] The first convolutional layer 1110 processes the third input signal 135 through a filters to obtain a feature signal 1115 of size (4096, a), with each filter having a size of 33.

[0165] The second convolutional layer 1120 processes the input signal through b filters to obtain an integrated feature signal 1128 of size (128, b), where each filter has a size of 33. During the training process described below, the fourth DNN 1100 can be trained to output the integrated feature signal 1128 through the second convolutional layer 1120, which is the same as / similar to the integrated feature signal 628 obtained by the second DNN 600 during the processing of the first audio signal 105.

[0166] When the frequency feature signal 136 is embedded in the third audio signal 135, the frequency feature signal 136 is extracted from the integrated feature signal 1128. More specifically, a predetermined number of samples from consecutive channels starting from the first channel or a predetermined number of samples from consecutive channels starting from the last channel can be extracted from the channels of the integrated feature signal 1128 as the frequency feature signal 136. As described above, the frequency feature signal 136 is sent to the weighted signal acquisition unit 912.

[0167] The third convolutional layer 1130 processes the input signal (e.g., the audio feature signal 1125 separated from the integrated feature signal 1128) through c filters to obtain a feature signal 1135 of size (256, c), with each filter having a size of 33.

[0168] The shaping layer outputs an intermediate audio signal 138 of size (16384, 4) by changing the position of the sample in the feature signal 1135 of size (256, c).

[0169] According to the embodiment, the fourth DNN 1100 obtains an intermediate audio signal 138 having the same duration and number of channels as the first audio signal 105. Therefore, assuming the fourth DNN 1100 can output such an intermediate audio signal 138, then the fourth DNN 1100 can have, in addition to... Figure 11 Various structures other than the one shown. In other words, although Figure 11The fourth DNN 1100 is shown to include three convolutional layers and one shaping layer, but this is merely an example. Therefore, the number of convolutional and shaping layers included in the fourth DNN 1100 can vary, as long as an intermediate audio signal 138 with the same duration and number of channels as the first audio signal 105 can be obtained. Similarly, the shaping layer can be replaced with convolutional layers, and the number and size of filters used in each convolutional layer can be different.

[0170] Reference Figure 9 The weighted signal output by the weighted signal acquirer 912 and the intermediate audio signal output by the AI ​​reducer 914 can be input to the combiner 916, and the combiner 916 can obtain the fourth audio signal 145 by applying samples of the weighted signal to samples of the intermediate audio signal. For example, the combiner 916 can obtain the fourth audio signal 145 by multiplying the sample values ​​of the intermediate audio signal by the corresponding sample values ​​of the weighted signal in a 1:1 ratio.

[0171] and Figure 9 Unlike the decoding device 900 shown, a conventional decoding device that cannot perform AI decoding can obtain a third audio signal 135 from the first decoded audio data. The conventional decoding device can then reproduce the third audio signal 135 by outputting it through a speaker. In other words, according to the embodiment, the audio data obtained as a result of the first encoding relative to the second audio signal 115 may have lower compatibility, applicable to both the decoding device 900 capable of performing AI decoding and the conventional decoding device that cannot.

[0172] In the following text, refer to Figures 12 to 17 The method for training the first DNN 400, the second DNN 600, the third DNN 1000, and the fourth DNN 1100 will now be described.

[0173] Figure 12 The method for training the first DNN 400, the second DNN 600, the third DNN 1000, and the fourth DNN 1100 is shown.

[0174] Figure 12 A method for training a second DNN 600 to embed frequency feature signals in a second audio signal 115 is shown.

[0175] Figure 12 The first training signal 1201 corresponds to the first audio signal 105, and the second training signal 1205 corresponds to the second audio signal 115. Furthermore, the third training signal 1206 corresponds to the third audio signal 135, and the fourth training signal 1210 corresponds to the fourth audio signal 145.

[0176] Frequency domain training signal 1202 is obtained by performing frequency transformation (1220) on the first training signal 1201, and the frequency domain training signal 1202 is input to the first DNN 400. The first DNN 400 processes the frequency domain training signal 1202 according to preset parameters to obtain a frequency feature signal 1203 for training. The frequency feature signal 1203 and the first training signal 1201 are input to the second DNN 600, and the second DNN 600 obtains a second training signal 1205 embedded with the frequency feature signal 1203 for training according to preset parameters.

[0177] Through first encoding and first decoding (1250), the second training signal 1205 is transformed into a third training signal 1206. More specifically, the audio data used for training is obtained through first encoding relative to the second training signal 1205, while the third training signal 1206 is obtained through first decoding relative to the audio data used for training. The third training signal 1206 is input to the fourth DNN 1100. The fourth DNN 1100 obtains a frequency feature signal 1207 and an intermediate audio signal 1209 for training from the third training signal 1206 using preset parameters. The third DNN 1000 processes the frequency feature signal 1207 for training using preset parameters to obtain a weight signal 1208 for training. The fourth training signal 1210 is obtained by combining the weight signal 1208 for training with the intermediate audio signal 1209 for training.

[0178] exist Figure 12 In this process, the frequency feature signal 1203 obtained by the first DNN 400 for training is input into the training DNN 1240, which is used to verify whether the first DNN 400 accurately generated the frequency feature signal 1203 for training. The training DNN 1240 may have a mirror structure of the first DNN 400. The training DNN 1240 reconstructs the frequency domain training signal 1204 by processing the training frequency feature signal 1203.

[0179] The generated loss information (LossDG) 1260 is a comparison between the frequency domain training signal (1202) obtained by frequency transformation (1220) and the frequency domain training signal 1204 obtained by the DNN 1240 used for training. The generated loss information (LossDG) 1260 may include at least one of the following: L1 norm value, L2 norm value, structural similarity (SSIM) value, peak signal-to-noise ratio-human visual system (PSNR-HVS) value, multi-scale SSIM (MS-SSIM) value, variance inflation factor (VIF) value, and video multi-method evaluation fusion (VMAF) value between the frequency domain training signal 1202 obtained by frequency transformation (1220) and the frequency domain training signal 1204 obtained by the DNN used for training. For example, the generated loss information 1260 can be expressed as Equation 1 below.

[0180] Formula 1

[0181] LossDG=||F(A nch )-D(C Emmbed )||2 2

[0182] In Equation 1, F() indicates frequency transformation (1220), A nch Indicates the first training signal 1201. D() indicates the DNN used for training 1240, C Embed The frequency characteristic signal 1203 is used for training.

[0183] The generated loss information 1260 indicates the degree of similarity between the frequency domain training signal 1204 obtained by processing the frequency feature signal 1203 used for training through the DNN 1240 used for training and the frequency domain training signal 1202 obtained by frequency transformation (1220).

[0184] The first training signal 1201 is transformed into a few-channel training signal through conventional reduction 1230, and the reduction loss information 1270 is obtained as a result of comparing the few-channel training signal and the second training signal 1205. The reduction loss information 1270 may include at least one of the following: L1 norm value, L2 norm value, SSIM value, PSNR-HVS value, MS-SSIM value, VIF value, and VMAF value between the few-channel training signal and the second training signal 1205. For example, the reduction loss information 1270 can be expressed as Equation 2 below.

[0185] Formula 2

[0186] LossDown=(1-β)·||S mch -S Label ||2 2+β-||F(S mch )-F(S label )||2 2

[0187] In Formula 2, β is the predetermined weight, and S mch It is the second training signal 1205, and S Label Indicates a few-channel training signal. F() indicates frequency shift.

[0188] The reduction loss information 1270 indicates the degree of similarity between the second training signal 1205, embedded with the frequency feature signal 1203 used for training, and the few-channel training signal obtained through conventional reduction 1230. Since the second training signal 1205 is more similar to the few-channel training signal, the quality of the third training signal 1206 can be improved. In particular, the quality of the signal reconstructed by a conventional decoding device can be improved.

[0189] Based on the comparison result between the first training signal 1201 and the fourth training signal 1210, amplification loss information (LossUp) 1280 is obtained. The amplification loss information (LossUp) 1280 may include at least one of the following: L1 norm value, L2 norm value, SSIM value, PSNR-HVS value, MS-SSIM value, VIF value, and VMAF value between the first training signal 1201 and the fourth training signal 1210. For example, the amplification loss information 1280 can be expressed as Equation 3 below.

[0190] Formula 3

[0191] LossUp=(1-β)·||A Pnch -A nch ||2 2 +β·||F(A pneh )-F(A nch )||2 2

[0192] In Formula 3, β is the predetermined weight, and A nch Indicates the first training signal 1201, A pnch Indicates the fourth training signal 1210. F() indicates frequency conversion.

[0193] The amplified loss information 1280 indicates the accuracy of generating the weight signal 1208 and the intermediate audio signal 1209 for training.

[0194] Based on the comparison result between the frequency feature signal 1203 output by the first DNN 400 and the frequency feature signal 1207 extracted by the fourth DNN 1100 for training, matching loss information (LossM) 1290 is obtained. Matching loss information (LossM) 1290 may include at least one of the following: L1 norm value, L2 norm value, SSIM value, PSNR-HVS value, MS-SSIM value, VIF value, and VMAF value between the two frequency feature signals 1203 and 1207 used for training. For example, matching loss information 1290 can be expressed as Equation 4 below.

[0195] Formula 4

[0196] LossM=||C Embed -C Extract ||2 2

[0197] In formula 4, C Embed Indicates the frequency feature signal 1203, C embedded in the second training signal 1205 for training. Extract The frequency feature signal 1207 extracted by the fourth DNN 1100 for training is indicated.

[0198] The matching loss information 1290 indicates the degree of similarity between the integrated feature signal output from the fourth DNN 1100 and the integrated feature signal obtained from the second DNN 600. When the integrated feature signal output from the fourth DNN 1100 is similar to the integrated feature signal obtained from the second DNN 600, their two frequency feature signals are also similar.

[0199] The first DNN 400, the second DNN 600, the third DNN 1000, and the fourth DNN 1100 can update parameters to reduce or minimize the final loss information obtained by combining at least one of the loss information generated by combining loss information 1260, reducing loss information 1270, amplifying loss information 1280, and matching loss information 1290.

[0200] In detail, the first DNN 400 and the DNN 1240 used for training can update their parameters to reduce or minimize the generated loss information 1260. Furthermore, the second DNN 600, the third DNN 1000, and the fourth DNN 1100 can each update their parameters to reduce or minimize the final loss information obtained as a combination of reduced loss information 1270, amplified loss information 1280, and matched loss information 1290.

[0201] The training formulas for the first DNN 400 and the DNN 1240 used for training are as follows.

[0202] Formula 5

[0203]

[0204] In Formula 5, ω Phase1 Indicates the parameter sets of the first DNN 400 and the DNN 1240 used for training. The first DNN 400 and the DNN 1240 used for training obtain the parameter sets through training to minimize the generation loss information (LossDG) 1260.

[0205] The training formulas for the second DNN 600, the third DNN 1000, and the fourth DNN 1100 are as follows.

[0206] Formula 6

[0207]

[0208] In formula 6, ω Phase2 The parameter sets of the second DNN 600, third DNN 1000, and fourth DNN 1100 are indicated, with α and γ indicating preset weights. The second DNN 600, third DNN 1000, and fourth DNN 1100 obtain parameter sets through training to minimize the final loss information, which is a combination of reduced loss information (LossDown) 1270, amplified loss information (LossUp) 1280, and matched loss information (LossM) 1290 according to preset weights.

[0209] In this embodiment, training of the first DNN 400 and the training DNN 1240, as well as training of the second DNN 600, the third DNN 1000, and the fourth DNN 1100, can be performed alternately. More specifically, the first DNN 400 and the training DNN 1240 process the input signal according to initially set parameters, and then update the parameters according to generated loss information 1260. Then, the first DNN 400 and the training DNN 1240 process the input signal according to the updated parameters, while the second DNN 600, the third DNN 1000, and the fourth DNN 1100 process the input signal according to initially set parameters. The second DNN 600, the third DNN 1000, and the fourth DNN 1100 each update their parameters according to at least one of matching loss information 1290, amplified loss information 1280, and reduced loss information 1270 obtained as a result of processing the loss input signal. When the parameters of the second DNN 600, the third DNN 1000, and the fourth DNN 1100 are updated, the parameters of the first DNN 400 and the training DNN 1240 are updated again. That is, according to the embodiment, the training of the first DNN 400 and the training DNN 1240 is performed alternately with the training of the second DNN 600, the third DNN 1000, and the fourth DNN 1100, so that the parameters of each DNN can be stably trained to a higher level of accuracy.

[0210] Figure 13 and Figure 14 A flowchart is shown to describe the process of training a first DNN 400, a second DNN 600, a third DNN 1000, and a fourth DNN 1100 by the training device 1300.

[0211] Reference Figure 12 The training of the described first DNN 400, DNN 1240 for training, second DNN 600, third DNN 1000, and fourth DNN 1100 can be performed by training device 1300. Training device 1300 may include the first DNN 400, DNN 1240 for training, second DNN 600, third DNN 1000, and fourth DNN 1100. For example, training device 1300 may be audio encoding device 200 or a separate server. The third DNN 1000 and fourth DNN 1100 obtained as a result of training can be stored in audio decoding device 900.

[0212] The training device 1300 initially sets the parameters of the first DNN 400, the DNN 1240 used for training, the second DNN 600, the third DNN 1000 and the fourth DNN 1100 (S1310).

[0213] The training device 1300 inputs the frequency domain training signal 1202 obtained from the first training signal 1201 through frequency transformation (1220) to the first DNN 400 (S1320). The first DNN 400 outputs the frequency feature signal 1203 used for training to the DNN 1240 used for training (S1330), and the DNN 1240 used for training outputs the reconstructed frequency domain training signal 1204 to the training device 1300 (S1340).

[0214] The training device 1300 compares the frequency domain training signal 1202 obtained through frequency transformation (1220) with the frequency domain training signal 1204 output from the DNN 1240 used for training, thereby calculating the generated loss information 1260 (S1350). Then, the first DNN 400 and the DNN 1240 used for training each update their parameters according to the generated loss information 1260 (S1360 and S1370).

[0215] The training device 1300 inputs the frequency domain training signal 1202 obtained from the first training signal 1201 through frequency transformation (1220) back to the first DNN 400 (S1380). The first DNN 400 processes the frequency domain training signal 1202 with updated parameters, thereby outputting the frequency feature signal 1203 used for training to the training device 1300 and the second DNN 600 (S1390).

[0216] Next, in Figure 14 In the process, the training device 1300 inputs the first training signal 1201 to the second DNN 600 (S1410), and the second DNN 600 outputs the second training signal 1205 to the training device 1300 by processing the frequency feature signal 1203 used for training and the first training signal 1201 (S1420).

[0217] The training device 1300 obtains reduction loss information 1270 (S1430) based on the comparison between the second training signal 1205 and the few-channel training signal conventionally reduced (1230) from the first training signal 1201.

[0218] The training device 1300 inputs the third training signal 1206 obtained by the first encoding and first decoding (1250) of the second training signal 1205 into the fourth DNN 1100 (S1440), and the fourth DNN 1100 outputs the frequency feature signal 1207 used for training to the third DNN 1000 and the training device 1300 (S1450).

[0219] The training device 1300 compares the frequency feature signal 1203 for training output by the first DNN 400 with the frequency feature signal 1207 for training output by the fourth DNN 1100 in operation S1390, thereby calculating the matching loss information 1290 (S1460).

[0220] The fourth DNN 1100 processes the third training signal 1206 and outputs an intermediate audio signal 1209 for training (S1470), and the third DNN 1000 processes the frequency feature signal 1207 for training and outputs a weight signal 1208 for training (S1480).

[0221] The training device 1300 obtains a fourth training signal 1210 by combining an intermediate audio signal 1209 used for training and a weight signal 1208 used for training, and obtains amplification loss information 1280 by comparing the first training signal 1201 and the fourth training signal 1210 (S1490).

[0222] The second DNN 600, the third DNN 1000, and the fourth DNN 1100 update parameters (S1492, S1494, and S1496) based on the final loss information obtained by combining at least one of the reduced loss information 1270, amplified loss information 1280, and matched loss information 1290.

[0223] The training device 1300 can repeat operations S1320 to S1496 until the parameters of the first DNN 400, the DNN 1240 used for training, the second DNN 600, the third DNN 1000 and the fourth DNN 1100 are optimized.

[0224] Figures 12 to 14 The training process for embedding frequency feature signals into the second audio signal 115 is shown, and will now be referred to... Figures 15 to 17 The training process describes the situation where the frequency characteristic signal is not embedded in the second audio signal 115.

[0225] Figure 15 Another method is shown for training the first DNN 400, the second DNN 600, the third DNN 1000, and the fourth DNN 1100.

[0226] Figure 15 The first training signal 1501 corresponds to the first audio signal 105, and the second training signal 1505 corresponds to the second audio signal 115. Furthermore, the third training signal 1506 corresponds to the third audio signal 135, and the fourth training signal 1510 corresponds to the fourth audio signal 145.

[0227] Frequency domain training signal 1502 is obtained by performing frequency transformation (1520) on the first training signal 1501, and the frequency domain training signal 1502 is input into the first DNN 400. The first DNN 400 processes the frequency domain training signal 1502 according to preset parameters to obtain the frequency feature signal 1503 for training.

[0228] The first training signal 1501 is input into the second DNN 600, and the second DNN 600 obtains the second training signal 1505 through preset parameters.

[0229] The frequency feature signal 1503 and the second training signal 1505 used for training are processed by a first encoding and a first decoding (1550). More specifically, the audio data used for training is obtained by first encoding relative to the frequency feature signal 1503 and the second training signal 1505, while the third training signal 1506 and the frequency feature signal 1507 used for training are obtained by first decoding relative to the audio data used for training. The frequency feature signal 1507 used for training is input to the third DNN 1000, and the third training signal 1506 is input to the fourth DNN 1100. The third DNN 1000 processes the frequency feature signal 1507 used for training with preset parameters to obtain the weight signal 1508 used for training.

[0230] The fourth DNN 1100 obtains the intermediate audio signal 1509 for training from the third training signal 1506 using preset parameters. The fourth training signal 1510 is obtained by combining the weight signal 1508 for training with the intermediate audio signal 1509 for training.

[0231] exist Figure 15 In this process, the frequency feature signal 1503 obtained by the first DNN 400 for training is input into the training DNN 1540, and the training DNN 1540 is used to verify whether the first DNN 400 accurately generated the frequency feature signal 1503 for training. The training DNN 1540 may have a mirror structure of the first DNN 400. The training DNN 1540 reconstructs the frequency domain training signal 1504 by processing the training frequency feature signal 1503.

[0232] The generated loss information (LossDG) 1560 is obtained as a comparison result between the frequency domain training signal 1502 obtained through frequency transformation (1520) and the frequency domain training signal 1504 obtained through the DNN 1540 used for training. The generated loss information (LossDG) 1560 may include at least one of the following: L1 norm value, L2 norm value, SSIM value, PSNR-HVS value, MS-SSIM value, VIF value, and VMAF value between the frequency domain training signal 1502 obtained through frequency transformation (1520) and the frequency domain training signal 1504 obtained through the DNN 1540 used for training. For example, the generated loss information 1560 can be expressed as Equation 1 above.

[0233] The first training signal 1501 is transformed into a few-channel training signal through conventional reduction 1530, and the reduction loss information 1570 is obtained as a comparison result between the few-channel training signal and the second training signal 1505. The reduction loss information 1570 may include at least one of the following: L1 norm value, L2 norm value, SSIM value, PSNR-HVS value, MS-SSIM value, VIF value, and VMAF value between the few-channel training signal and the second training signal 1505. For example, the reduction loss information 1570 can be expressed as Equation 2 above.

[0234] Based on the comparison result between the first training signal 1501 and the fourth training signal 1510, amplification loss information (LossUp) 1580 is obtained. The amplification loss information (LossUp) 1580 may include at least one of the following: L1 norm value, L2 norm value, SSIM value, PSNR-HVS value, MS-SSIM value, VIF value, and VMAF value between the first training signal 1501 and the fourth training signal 1510. For example, the amplification loss information 1580 can be expressed as in Equation 3 above.

[0235] and Figure 12 Compared to the training process described in [the text], in [the text] Figure 15 During training, no matching loss information (LossM) 1290 was obtained. This is because, in Figure 15 During the training process, the frequency feature signal 1503 used for training is not embedded in the second training signal 1505, and the frequency feature signal 1507 used for training obtained through the first decoding is the same as the frequency feature signal 1503 used for training obtained through the first DNN 400.

[0236] The first DNN 400, the second DNN 600, the third DNN 1000, and the fourth DNN 1100 can each update their parameters to reduce or minimize the final loss information obtained by combining and generating loss information 1560, reducing loss information 1570, and amplifying loss information 1580.

[0237] In detail, the first DNN 400 and the DNN 1540 used for training can update their parameters to reduce or minimize the generated loss information 1560. Furthermore, the second DNN 600, the third DNN 1000, and the fourth DNN 1100 can each update their parameters to reduce or minimize the final loss information obtained as a result of the combination of reduced loss information 1570 and amplified loss information 1580.

[0238] The training of the first DNN 400 and the DNN 1540 used for training can be expressed as Equation 5 above, and the training of the second DNN 600, the third DNN 1000 and the fourth DNN 1100 can be expressed as Equation 7 below.

[0239] Formula 7

[0240]

[0241] In equation 7, ω Phase2 The parameter set of the second DNN 600, third DNN 1000, and fourth DNN 1100 is indicated, and α indicates the preset weights. The second DNN 600, third DNN 1000, and fourth DNN 1100 obtain the final loss information obtained by training the parameter set to minimize the combination of loss information reduction (LossDown) 1570 and loss information amplification (LossUp) 1580.

[0242] In this embodiment, training of the first DNN 400 and the training DNN 1540, as well as training of the second DNN 600, the third DNN 1000, and the fourth DNN 1100, can be performed alternately. More specifically, the first DNN 400 and the training DNN 1540 process the input signal according to initially set parameters, and then update the parameters based on generated loss information 1560. Then, the first DNN 400 and the training DNN 1540 process the input signal according to the updated parameters, while the second DNN 600, the third DNN 1000, and the fourth DNN 1100 process the input signal according to initially set parameters. Each of the second DNN 600, the third DNN 1000, and the fourth DNN 1100 updates its parameters based on at least one of amplified loss information 1580 and reduced loss information 1570 obtained as a result of processing the input signal. When the second DNN 600, the third DNN 1000, and the fourth DNN 1100 have finished updating their parameters, the first DNN 400 and the DNN 1540 used for training update their parameters again.

[0243] Figure 16 and Figure 17 A flowchart is shown to describe the process of training a first DNN 400, a second DNN 600, a third DNN 1000, and a fourth DNN 1100 by the training device 1300.

[0244] Reference Figure 15 The training of the described first DNN 400, DNN 1240 for training, second DNN 600, third DNN 1000, and fourth DNN 1100 can be performed by training device 1300. Training device 1300 may include the first DNN 400, DNN 1240 for training, second DNN 600, third DNN 1000, and fourth DNN 1100. For example, training device 1300 may be audio encoding device 200 or a separate server. The third DNN 1000 and fourth DNN 1100 obtained as a result of training can be stored in audio decoding device 900.

[0245] Reference Figure 16 The training device 1300 initially sets the parameters of the first DNN 400, the DNN 1240 used for training, the second DNN 600, the third DNN 1000 and the fourth DNN 1100 (S1610).

[0246] The training device 1300 inputs the frequency domain training signal 1502 obtained from the first training signal 1501 through frequency transformation (1520) to the first DNN 400 (S1620). The first DNN 400 outputs the frequency feature signal 1503 used for training to the DNN 1540 used for training (S1630), and the DNN 1540 used for training outputs the reconstructed frequency domain training signal 1504 to the training device 1300 (S1640).

[0247] The training device 1300 compares the frequency domain training signal 1502 obtained through frequency transformation (1520) with the frequency domain training signal 1504 output from the DNN 1540 used for training, thereby calculating the generated loss information 1560 (S1650). Then, the first DNN 400 and the DNN 1540 used for training each update their parameters according to the generated loss information 1560 (S1660 and S1670).

[0248] The training device 1300 inputs a frequency domain training signal 1502, obtained from the first training signal 1501 through frequency transformation (1520), back to the first DNN 400 (S1680). The first DNN 400 processes the frequency domain training signal 1502 with updated parameters, thereby outputting a frequency feature signal 1503 for training back to the training device 1300 (S1690). Figure 13 In contrast, the frequency feature signal 1503 used for training is not embedded in the second training signal 1505, therefore, in Figure 16 In this process, the frequency feature signal 1503 used for training is not input into the second DNN 600.

[0249] Next, in Figure 17 In the process, the training device 1300 inputs the first training signal 1501 to the second DNN 600 (S1710), and the second DNN 600 processes the first training signal 1501 and outputs the second training signal 1505 to the training device 1300 (S1720).

[0250] The training device 1300 obtains reduction loss information 1570 (S1730) based on the comparison between the second training signal 1505 and the reduced few-channel training signal (1530) conventionally reduced from the first training signal 1501.

[0251] The training device 1300 inputs the third training signal 1506 obtained through the first encoding and the first decoding and the frequency feature signal 1507 for training into the fourth DNN 1100 and the third DNN 1000 respectively (S1740 and S1750). The fourth DNN 1100 outputs an intermediate audio signal 1509 for training by processing the third training signal 1506 (S1760), and the third DNN 1000 outputs a weight signal 1508 for training by processing the frequency feature signal 1507 for training (S1770).

[0252] The training device 1300 obtains a fourth training signal 1510 by combining the intermediate audio signal 1509 for training and the weight signal 1508 for training, and obtains amplification loss information 1580 by comparing the first training signal 1501 and the fourth training signal 1510 (S1780).

[0253] The second DNN 600, the third DNN 1000, and the fourth DNN 1100 update parameters according to the final loss information obtained by combining at least one of the reduction loss information 1570 and the amplification loss information 1580 (S1792, S1794, and S1796).

[0254] The training device 1300 may repeat operations S1620 to S1796 until the parameters of the first DNN 400, the DNN 1540 for training, the second DNN 600, the third DNN 1000, and the fourth DNN 1100 are optimized.

[0255] Figure 18 A flowchart for describing an audio encoding method according to an embodiment is shown.

[0256] In S1810, the encoding device 200 transforms the first audio signal 105 including n channels from the time domain to the frequency domain. As a result of the transformation, the first audio signal in the frequency domain may have n channels.

[0257] In S1820, the encoding device 200 processes the first audio signal in the frequency domain through the first DNN 400 to obtain a frequency feature signal, and the number of samples per channel of the frequency feature signal within a predetermined time period is less than the number of samples per channel of the first audio signal in the frequency domain.

[0258] In S1830, the encoding device 200 obtains a second audio signal 115 including m channels (where m < n) from the first audio signal 105 by using the second DNN 600. The time length of the second audio signal 115 may be equal to the time length of the first audio signal 105, and the number of channels of the second audio signal 115 may be less than the number of channels of the first audio signal 105.

[0259] In S1840, the encoding device 200 obtains audio data by performing a first encoding on the second audio signal 115 and the frequency feature signal. As described above, the frequency feature signal can be embedded in the second audio signal 115 and then the first encoding can be performed, or each of the second audio signal 115 and the frequency feature signal can be first encoded and then included in the audio data.

[0260] Figure 19 A flowchart is shown to describe an audio decoding method according to an embodiment.

[0261] In S1910, the decoding device 900 obtains a third audio signal 135 including m channels and a frequency feature signal by performing a first decoding on the audio data. The fourth DNN 1100 can extract the frequency feature signal relative to the third audio signal 135 during processing.

[0262] In operation S1920, the decoding device 900 obtains a weighted signal from the frequency feature signal using a third DNN 1000. The duration and number of channels of the weighted signal can be equal to the duration and number of channels of the first audio signal 105 and the fourth audio signal 145.

[0263] In S1930, the decoding device 900 obtains an intermediate audio signal comprising n channels from the third audio signal 135 using a fourth DNN 1100. The duration and number of channels of the intermediate audio signal can be equal to the duration and number of channels of the first audio signal 105 and the fourth audio signal 145.

[0264] In operation S1940, the decoding device 900 obtains a fourth audio signal 145 comprising n channels by applying a weighting signal to the intermediate audio signal.

[0265] The fourth audio signal 145 can be output to a reproduction device (e.g., a loudspeaker) for reproduction.

[0266] The embodiments described above can be written as computer executable programs that can be stored in a medium.

[0267] The medium can continuously store computer-executable programs or temporarily store them for execution or download. Furthermore, the medium can be any of various recording or storage media, in which single or multiple hardware components are combined, and the medium is not limited to those directly connected to a computer system but can be distributed across a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as optical discs; and read-only memory (ROM), random access memory (RAM), and flash memory, which are configured to store program instructions. Other examples of media include recording and storage media managed by application stores that distribute applications or by websites, servers, etc., that provide or distribute various other types of software.

[0268] While the technical concepts of this disclosure have been described with reference to exemplary embodiments, this disclosure is not limited to these embodiments, and various modifications and changes can be made by those skilled in the art without departing from the technical concepts of this disclosure.

Claims

1. An audio signal processing device, comprising: a memory that stores one or more instructions; and a processor configured to execute the one or more instructions stored in the memory, wherein the processor is configured to: perform a frequency transformation on a first audio signal including n channels to generate a first audio signal in the frequency domain, generate a frequency feature signal for each channel from the first audio signal in the frequency domain based on a first deep neural network (DNN), wherein the frequency feature signal represents the correlation between the n channels of the first audio signal detected from the first audio signal in the frequency domain and has a smaller number of samples than the first audio signal in the frequency domain, generate a second audio signal including m channels from the first audio signal based on a second DNN, where m < n, and generate an output audio signal by encoding the second audio signal and the frequency feature signal, where the first audio signal is a high-order surround sound signal including a zero-order signal and multiple first-order signals, and the second audio signal includes one of a mono signal and a stereo signal.

2. The audio signal processing device according to claim 1, wherein the frequency feature signal includes a representative value for each channel, and the representative value for each channel is a value of multiple frequency bands corresponding to each channel of the first audio signal in the frequency domain.

3. The audio signal processing device according to claim 1, wherein the second DNN obtains an audio feature signal from the first audio signal and outputs the second audio signal from an integrated feature signal in which the audio feature signal and the frequency feature signal are combined.

4. The audio signal processing device according to claim 3, wherein the integrated feature signal is obtained by replacing samples of some channels in the channels of the audio feature signal with samples of the frequency feature signal.

5. The audio signal processing device according to claim 4, wherein the some channels include a predetermined number of consecutive channels starting from the first channel in the channels of the audio feature signal or a predetermined number of consecutive channels starting from the last channel in the channels of the audio feature signal.

6. The audio signal processing device according to claim 3, wherein the time length of the audio feature signal is equal to the time length of the frequency feature signal.

7. The audio signal processing device according to claim 1, wherein in the frequency feature signal, the number of samples for each channel during a predetermined time period is 1.

8. The audio signal processing device according to claim 1, wherein the output audio signal is represented as a bitstream, and the frequency feature signal is included in a supplementary area of the bitstream.

9. The audio signal processing device according to claim 1, wherein the processor is configured to obtain the second audio signal by combining an intermediate audio signal output from the second DNN with a few-channel audio signal reduced from the first audio signal.

10. The audio signal processing device according to claim 1, wherein the first DNN is trained based on the result of comparing a frequency domain training signal transformed from a first training signal with a frequency domain training signal reconstructed from a frequency feature signal for training by a DNN for training, and The frequency feature signal used for training is obtained from the frequency domain training signal by the first DNN.

11. The audio signal processing apparatus according to claim 10, wherein... The second DNN is trained based on at least one of the following results: a comparison between a second training signal obtained from the first training signal via the second DNN and a reduced few-channel training signal from the first training signal. The result of comparing the first training signal with the fourth training signal reconstructed from the audio data used for training, and The result is a comparison between the frequency feature signal used for training and the frequency feature signal used for training obtained from the audio data used for training.

12. The audio signal processing apparatus according to claim 11, wherein... The first DNN and the second DNN are trained alternately.

13. An audio signal processing apparatus, comprising: Memory, which stores one or more instructions; as well as A processor is configured to execute one or more instructions stored in memory. The processor is configured as follows: By decoding the input audio signal, a third audio signal and a frequency feature signal are generated. The third audio signal includes m channels corresponding to m channels of a second audio signal, which is generated from the first audio signal. The frequency feature signal represents the correlation between n channels of the first audio signal detected in the frequency domain and has fewer samples than the first audio signal in the frequency domain. Based on a third-generation deep neural network (DNN), a weighted signal comprising n channels is generated from the frequency feature signal, where n > m. By applying the weighted signal to an intermediate audio signal comprising n channels generated from the third audio signal via a fourth DNN, a fourth audio signal comprising n channels is generated. The third audio signal includes either a mono signal or a stereo signal, and The fourth audio signal is a high-order surround sound signal that includes a zero-order signal and multiple first-order signals.

14. The audio signal processing apparatus according to claim 13, wherein... The fourth DNN obtains an integrated feature signal by processing the third audio signal, and outputs an intermediate audio signal from the audio feature signals included in the integrated feature signal. The frequency feature signal is extracted from the integrated feature signal and then input into the third DNN.

15. The audio signal processing apparatus according to claim 13, wherein... The processor is configured to obtain a fourth audio signal by multiplying samples of the intermediate audio signal with samples of the weighted signal.

Images