Decoding device, decoding method, program, and encoding device

Abstract

The present disclosure pertains to a decoding device, a decoding method, a program, and an encoding device which make it possible to maintain favorable sound quality even at a low bit rate. A demultiplexing unit of the present disclosure separates first encoded data in which an envelope component of a content signal is encoded by frequency conversion and second encoded data in which a flattened waveform of the content signal is encoded at a different bit number than is the envelope component from an encoded bitstream of the content signal for each band. A band decoding unit of the present disclosure generates a band division signal by synthesizing the envelope component obtained by decoding the first encoded data and the flattened waveform obtained by decoding the second encoded data. A band synthesis unit of the present disclosure generates a content signal by synthesizing the band division signal for each band. The technology according to the present disclosure can be applied to, for example, a voice signal transmission system that transmits an encoded bitstream.

Description

Decoding device, decoding method, program, and encoding device

[0001] The present disclosure relates to a decoding device, a decoding method, a program, and an encoding device, and more particularly to a decoding device, a decoding method, a program, and an encoding device that are capable of maintaining good sound quality even at a low bit rate.

[0002] In recent years, with the expansion of network transmission bandwidth, the number of remote live events has been increasing. A remote live performance is a live performance of entertainment venues such as music or theater, in which video and audio recordings of the performers alone or the performers and audience are broadcast in real time to an audience outside the venue (remote audience).

[0003] In streaming remote live performances, it is necessary to simultaneously transmit high-quality audio signals to the devices of many remote audience members without compromising the sense of presence at the live venue. However, conventional waveform coding methods have limitations in compression performance, making it difficult to significantly reduce bit rates while maintaining quality. On the other hand, parameter coding, which models and synthesizes audio as parameters, can significantly reduce bit rates, but it is limited to voice and suffers from quality degradation when using general audio signals, making it difficult to transmit and restore audio without compromising the sense of presence at the live venue.

[0004] Furthermore, in order to improve the coding efficiency, there is a technique of extracting and coding the envelope components of a signal waveform or a frequency spectrum. For example, Patent Document 1 discloses a technique of using a vector quantization codebook to code envelope information and outputting it as an index.

[0005] Japanese Patent Application Publication No. 9-146593

[0006] On the other hand, it has recently become clear that the envelope components of speech signals have a significant impact on auditory perception. However, with conventional coding methods, the envelope components tend to become distorted at low bit rates, which can lead to auditory problems.

[0007] The present disclosure has been made in view of such circumstances, and aims to make it possible to maintain good sound quality even at a low bit rate.

[0008] A decoding device according to a first aspect of the present disclosure is a decoding device including: a demultiplexing unit that separates, for each band, from an encoded bit stream of a content signal, first encoded data in which an envelope component of the content signal is encoded by frequency conversion, and second encoded data in which a flattened waveform of the content signal is encoded with a different number of bits than the envelope component; a band decoding unit that generates a band division signal by synthesizing the envelope component decoded from the first encoded data and the flattened waveform decoded from the second encoded data; and a band synthesis unit that generates the content signal by synthesizing the band division signals of each band.

[0009] A decoding method according to a first aspect of the present disclosure is a decoding method in which a decoding device separates, for each band, an encoded bit stream of a content signal into first encoded data in which an envelope component of the content signal is encoded by frequency conversion, and second encoded data in which a flattened waveform of the content signal is encoded with a different number of bits than the envelope component, generates a band-split signal by synthesizing the envelope component obtained by decoding the first encoded data and the flattened waveform obtained by decoding the second encoded data, and generates the content signal by synthesizing the band-split signals of each band.

[0010] A program according to a first aspect of the present disclosure is a program for causing a computer to execute a process of separating, for each band, an encoded bit stream of a content signal into first encoded data in which an envelope component of the content signal is encoded by frequency conversion and second encoded data in which a flattened waveform of the content signal is encoded with a different number of bits than the envelope component, generating a band-split signal by synthesizing the envelope component obtained by decoding the first encoded data and the flattened waveform obtained by decoding the second encoded data, and generating the content signal by synthesizing the band-split signals of each band.

[0011] A coding device according to a second aspect of the present disclosure is a coding device including: a band splitting unit that splits a content signal into band split signals for each band; a band coding unit that separates the band split signals into an envelope component and a flattened waveform, encodes the envelope component by frequency conversion, and encodes the flattened waveform with a different number of bits than the envelope component; and a multiplexing unit that generates a coded bit stream by multiplexing first coded data in which the envelope component is coded and second coded data in which the flattened waveform is coded.

[0012] In a first aspect of the present disclosure, first encoded data in which an envelope component of the content signal is encoded by frequency conversion and second encoded data in which a flattened waveform of the content signal is encoded with a different number of bits than the envelope component are separated for each band from the encoded bit stream of the content signal, the envelope component decoded from the first encoded data and the flattened waveform decoded from the second encoded data are synthesized to generate a band division signal, and the band division signals of each band are synthesized to generate the content signal.

[0013] In a second aspect of the present disclosure, a content signal is divided into band-divided signals for each band, the band-divided signals are separated into an envelope component and a flattened waveform, the envelope component is encoded by frequency conversion, and the flattened waveform is encoded with a different number of bits from the envelope component, and a first encoded data in which the envelope component is encoded and a second encoded data in which the flattened waveform is encoded are multiplexed to generate an encoded bit stream.

[0014] 1 is a diagram illustrating a relationship between an envelope component of an audio signal and timbre. FIG. 1 is a diagram illustrating an example configuration of an audio signal transmission system according to an embodiment of the present disclosure. FIG. 2 is a block diagram illustrating an example functional configuration of an encoding device. FIG. 3 is a diagram illustrating separation of an envelope component and a flattened waveform. FIG. 4 is a diagram illustrating examples of parameters according to the type of content. FIG. 5 is a diagram illustrating an example transmission format of an encoded bitstream. FIG. 6 is a flowchart illustrating a flow of encoding processing for an audio signal. FIG. 7 is a block diagram illustrating an example functional configuration of a decoding device. FIG. 8 is a block diagram illustrating an example configuration of a synthesis parameter determination unit. FIG. 9 is a diagram illustrating examples of parameters according to the type of content. FIG. 10 is a block diagram illustrating an example configuration of an envelope weighting unit. FIG. 11 is a diagram illustrating an example of weight information. FIG. 12 is a block diagram illustrating another example configuration of an envelope weighting unit. FIG. 13 is a diagram illustrating an example of parameter learning by a neural network. FIG. 14 is a flowchart illustrating a flow of decoding processing for an audio signal. FIG. 15 is a block diagram illustrating another example functional configuration of a decoding device. FIG. 16 is a block diagram illustrating another example configuration of an envelope processing unit. FIG. 17 is a block diagram illustrating another example configuration of a synthesis parameter determination unit. FIG. 18 is a block diagram illustrating yet another example configuration of a synthesis parameter determination unit. FIG. 19 is a block diagram illustrating yet another example functional configuration of a decoding device. FIG. 19 is a block diagram illustrating another example functional configuration of an encoding device. FIG. 19 is a diagram illustrating an example editing screen of a music production editing tool. FIG. 19 is a diagram illustrating examples of parameters according to the timbre. FIG. 19 is a block diagram illustrating an example configuration of computer hardware.

[0015] Modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described below in the following order.

[0016] 1. Prior art and its problems 2. Relationship between envelope components of audio signals and timbre 3. Overview of technology related to the present disclosure and audio signal transmission system 4. Configuration and operation of encoding device 5. Configuration and operation of decoding device 6. Modified example of decoding device 7. Modified example of encoding device and music production editing tool 8. Summary 9. Example of computer hardware configuration

[0017] 1. Prior Art and Its Issues (Prior Art) In recent years, with the expansion of network transmission bandwidth, the number of remote live events has been increasing. A remote live performance refers to the real-time distribution of video and audio recordings of a live performance by performers alone, or by performers and audience members, at a live entertainment venue such as a music or theater venue to an audience outside the venue (remote audience).

[0018] Among these, for example, a system is known that displays video that reflects the movements of other remote spectators, allowing remote spectators participating from outside the live venue to feel as if they are actually participating in the event and to feel a sense of unity with the performers and other spectators. Another system is in operation in which pre-selected remote spectators record video and audio using their own cameras and microphones, transmit the data in real time to the live venue, and display their facial expressions and movements on a display at the venue while outputting audio from speakers, allowing them to cheer on the performers from outside the venue.

[0019] When streaming remote live events, it is necessary to simultaneously transmit high-quality audio signals to the devices of a large number of remote audience members without compromising the sense of realism of the live venue. Furthermore, it is important to enable simultaneous, bidirectional transmission with low latency between a large number of users so that the reactions and cheers of the remote audience can be fed back to the live venue. In recent years, the transmission of audio data along with high-quality video data such as 4K video has necessitated the transmission of enormous amounts of data. However, even with the expansion of network bandwidths such as fiber optic lines and 5G, when thousands or tens of thousands of people participate simultaneously, network lines can become congested, resulting in insufficient bandwidth and connection difficulties, audio dropouts, significant delays, and extremely poor sound quality, significantly diminishing the sense of realism of the content.

[0020] In order to prevent such deterioration in transmission quality, it is necessary to significantly reduce the amount of information to be transmitted without compromising the inherent realism and quality of the content. To date, many research efforts have been conducted into audio encoding and compression technologies (codecs) aimed at reducing the amount of information to be transmitted. Recent representative audio codecs include, for example, MPEG4 HE-AAC (High-Efficiency Advanced Audio Coding) (ISO / IEC 14496-3) and 3GPP (registered trademark) EVS (3rd Generation Partnership Project Enhanced Voice Services).

[0021] (Issues with Conventional Coding Methods) However, conventional waveform coding methods have limitations in compression performance, making it difficult to significantly reduce the bit rate while maintaining quality. On the other hand, parameter coding, which models speech and synthesizes it as parameters, can significantly reduce the bit rate, but it is limited to voice and quality deteriorates with general acoustic signals, making it difficult to transmit and restore without losing the sense of realism of a live venue.

[0022] Furthermore, in order to improve the coding efficiency, there is a technique of extracting and coding the envelope components of a signal waveform or a frequency spectrum. For example, Patent Document 1 discloses a technique of using a vector quantization codebook to code envelope information and outputting it as an index.

[0023] In recent years, it has become clear that the envelope components of an audio signal have a significant impact on auditory perception. However, conventional coding methods cannot directly control the quality of the envelope components, and therefore the envelope components tend to become distorted at low bit rates, resulting in auditory problems.

[0024] In addition, conventional encoding methods were not intended to reproduce subjective realism or sound perception, which are heavily influenced by envelope components, and the decoded audio signals were unable to fully reproduce the realism that conveys the atmosphere of a live concert venue, etc. Furthermore, conventional technologies did not make it easy for distributors or viewers to freely change the tone to create various atmospheres to suit the environment during distribution or playback.

[0025] 2. Relationship between the envelope components of an audio signal and timbre There is research that shows that the envelope components of a waveform are important in reproducing an audio signal.

[0026] For example, "Masayuki Takada, Calculation Methods and Application Examples of Sound Quality Evaluation Indices, Journal of the Acoustical Society of Japan, 2019, Vol. 75, No. 10, pp. 582-589," discloses a method for calculating sound quality evaluation indices such as "roughness," which indicates the roughness of a sound, and "fluctuation strength," which indicates the sense of fluctuation, based on the fact that the intensity and frequency (modulation frequency) of envelope components (amplitude modulation components) strongly affect the subjective perception of sound, such as the roughness and sense of fluctuation of a sound. Furthermore, it discloses that roughness of a sound occurs at modulation frequencies of 15 to 300 Hz, becomes larger at 30 to 150 Hz, and is particularly maximized around 70 Hz, and that the sense of fluctuation (fluctuation strength) occurs at modulation frequencies of 20 Hz or less, and is particularly maximized at 4 to 5 Hz.

[0027] A model that explains the results of these psychophysical experiments is proposed in Dau, T et al. "Modeling auditory processing of amplitude modulation. I. Detection and masking with narrow-band carriers." The Journal of the Acoustical Society of America vol. 102,5 Pt 1 (1997): 2892-905. In this model, it is assumed that in the human auditory system, the envelope components (amplitude modulation components) of the waveform after band division by the "auditory filter bank" are further analyzed by the "modulation filter bank."

[0028] Thus, envelope components play an important role in reproducing audio signals. Therefore, by extracting and efficiently transmitting and reproducing the envelope components of audio signals, it may be possible to efficiently reproduce a sense of presence with less transmission information. Furthermore, depending on the content and type of sound, it may be possible to improve coding efficiency by encoding with high precision the components necessary to optimize the subjective perception of sound, and coarsely quantizing or deleting unimportant envelope components.

[0029] As described above, modulation frequency components (envelope components) play an important role in reproducing audio signals. For example, if unintended components occur around the modulation frequency of 30 to 150 Hz, which has a significant effect on "roughness," the sound will be perceived as harsh and rough.

[0030] Based on this knowledge, for example, S. van de Par, S. Disch, A. Niedermeier, and B. Edler, "Informed postprocessing for auditory roughness removal for low-bitrate audio coders," AES Convention Paper 10534, (2021 October) discloses a method for suppressing modulation frequency components that are not present in the original sound and arise during encoding by processing in the frequency spectrum domain (without extracting envelope components). As such, conventional technology only suppresses "unnecessary modulation frequency components" and does not intend to actively change the timbre by processing the modulation frequency components.

[0031] On the other hand, as shown in Figure 1, for example, by extracting envelope components by band division and envelope separation of the sound source signal x(t), and then processing some of them to perform envelope synthesis and band synthesis, it is possible to change the timbre to suit individual preferences and the type of sound source. For example, by processing the envelope components below 20 Hz, which affect "fluctuation intensity," the sense of fluctuation can be changed. Alternatively, by reducing the envelope components around 30 to 150 Hz, which affect "roughness," the roughness can be suppressed to create a more subdued sound that is easier to listen to, or by increasing the envelope components above 200 Hz, the sense of attack, sharpness, and even dryness can be emphasized.

[0032] In this way, in addition to extracting and efficiently transmitting the envelope components of the audio signal, processing some of the extracted envelope components depending on the content and type of sound makes it easy to freely change the subjective perception of the sound during distribution or playback.

[0033] 3. Overview of Technology According to the Present Disclosure and Audio Signal Transmission System> FIG. 2 is a diagram illustrating an example configuration of an audio signal transmission system according to an embodiment of the present disclosure.

[0034] The audio signal transmission system 1 shown in FIG. 2 is configured to include an encoding device 100 and a decoding device 200.

[0035] For example, the audio signal transmission system 1 may be configured as a system for realizing a remote live performance. In this case, the encoding device 100 is configured as a server (distribution device) at the live performance venue, and the decoding device 200 is configured as a playback terminal on the remote audience side.

[0036] In the audio signal transmission system 1, the encoding device 100 separates the band-divided signals, which are obtained by dividing the input audio signal into bands, into envelope components and the remaining flat waveforms. Considering the importance of the envelope components to hearing, the encoding device 100 encodes and quantizes the envelope components by frequency conversion, while encoding the flattened waveform with a lower number of bits than the envelope components. The encoded bitstream resulting from the encoded audio signal in this manner is transmitted to the decoding device 200.

[0037] The decoding device 200 decodes, for each band, the coded data of the envelope component and the coded data of the flattened waveform from the coded bit stream transmitted from the coding device 100. The decoding device 200 generates a band division signal by synthesizing the decoded envelope component and the flattened waveform, and outputs a speech signal by synthesizing the band division signals of each band.

[0038] The encoding device 100 and the decoding device 200 will be described in detail below.

[0039] 4. Configuration and Operation of Encoding Apparatus (Configuration of Encoding Apparatus) FIG. 3 is a block diagram showing an example of the functional configuration of the encoding apparatus 100. As shown in FIG.

[0040] As shown in FIG. 3, the encoding device 100 includes a band dividing unit 110, a band encoding unit 120, a content type obtaining unit 130, a network status detecting unit 140, and an encoding / multiplexing unit 150.

[0041] The band splitting unit 110 splits the audio signal input to the encoding device 100 into a predetermined K bands (e.g., K=16) using a band splitting filter. Examples of the band splitting filter include a quadrature mirror filter (QMF) and a polyphase quadrature filter (PQF). The band split signals split into bands are supplied to the band coding unit 120. Note that, in this embodiment, audio signals are basically the target of processing, but the processing is not limited thereto, and content signals including video signals, haptic signals, and other types of signals may also be the target of processing.

[0042] The band coding unit 120 has a band coding unit 120A and a band coding unit 120B for coding the band division signals of each band. For example, the band coding unit 120 is provided with a band coding unit 120A corresponding to K' bands out of K bands, and a band coding unit 120B corresponding to (K-K') bands.

[0043] The band coding unit 120A separates the envelope component and the flattened waveform from the band division signal, then encodes the envelope component by frequency transform, and encodes the flattened waveform with a different number of bits than the envelope component (specifically, a lower number of bits than the envelope component). On the other hand, the band coding unit 120B encodes the band division signal directly by frequency transform (without separating the envelope component). Note that the flattened waveform may be encoded with a higher number of bits than the envelope component, rather than a lower number of bits than the envelope component.

[0044] The band coding units 120A and 120B do not have to be provided to correspond to consecutive bands, as long as the band dividing unit 110 can switch whether each band corresponds to the band coding unit 120A or the band coding unit 120B.

[0045] That is, the band dividing section 110 can determine the encoding method for each band, that is, whether to separate and encode the envelope component and flattened waveform from the band divided signal, or to encode the band divided signal as is.

[0046] In addition, the bands to be assigned to the band coding unit 120A may be switched depending on the transmission conditions of the network and the processing load of the playback terminal (decoding device 200). For example, since the processing load of synthesizing the separated envelope components and the flattened waveform is relatively large, if the resource load of the playback terminal is large, the processing load of the playback terminal can be reduced by reducing the number of bands to be assigned to the band coding unit 120A.

[0047] In this case, the band dividing unit 110 can determine the band to be assigned to the band coding unit 120A based on the network transmission status detected by the network status detection unit 140, which will be described later.

[0048] The band coding unit 120 A includes an envelope separation unit 161 , a frequency conversion unit 162 , an envelope quantization unit 163 , and a parameterization unit 164 .

[0049] The envelope separation unit 161 separates the band division signal for each band into an envelope component and other components. Specifically, as shown in FIG. 4 , the envelope separation unit 161 detects the envelope component of a time signal (band division signal) whose amplitude varies with respect to the time axis, and separates the envelope component from other components (flattened waveform). The method for separating the envelope component is not particularly limited, but commonly used methods include, for example, a method using a Hilbert transform or a method of squaring the signal waveform and then passing it through a low-pass filter. The separated envelope component is supplied to a frequency conversion unit 162, and the flattened waveform is supplied to a parameterization unit 164.

[0050] The frequency transform unit 162 transforms the envelope components from the envelope separation unit 161 into envelope spectrum coefficients by frequency transform, and supplies the envelope quantization unit 163. For the frequency transform, a modified discrete cosine transform (MDCT) is generally used, but a discrete cosine transform (DCT), a DFT (Discrete Fourier Transform) filter bank, a cosine modulation filter bank, or the like may also be used (transform coding is equivalent to a filter bank).

[0051] Here, the content type acquisition unit 130 acquires the type of content or sound as the content type of the audio signal. The content type is acquired from content metadata or the like, or is automatically detected, as information such as the program type (music, news, movie, sports, etc.) and category information of video content, the category (classical, pop, jazz, etc.) and instrument type of music content, etc. These content types may be selected by user input. Furthermore, when the content to be encoded is object data related to each audio sound source, the instrument type and object priority of the object may be acquired as the content type.

[0052] The network status detection unit 140 also acquires the transmission bit rate, delay amount, and the like from the system as the network transmission status, and acquires it as the load processing status from the playback terminal (decoding device 200).

[0053] The envelope quantization unit 163 quantizes the envelope spectrum coefficients from the frequency conversion unit 162 using a predetermined number of bits based on the content type acquired by the content type acquisition unit 130 and the network transmission status detected by the network status detection unit 140, and outputs the resulting quantized spectrum to the encoding / multiplexing unit 150. The envelope spectrum coefficients are normalized in units of a predetermined quantization band using the maximum value within the quantization band, and are quantized using a predetermined number of bits.

[0054] At this time, bits are dynamically allocated depending on the type of content or sound obtained as the content type so that the quantization accuracy of important frequency bands in the envelope components (hereinafter also referred to as modulation frequency bands) is high and the quantization accuracy of less important modulation frequency bands is coarse.

[0055] For example, as shown in Figure 5, if the type of content is healing music or environmental sounds, the type of sound (tone and atmosphere) should be calming, relaxing, and smooth. In this case, the modulation frequency band from 5 to 20 Hz is considered important, and the quantization bandwidth is narrowed and the number of bits allocated to this band is increased.

[0056] In addition, when the content type is classical or jazz, the sound type (tone and atmosphere) only needs to be calm and smooth. In this case, the modulation frequency band of 30 to 150 Hz is considered important, and the quantization bandwidth for this band is narrowed and the number of bits allocated is increased.

[0057] Furthermore, when the type of content is pop music, rock, movie audio, etc., the type of sound (tone and atmosphere) may be sharp, clear, with a strong attack, dry sound, etc. In this case, the modulation frequency band above 200 Hz is not considered to be very important, and the quantization bandwidth for that band is widened and the number of bits allocated is reduced.

[0058] Furthermore, bits may be dynamically allocated according to not only the type of content or sound but also the network transmission status and object priority. For example, when the network transmission status is insufficient, the number of quantization bits for a modulation frequency band determined to be less important for the content type may be reduced or no bits may be allocated to that band, thereby preventing information about that band from being transmitted. Furthermore, the processing load status of the playback terminal may be transmitted to the server (encoding device 100), and the band to which information is not transmitted may be determined according to the processing load status. Furthermore, for low-priority objects, the transmission bandwidth can be reduced by reducing or no bits may be allocated to bands and modulation frequency bands that are less important for the content, thereby preventing information from being transmitted.

[0059] The parameterization unit 164 performs parameter coding to represent the flattened waveform from the envelope separation unit 161 using predetermined parameters. For example, in parameter coding, a plurality of signal waveform patterns are prepared in advance as a table, and the flattened waveform is replaced with the index number of the signal waveform pattern that most closely resembles it. Furthermore, if the flattened waveform resembles a sine wave or a multi-tone waveform, the flattened waveform may be converted into several sine wave parameters such as the frequency, gain, and phase of the sine wave. The flattened parameters obtained by parameter coding are output to the encoding / multiplexing unit 150.

[0060] On the other hand, the band coding unit 120 B is made up of a frequency conversion unit 171 and a quantization unit 172 .

[0061] The frequency transform unit 171 transforms the band division signal for each band into frequency spectrum coefficients by frequency transform, and supplies the frequency spectrum coefficients to the quantization unit 172 .

[0062] The quantization unit 172 quantizes the frequency spectrum coefficients from the frequency conversion unit 171 using a predetermined number of bits, and outputs the resulting quantized spectrum to the encoding / multiplexing unit 150 .

[0063] The encoding / multiplexing unit 150 then converts the quantized spectrum from the band encoding unit 120A (envelope quantization unit 163) for each band and the quantized spectrum from the band encoding unit 120B (quantization unit 172) for each band into entropy codes such as Huffman codes. Furthermore, the encoding / multiplexing unit 150 multiplexes the converted entropy codes together with flattening parameters from the band encoding unit 120A (parameterization unit 164) for each band in a predetermined format to generate an encoded bitstream.

[0064] (Transmission Format of Encoded Bitstream) FIG. 6 is a diagram showing an example of a transmission format of an encoded bitstream.

[0065] In the example of Fig. 6, the syntax of the transmission format and related processing are described. In Fig. 6, the bold text indicates the syntax recorded in the bitstream, and the syntax is recorded with the number of bits listed in the right column of the table. Text other than the bold text is auxiliary information such as syntax-related processing and the number of loops.

[0066] Num_band is the number of transmission bands, which is the number of band division signals, among those divided by the band division unit 110 in Fig. 3, that are coded and transmitted to the decoding device 200. For example, if the band division signal is divided into 16 parts, Num_band is expressed in 4 bits. The number of transmission bands is controlled according to the network transmission conditions detected by the network condition detection unit 140, and when the transmission conditions are severe, the transmission bandwidth can be saved by reducing the number of transmission bands.

[0067] isEnvelopeSpectral is a 1-bit flag that indicates True if the spectral data of band k is the spectrum of the envelope component, and False if not. In other words, isEnvelopeSpectral indicates whether the processing for band k includes envelope component separation and synthesis processing. When isEnvelopeSpectral is True, the subsequent spectra represent envelope spectra, and when isEnvelopeSpectral is False, the subsequent spectra represent the spectrum of the band-decomposed signal itself.

[0068] Num_qu[k] is the number of quantization bands transmitted in band k. A quantization band is a unit for quantizing a spectrum, and one quantization band contains several spectra. In the example of FIG. 6, Num_qu[k] is expressed in 4 bits. By changing the number of quantization bands depending on the content type and network transmission conditions, it is possible to reduce the amount of information to be transmitted while retaining modulation frequency components that are subjectively important to the content. For example, based on the parameters of important bands for each content as shown in FIG. 5, the quantization bands are determined depending on the network transmission conditions so as to include modulation frequency components that are important to the tone and content.

[0069] The algorithm for determining the quantization band based on the given Num_qu[k] is switched depending on whether isEnvelopeSpectral is True or False. This is because the appropriate quantization bandwidth differs between the spectrum of the envelope component and the spectrum of the band-decomposition signal itself.

[0070] scale_factor_index[k][q] is the index of the scale factor in quantization band q of band k. The scale factor is a coefficient used to normalize the spectrum, and the exponent part of the power of 2 is the index. For example, in the case of 5 bits, 2 5 =32 levels of normalization can be expressed.

[0071] quant_nbits[k][q] represents the value {(number of quantization bits)-1} applied to band k and quantization band q (because 0 bits are not used). By changing the number of quantization bits for each spectrum depending on the content type and network transmission conditions, it is possible to efficiently reduce the information to be transmitted while retaining modulation frequency components that are subjectively important to the content. As with the number of quantization bands described above, more bits can be allocated to frequency components and modulation frequency components that are important to the content based on parameters such as those shown in Figure 5.

[0072] The underlined Num_spec(k,q) is a function that obtains the quantization bandwidth (number of included spectra) in band k and quantization band q. The quantization bandwidth of each band is specified in advance.

[0073] code_spec[k][q] is the code of the spectrum that has been entropy-encoded using a Huffman code or similar. Since it is a variable-length code, the number of bits is variable. The underlined Huff_decode() is a function that decodes the Huffman code using a specified table to obtain a quantized spectrum. For simplicity, this example shows coding each spectral coefficient individually, but the same applies to multidimensional Huffman codes that group multiple spectra.

[0074] The underlined spectral coefficient Spec[k][n] of band k is obtained by dividing the decoded quantized spectrum by the quantization precision resolution to perform inverse quantization, and then multiplying it by a scale factor to perform inverse normalization. As described above, when isEnvelopeSpectral is True, the obtained spectral coefficient is the spectrum of the envelope component, and when isEnvelopeSpectral is False, the obtained spectral coefficient is the spectrum of the band-decomposition signal itself.

[0075] If isEnvelopeSpectral is True, parameter-encoded information about the flattened waveform (carrier signal) is also recorded. wave_type[k] represents the type of flattened waveform for band k. Here, 0 represents a general waveform such as noise or a complex signal. 1 to 3 represent multi-tone waveforms consisting of the corresponding number of sine waves. When wave_type[k] represents a general waveform, wave_index[k] represents the index of the waveform pattern, and wave_gain[k] represents its gain index. The flattened waveform is generated using the waveform pattern and gain corresponding to this index. Furthermore, when wave_type[k] represents multi-tone, the tone frequency index tone_freq, gain index tone_gain, and phase index tone_phase are recorded repeatedly for the number of tones.

[0076] The coded bit stream generated in the coding device 100 is transmitted to the decoding device 200 based on the above-described transmission format.

[0077] (Operation of Encoding Device) Next, the flow of encoding processing of an audio signal by the encoding device 100 will be described with reference to the flowchart in Fig. 7. In Fig. 7, the flow of encoding processing for bands where isEnvelopeSpectral is True will be mainly described, but encoding processing for bands where isEnvelopeSpectral is False is also performed in parallel.

[0078] In step S101, the band dividing unit 110 divides the audio signal input to the encoding device 100 into band division signals for each band.

[0079] In step S102, the band coding unit 120A (envelope separating unit 161) separates the band division signal for each band into an envelope component and a flattened waveform.

[0080] In step S103, the band coding unit 120A (the frequency transform unit 162 and the envelope quantization unit 163) encodes the envelope components to obtain a quantized spectrum. Specifically, the frequency transform unit 162 converts the envelope components into envelope spectral coefficients by frequency transform, and the envelope quantization unit 163 quantizes the envelope spectral coefficients to obtain a quantized spectrum.

[0081] In step S104, the band coding unit 120A (parameterization unit 164) encodes the flattened waveform by parameter coding to obtain flattening parameters.

[0082] In step S105, the encoding / multiplexing unit 150 generates an encoded bitstream by multiplexing the quantized spectrum and the flattening parameter for each band.

[0083] With the above configuration and processing, the band-divided signal is separated into an envelope component and other components, and the envelope component that is important for the timbre and sound atmosphere is coded with high precision by frequency conversion, while the other components are coded with fewer bits. Therefore, even with a small amount of information transmitted, the original timbre and sound atmosphere, and ultimately the sense of realism, can be restored without loss. In other words, good sound quality can be maintained even at low bit rates.

[0084] 5. Configuration and Operation of Decoding Device> (Configuration of Decoding Device) FIG. 8 is a block diagram showing an example of the functional configuration of the decoding device 200. As shown in FIG.

[0085] As shown in FIG. 8, the decoding device 200 includes a decoding and demultiplexing unit 210, a band decoding unit 220, a synthesis parameter determining unit 230, and a band synthesis unit 240.

[0086] The decoding and demultiplexing unit 210 performs a decoding process on the coded bit stream of the audio signal input to the decoding device 200 .

[0087] Specifically, the decoding / demultiplexing unit 210 separates, for each band, from the coded bit stream of the audio signal, a quantized spectrum (first coded data) obtained by coding the envelope component of the audio signal by frequency conversion, and a flattening parameter (second coded data) obtained by coding (parameter coding) the flattened waveform of the audio signal with a lower number of bits than the envelope component. The decoding / demultiplexing unit 210 also separates, for each band, a frequency spectrum (third coded data) obtained by frequency converting the band division signal itself, from the coded bit stream of the audio signal.

[0088] At this time, the decoding / demultiplexing unit 210 determines, for each band, whether to separate the quantized spectrum and flattening parameter or only the quantized spectrum, according to a flag (isEnvelopeSpectral) included in the coded bitstream. That is, if isEnvelopeSpectral is True, the quantized spectrum and flattening parameter are separated from the coded bitstream of the corresponding band, and if isEnvelopeSpectral is False, only the quantized spectrum (a frequency spectrum obtained by frequency-transforming the band division signal itself) is separated from the coded bitstream of the corresponding band. The quantized spectrum (flattening parameter) separated for each band is supplied to the band decoding unit 220.

[0089] The band decoding unit 220 has a band decoding unit 220A and a band decoding unit 220B for decoding the quantized spectrum (flattening parameter) of each band. For example, the band decoding unit 220 is provided with a band decoding unit 220A corresponding to K' bands out of K bands, and a band decoding unit 220B corresponding to (K-K') bands.

[0090] The band decoding unit 220A generates a band division signal by combining an envelope component obtained by decoding the quantized spectrum with a flattened waveform obtained by decoding the flattening parameter for a band where isEnvelopeSpectral is True. On the other hand, the band decoding unit 220B outputs a band division signal by decoded the quantized spectrum for a band where isEnvelopeSpectral is False.

[0091] The band decoding units 220A and 220B do not have to be provided to correspond to consecutive bands, respectively. Whether the band decoding unit 220A or the band decoding unit 220B corresponds to each band can be switched by isEnvelopeSpectral included in the coded bitstream.

[0092] The band decoding unit 220A is made up of an inverse quantization unit 251, an envelope weighting unit 252, an inverse frequency conversion unit 253, a flattened waveform generation unit 254, and an envelope synthesis unit 255.

[0093] The inverse quantization unit 251 performs a predetermined inverse quantization process on the quantized spectrum from the decoding / demultiplexing unit 210, such as converting the quantization index back into a spectrum or multiplying the spectrum by the inverse gain of the normalization process, to obtain N envelope spectrum coefficients, where N is the number of frequency bins obtained when the envelope component is frequency-transformed. The envelope spectrum coefficients are supplied to the envelope weighting unit 252.

[0094] The envelope weighting unit 252 performs weighting processing on the envelope spectrum coefficients for each band based on weighting information (synthesis parameters) supplied from the synthesis parameter determination unit 230. Specifically, the envelope weighting unit 252 multiplies the envelope spectrum coefficients by different weighting coefficients for each frequency bin based on the weighting information from the synthesis parameter determination unit 230 to obtain weighted envelope spectrum coefficients. The obtained weighted envelope spectrum coefficients are supplied to the inverse frequency transform unit 253. The synthesis parameter determination unit 230 determines weighting information for setting the weighting coefficients in the weighting processing using any of pre-prepared parameters. Details of the synthesis parameter determination unit 230 and the weighting information will be described later with reference to FIG. 9 .

[0095] The inverse frequency transform unit 253 converts the weighted envelope spectrum coefficients from the envelope weighting unit 252 back into a time signal of the envelope component by inverse frequency transform, and supplies the time signal to the envelope synthesis unit 255 .

[0096] The flattened waveform generator 254 generates a flattened waveform by generating a signal waveform of a pattern corresponding to the parameter index from the flattening parameters from the decoder / demultiplexer 210, or by synthesizing multi-tones from the sine wave parameters. The obtained flattened waveform is supplied to the envelope synthesizer 255.

[0097] The envelope synthesis unit 255 generates a band division signal by synthesizing the envelope component from the inverse frequency conversion unit 253 and the flattened waveform from the flattened waveform generation unit 254. The band division signal is generated by amplitude modulating the flattened waveform as a carrier wave with the envelope component. The generated band division signal is output to the band synthesis unit 240.

[0098] On the other hand, the band decoding unit 220 B is made up of an inverse quantization unit 261 and an inverse frequency conversion unit 262 .

[0099] The inverse quantization unit 261 performs a predetermined inverse quantization process on the quantized spectrum from the decoding and demultiplexing unit 210 to obtain frequency spectrum coefficients, and supplies the frequency spectrum coefficients to the inverse frequency transform unit 262 .

[0100] The inverse frequency transform unit 262 performs inverse frequency transform on the frequency spectrum coefficients from the inverse quantization unit 261 to return them to band division signals, and outputs them to the band synthesis unit 240 .

[0101] The band synthesis unit 240 then synthesizes the band division signals of each band from the band decoding unit 220 using a synthesis filter bank, thereby generating an audio signal of a time waveform signal.

[0102] (Configuration of Synthesis Parameter Determination Unit) Here, the configuration of the synthesis parameter determination unit 230 will be described with reference to FIG.

[0103] As shown in FIG. 9, the synthesis parameter determination unit 230 includes a content type acquisition unit 271 , preset synthesis parameters 272 , and an envelope synthesis control unit 273 .

[0104] The content type acquisition unit 271 has the same function as the content type acquisition unit 130 of Fig. 7 and acquires the type of content or sound as the content type of the audio signal. For example, the content type is acquired from content metadata or automatically detected as information such as the program type (music, news, movie, sports, etc.) or category information of video content, or the category (classical, pop, jazz, etc.) or instrument type of music content. These content types may be selected by user input. Furthermore, when the content to be encoded is data of an audio object (audio object data), for example, the instrument type or object priority of the audio object may be acquired as the content type.

[0105] The preset synthesis parameters 272 are preset parameters (parameter sets) for weighting the envelope components so that they have a tone and sound atmosphere corresponding to the sound and type of content obtained as the content type.

[0106] For example, as shown in Fig. 10, if the type of content is healing music or environmental sounds, the type of sound (tone and atmosphere) may be calming, relaxing, and smooth. As a corresponding parameter, a parameter is provided that reduces (-6 dB) the modulation frequency band from 5 to 20 Hz for all frequency bands.

[0107] In addition, when the type of content is classical or jazz, the type of sound (tone and atmosphere) needs to be calm and smooth. As a corresponding parameter, a parameter is provided that reduces (-6 dB) the modulation frequency band from 30 to 150 Hz for frequency bands above 100 Hz.

[0108] Furthermore, when the type of content is pop music, rock, movie audio, etc., the type of sound (tone and atmosphere) may be sharp, clear, with a strong attack, dry sound, etc. As a corresponding parameter, a parameter is provided that increases (+6 dB) the modulation frequency band from 200 to 1 kHz for the frequency band from 1 kHz to 3.5 kHz.

[0109] 9 , the envelope synthesis control unit 273 selects parameters that are optimal for the content type (category, instrument type, tone color, etc.) obtained by the content type acquisition unit 271 from parameters prepared in advance as preset synthesis parameters 272. The selected parameters are supplied to the envelope weighting unit 252 as weight information.

[0110] (Detailed Configuration of Envelope Weighting Unit) Here, the detailed configuration of the envelope weighting unit 252 will be further described.

[0111] FIG. 11 is a block diagram showing an example of the configuration of the envelope weighting unit 252.

[0112] The weighting coefficient setting unit 281 sets a weighting coefficient for each envelope spectrum coefficient based on the weighting information from the synthesis parameter determination unit 230. The weighting unit 282 calculates the N envelope spectrum coefficients X k,n (n=0, 1, . . . , N−1), a different weighting factor W is assigned to each frequency bin. k,n By multiplying the weighted envelope spectrum coefficients Y k,n (n=0, 1, ..., N-1) is output.

[0113] Since the above-described processing is performed for all bands k, the weight information from the synthesis parameter determination unit 230 is in the form of a two-dimensional table as shown in Fig. 12. The weight information shown in Fig. 12 is a table having Num_band rows and N columns. As described above, Num_band represents the number of transmission bands.

[0114] In this example, a simple weighting factor W is applied to the envelope spectrum coefficients. k,n However, a weighted sum of N envelope spectrum coefficients may be used. That is, for each band k, N×N weighting coefficients W k,n,m Using Y k,n =Σm[W k,n,m ×X k,n ] (n, m=0, 1, . . . , N−1) may be performed.

[0115] Furthermore, a neural network may be used for weighting the envelope spectrum coefficients.

[0116] FIG. 13 is a block diagram showing another example of the configuration of the envelope weighting unit 252.

[0117] The weighting coefficient setting unit 291 sets weighting coefficients for the network unit 292 based on weighting information from the synthesis parameter determination unit 230. The network unit 292 calculates N envelope spectrum coefficients X k,n (n=0, 1, . . . , N−1) are weighted using a neural network to obtain N weighted envelope spectrum coefficients Y k,n (n=0, 1, ..., N-1) is output.

[0118] The neural network parameters are trained using reference signals, which are manually edited sound sources that have been designed to have the desired timbre and sound atmosphere.

[0119] For example, parameters can be learned in the flow shown in Fig. 14. First, the signal x(t) of the sound source to be processed is subjected to band division and envelope separation using a configuration similar to that of the encoding device 100 in Fig. 3 to obtain the envelope component Ev x (k, f) is obtained (k is the band index, and f is the envelope spectrum coefficient index). x (k, f) is input to a network such as a DNN (Deep Neural Network), and the resulting output is called Ev x’ (k, f). The network configuration here may be arbitrary.

[0120] On the other hand, the envelope component obtained by similarly dividing and envelope-separating the signal y(t) of the reference sound source having the desired timbre is expressed as Ev y Let (k, f).

[0121] Then, the above-mentioned network output Ev x’ (k, f) and the envelope component Ev of the reference sound source y The network parameters are trained so that (k, f) are as close as possible, for example, by minimizing the mean square error (MSE) between them as a loss function.

[0122] In addition, a generative adversarial network (GAN) using a classifier that identifies whether an input signal is a signal of a reference sound source or a signal generated by a network (generator) may be used in combination. In addition to the above-mentioned learning methods, reinforcement learning, unsupervised learning, etc. may also be used.

[0123] (Operation of Decoding Device) Next, the flow of decoding processing of an audio signal by the decoding device 200 will be described with reference to the flowchart in Fig. 15. In Fig. 15, the flow of decoding processing for bands where isEnvelopeSpectral is True will be mainly described, but decoding processing for bands where isEnvelopeSpectral is False is also performed in parallel.

[0124] In step S201, the decoding / demultiplexing unit 210 separates the quantized spectrum and the flattening parameter for each band from the coded bit stream of the audio signal input to the decoding device 200.

[0125] In step S202, the band decoding unit 220A (the inverse quantization unit 251, the envelope weighting unit 252, and the inverse frequency transform unit 253) decodes the quantized spectrum separated from the coded bitstream to obtain envelope components. Specifically, the inverse quantization unit 251 inversely quantizes the quantized spectrum to obtain envelope spectrum coefficients, the envelope weighting unit 252 performs weighting processing on the envelope spectrum coefficients, and the inverse frequency transform unit 253 inversely transforms the weighted envelope spectrum coefficients into envelope components by inverse frequency transform.

[0126] In step S203, the band decoding unit 220A (flattened waveform generating unit 254) decodes the flattening parameters separated from the coded bitstream to obtain a flattened waveform.

[0127] In step S204, the band decoding unit 220A (envelope synthesis unit 255) generates a band division signal by synthesizing the envelope component and the flattened waveform.

[0128] Then, in step S205, the band synthesis unit 240 synthesizes the band division signals of the respective bands to generate an audio signal.

[0129] With the above configuration and processing, envelope components important to the timbre and sound atmosphere are coded with high accuracy by frequency conversion, and the coded bitstream in which other components are coded with fewer bits is decoded. Therefore, even with a small amount of information transmitted, the original timbre and sound atmosphere, and ultimately the sense of realism, can be restored without loss. In other words, good sound quality can be maintained even at low bit rates.

[0130] 6. Modifications of the Decoding Device A modification of the above-described decoding device 200 will be described.

[0131] (Weighting of envelope components after inverse frequency transformation)

[0132] FIG. 16 is a block diagram showing another example of the functional configuration of the decoding device 200.

[0133] The decoding device 200 of Figure 16 differs from the decoding device 200 of Figure 8 in that the band decoding unit 220A does not have an envelope weighting unit 252, and an envelope processing unit 311 is newly provided between the inverse frequency conversion unit 253 and the envelope synthesis unit 255.

[0134] 16 (band decoding unit 220A) does not directly perform weighting processing on the N envelope spectral coefficients obtained by decoding and inverse quantization, but instead converts the envelope spectral coefficients into envelope components by inverse frequency transform, and then the envelope processing unit 311 performs weighting processing on a desired (specific) frequency range of the envelope components. In this way, the envelope processing unit 311 also performs weighting processing on the envelope spectral coefficients for each band based on the weight information supplied from the synthesis parameter determination unit 230.

[0135] In this case, modulation frequency components can be processed in a manner more suited to human auditory characteristics, regardless of the number of divisions or bandwidth constraints of a band division filter such as MDCT. For example, the envelope components can be divided into non-equal parts using a constant-Q filter bank modeled after the "modulation filter bank" disclosed in "Dau, T et al. "Modeling auditory processing of amplitude modulation. I. Detection and masking with narrow-band carriers." The Journal of the Acoustical Society of America, vol. 102,5 Pt 1 (1997): 2892-905.", and then some of the divided signals can be processed. Note that when the signal input to the encoding device 100 is a video signal or a haptic signal, processing suited to human visual characteristics or haptic characteristics, respectively, may be performed on the decoding device 200 side.

[0136] FIG. 17 is a block diagram showing an example of the configuration of the envelope processing unit 311.

[0137] The weighting coefficient setting unit 321 sets a weighting coefficient for each band based on the weighting information from the synthesis parameter determination unit 230. The analysis filter bank unit 322 calculates the envelope component x k M band signals u k,m (m=0, 1, . . . , M−1). The weighting unit 323 divides the M band signals u k,m For each divided band, the weighting coefficient w k,m The synthesis filter bank 324 multiplies the weighted M band signals u k,m By recombining the processed envelope component y k Output.

[0138] Note that the operations of the envelope processing unit 311 described above and the envelope weighting unit 252 in Fig. 8 are premised on the fact that the relevant band has been envelope separated (isEnvelopeSpectral is True). In other words, since the decoding device 200 cannot apply envelope processing to bands that have not been envelope separated, it is desirable that the encoding device 100 perform envelope separation on bands whose envelope components may be processed during decoding.

[0139] In this case, for a band that has not been envelope separated (isEnvelopeSpectral is False), the weight information from the synthesis parameter determination unit 230 relating to that band is ignored.

[0140] If the envelope components of a band that has not been envelope separated are to be processed, they may be processed after envelope separation, similar to the processing in the encoding device 100. Specifically, the envelope separation unit 161 or the frequency conversion unit 162 in Fig. 3 may be provided downstream of the band decoding unit 220B (inverse frequency conversion unit 262), and the envelope components (envelope spectrum coefficients) and flattened waveform may be acquired, processed by the envelope weighting unit 252 (Fig. 8) or the envelope processing unit 311 (Fig. 16), and then re-synthesized by the envelope synthesis unit 255 for output.

[0141] (Modification of Combining Parameter Determination Unit) FIG. 18 is a block diagram showing another example of the configuration of the combining parameter determination unit 230 included in the decoding device 200. In FIG.

[0142] The synthesis parameter determination unit 230 in FIG. 18 differs from the synthesis parameter determination unit 230 in FIG. 9 in that a user profile acquisition unit 331 is provided instead of the content type acquisition unit 271 .

[0143] The user profile acquisition unit 331 acquires profile information of the user who is viewing the content, such as the user's hearing characteristics, the user's preferences, and the device characteristics of the hearable device worn by the user.

[0144] For example, the user profile acquisition unit 331 acquires the user's preferred tone, the device characteristics of the earphones they are using, and if the user is hard of hearing, their hearing characteristics and the device characteristics of their hearing aids from a pre-recorded user profile, and outputs this as profile information to the envelope synthesis control unit 273. Furthermore, the pre-recorded user profile may be individually optimized for each user and updated, or new data may be added.

[0145] The envelope synthesis control unit 273 determines weighting information by selecting parameters that are optimal for the profile information from the user profile acquisition unit 331 from parameters prepared in advance as preset synthesis parameters 272. The preset synthesis parameters 272 are prepared with parameters for each band to produce an optimal sound that matches the user's preferred tone, hearing characteristics (especially in the case of a person with hearing loss), characteristics of the device being used, etc.

[0146] For example, if a user prefers a tone with less roughness, but the characteristics of the device they use increase the sensitivity of the modulation frequency band related to roughness, causing the sound to sound rough, the tone can be improved by lowering the intensity of the modulation frequency component in that band.

[0147] Furthermore, based on the finding that people with presbycusis often have a higher modulation depth threshold in the TMTF (temporal modulation depth transfer function) compared to people with normal hearing, as disclosed in "Okamoto Yasuhide, Kanzaki Akira, Nukuno Ayako, Nakaichi Takeshi, Morimoto Takashi, Harada Kota, Kubota Eri, Ogawa Iku, Temporal Resolution Measurement in Presbycusis," AUDIOLOGY JAPAN, 2014, Vol. 57, No. 6, pp. 694-702, "it is possible to improve the difficulty in hearing voices and music caused by hearing loss by increasing the intensity of modulation frequency components that result in a higher modulation depth threshold compared to people with normal hearing.

[0148] Furthermore, for a user who has difficulty hearing human voices due to hearing loss, the corresponding envelope spectrum coefficient may be increased based on the finding that "the modulation frequency of the amplitude envelope of speech is 8 to 10 Hz" as disclosed in "Drullman, R et al. "Effect of reducing slow temporal modulations on speech reception," The Journal of the Acoustical Society of America, vol. 95,5 Pt 1 (1994): 2670-80." Specifically, for a frequency band around 100 to 2000 Hz (the frequency band corresponding to human voices), the envelope spectrum corresponding to the modulation frequency band of 8 to 10 Hz may be increased by +6 dB.

[0149] FIG. 19 is a block diagram showing yet another example configuration of the synthesis parameter determination unit 230 included in the decoding device 200.

[0150] The combining parameter determination unit 230 in FIG. 19 differs from the combining parameter determination unit 230 in FIG. 9 in that a terminal information acquisition unit 341 is provided instead of the content type acquisition unit 271 .

[0151] The terminal information acquisition unit 341 acquires terminal information of the playback terminal (decoding device 200), the transmission status (network status) of the network through which the coded bitstream is transmitted, and the like.

[0152] For example, the terminal information acquisition unit 341 acquires the processing resources and load status of the decoding device 200, such as the remaining battery capacity, CPU usage rate, and memory usage rate, and outputs this to the envelope synthesis control unit 273 as terminal information.

[0153] The envelope synthesis control unit 273 determines weight information by selecting parameters that are optimal for the terminal information from the terminal information acquisition unit 341 from parameters prepared in advance as the preset synthesis parameters 272 .

[0154] When the playback terminal is running low on battery power or processing resources, it is possible to omit some of the calculation processing in the inverse frequency transform unit 253 and the envelope synthesis unit 255 downstream of the envelope weighting unit 252 by ignoring envelope spectral coefficients that do not significantly affect the main timbre. As an example of further reduction in the amount of processing, as described above, by ignoring 1 / 2 or 3 / 4 of the high-frequency side of the envelope spectral coefficients, the transform size of the inverse frequency transform such as the inverse MDCT can be reduced to 1 / 2 or 1 / 4 of the original size, thereby enabling conversion to downsampled envelope components with a smaller amount of calculation. In this case, the envelope synthesis unit 255 may perform envelope synthesis processing on the downsampled envelope components as they are.

[0155] In addition, if the network is poor and packet loss or transmission errors cause crackling noise, the appropriate preset parameters can be used to convert the noise into a less noticeable tone. For example, by converting the tone into one with more attack and roughness, like rock music, the noise can be made less noticeable.

[0156] Furthermore, the envelope synthesis control unit 273 may control the band decoding unit 220A and the band synthesis unit 240 for each band based on terminal information. When the processing load is imminent, the processing load can be significantly reduced by not performing all decoding and synthesis processes for the predetermined high frequency bands divided into bands. Also, the load status of the playback terminal may be notified to the encoding device 100, and the band to be encoded and transmitted may be switched on the server side depending on the load status of the playback terminal. In this case, the network status detection unit 140 (FIG. 3) may receive and detect the load status of the playback terminal.

[0157] (Configuration for Decoding Audio Object Coded Data) The decoding device 200 may be configured to decode audio object coded data that includes audio object data and audio object metadata as a coded bitstream.

[0158] FIG. 20 is a block diagram showing yet another example of the functional configuration of the decoding device 200.

[0159] The decoding device 200 in Fig. 20 differs from the decoding device 200 in Fig. 8 and Fig. 16 in that a rendering unit 351 is newly provided after each of the band decoding units 220A and 220B provided for each band. Note that in the decoding device 200 in Fig. 20, the same components as those in the decoding device 200 in Fig. 8 and Fig. 16 are denoted by the same reference numerals, and descriptions thereof will basically be omitted.

[0160] However, in addition to the above-mentioned functions, the decoding / demultiplexing unit 210 decodes audio object data and object metadata from the coded bitstream. The object metadata is supplied to the rendering unit 351 for use in rendering, and is also supplied to the synthesis parameter determination unit 230.

[0161] The rendering unit 351 is provided for each divided band, performs rendering processing of audio objects for each frequency band, and outputs a rendered audio signal that has been rendered into a multi-channel signal. The rendered audio signals for each band are synthesized in the band synthesis unit 240, and a time waveform signal for each channel is obtained.

[0162] Furthermore, the content type acquisition unit 271 (FIG. 9) of the synthesis parameter determination unit 230 acquires object information of the audio object from the object metadata decoded by the decoding / demultiplexing unit 210. The object information includes the content category (attribute information) to which the audio object belongs, the type of sound (instrument, vocal, environmental sound, etc.), the type of instrument, timbre information (tone type, parameters of timbre elements such as roughness and sharpness), object priority (priority information), and the extent of spread of the audio object (spread information).

[0163] The envelope synthesis control unit 273 (FIG. 9) of the synthesis parameter determination unit 230 determines weighting information by selecting optimal preset parameters based on the object information acquired by the content type acquisition unit 271. Since audio objects are typically categorized by instrument or sound type, parameters that will result in the optimal timbre are selected according to the category, sound type, instrument type, etc. included in the object information. Furthermore, if the object metadata includes timbre information (timbre type, roughness, sharpness, and other timbre element parameters), a parameter set that is close to each timbre element is selected according to the timbre information.

[0164] Furthermore, a parameter set may be selected according to the object priority. For example, when the object priority is high, it is considered that the effect on the entire reproduced sound is large, so a parameter set that particularly emphasizes the timbre components according to the sound type or category of the audio object is selected. Conversely, when the object priority is low, only the envelope components important to that category are left, and the other envelope components are replaced with zero, thereby further reducing the processing load during decoding.

[0165] In this way, optimal parameters are selected based on the object information and output as weight information.

[0166] In object coding, signals from sound sources such as instruments and vocals located in the same position are coded as audio objects for each sound source. Sounds without a clear sound source location, such as reverberation or audience cheers, can be represented by several virtual objects. When decoding and playing back coded object data, the rendering unit 351 decodes the audio signals of multiple audio objects contained in the coded bitstream and converts them back into time waveform signals. The rendering unit 351 then maps the sounds into a three-dimensional space based on the position information and gain information of the audio objects contained in the object metadata, and maps them to multiple channel signals according to the playback environment (rendering process).

[0167] For example, in the case of speaker playback, the spatial position of the sound image is reproduced according to the arrangement and number of speakers using a technique called VBAP (Vector-based Amplitude Panning), and the audio signal of each audio object is mapped to each channel signal based on the position information of the audio object included in the object metadata. In the case of headphone playback, a room reverberation impulse response such as a head-related transfer function (HRTF) or RIR (Room Impulse Response) is applied to generate three-dimensional sound (also called a virtualizer) so that the sound image is localized at the position of each audio object when listening. In addition, BRIR (Binaural Room Impulse Response) or HOA (Higher Order Ambisonics) may be used for the rendering process.

[0168] Typically, such rendering processing is performed after decoding all audio waveforms of required audio objects. That is, it is necessary to provide a decoding device 200 for each audio object, or to repeat the decoding processing as many times as the number of audio objects before performing the rendering processing. Furthermore, different rendering processing may be required for each frequency band. For example, in the case of speakers, it is desirable to perform separate rendering processing for each frequency band, since the device characteristics and appropriate placement differ for each frequency band, such as the woofer, mid, and tweeter (the same applies to headphones).

[0169] As described above, the decoding process and rendering process of audio objects impose a large processing load, and therefore, in the decoding device 200 of Fig. 20, rendering process is performed for each band prior to the band synthesis unit 240. With this configuration, when performing different rendering processes for each frequency, there is no need to perform band splitting process again at a later stage, thereby reducing the amount of processing.

[0170] 20, the rendering unit 351 is provided downstream of the band decoding units 220A and 220B for each band. However, considering that the envelope component significantly contributes to the sense of sound image localization, the rendering unit 351 may be provided between the inverse frequency transform unit 253 and the envelope synthesis unit 255 inside the band decoding unit 220A (FIG. 8). Alternatively, the rendering unit 351 may be provided upstream of the inverse frequency transform units 253 and 262, and rendering processing (multiplying a different gain for each frequency) may be performed on the envelope spectrum coefficients (frequency spectrum coefficients) in the frequency domain.

[0171] It should be noted that the contribution of the flattened waveform to the sense of sound image localization is thought to be small (especially in the low frequency range). Therefore, the flattened waveform of each object may be added as is and mixed down to each channel signal, or rendering may be performed using a simple method such as VBAP, in which gain adjustment is performed on the gain parameters obtained by parameter coding the flattened waveform.

[0172] 7. Modified Example of Encoding Device and Music Creation Editing Tool (Modified Example of Encoding Device) FIG. 21 is a block diagram showing another example of the functional configuration of the encoding device 100. In FIG.

[0173] The encoding device 100 of Fig. 21 differs from the encoding device 100 of Fig. 3 in that an envelope correction unit 411 is provided between the frequency transform unit 162 and the envelope quantization unit 163 in the band encoding unit 120A, and a synthesis parameter determination unit 420 is newly provided. Note that in the encoding device 100 of Fig. 21, the same components as those in the encoding device 100 of Fig. 3 are assigned the same reference numerals, and descriptions thereof will basically be omitted. Also, in Fig. 21, the content type acquisition unit 130 and network status detection unit 140 provided in the encoding device 100 of Fig. 3 are not shown.

[0174] The envelope correction unit 411 performs weighting processing on the envelope spectrum coefficients for each band based on the weighting information supplied from the synthesis parameter determination unit 420. Specifically, the envelope correction unit 411 acquires weighted envelope spectrum coefficients by multiplying the envelope spectrum coefficients from the frequency conversion unit 162 by weighting coefficients that differ for each frequency bin, based on the weighting information from the synthesis parameter determination unit 420. The acquired weighted envelope spectrum coefficients are supplied to the envelope quantization unit 163.

[0175] The synthesis parameter determination unit 420 is composed of a tone color selection unit 431 , preset synthesis parameters 432 , and an envelope synthesis control unit 433 .

[0176] The preset synthesis parameters 432 and the envelope synthesis control unit 433 have basically the same functions as the preset synthesis parameters 272 and the envelope synthesis control unit 273 provided in the synthesis parameter determination unit 230 shown in FIG. 9, respectively.

[0177] The timbre selection unit 431 acquires timbre information of the content in response to an input from a predetermined GUI (Graphical User Interface), and outputs it to the envelope synthesis control unit 433 .

[0178] The envelope synthesis control unit 433 selects parameters that are optimal for the tone color information from the tone color selection unit 431 from among parameters prepared in advance as preset synthesis parameters 432 .

[0179] The encoding device 100 shown in FIG. 21 configured as above has two operation modes: an edit mode and an archive mode.

[0180] The edit mode is used when the encoding device 100 is called from the edit screen (GUI) of a music production and editing tool, which will be described later, and an editor converts a desired signal waveform into a desired tone color.

[0181] In the edit mode, the envelope correction unit 411 multiplies each component of the envelope spectrum coefficients from the frequency conversion unit 162 by a weighting factor based on weighting information from the synthesis parameter determination unit 420, and outputs weighted envelope spectrum coefficients.

[0182] Meanwhile, the timbre selection unit 431 of the synthesis parameter determination unit 420 acquires the timbre type of the content input from the editing screen and supplies it to the envelope synthesis control unit 433. In accordance with the timbre information from the timbre selection unit 431, the envelope synthesis control unit 433 selects the parameters that are optimal for the timbre information from among the preset synthesis parameters 432 prepared in advance, and outputs them as weighting information.

[0183] In this way, the encoding device 100 of FIG. 21 can convert a desired signal into a sound with a desired tone or atmosphere during encoding.

[0184] Archive mode is used, for example, when automatically archiving content. If the envelope components (modulation frequency components) important to the timbre of the signal being archived are abnormally strong or change suddenly, this can cause an unnatural sound, even if there is no clipping. Therefore, during archiving, the system detects abnormalities in the envelope components that affect the timbre, and if an abnormality is detected, corrects the envelope components to optimize the timbre.

[0185] In archive mode, the envelope correction unit 411 determines whether the envelope spectral coefficients from the frequency transform unit 162 are within a predetermined range. If the envelope spectral coefficients exceed a predetermined threshold or are not within the range, the envelope correction unit 411 corrects the coefficients to optimal coefficients by multiplying them by a predetermined weighting factor based on the parameters of the weighting information from the synthesis parameter determination unit 420, and outputs the weighted envelope spectral coefficients. The weighted envelope spectral coefficients are supplied to the envelope quantization unit 163, and subsequent processing is performed in the same manner as in the encoding device 100 of FIG. 3.

[0186] Meanwhile, a timbre selection unit 431 of the synthesis parameter determination unit 420 acquires the timbre type of the content, which has been manually input or automatically detected from the audio signal, and supplies it to an envelope synthesis control unit 433. In accordance with the timbre information from the timbre selection unit 431, the envelope synthesis control unit 433 selects parameters that are optimal for the timbre information from preset synthesis parameters 432 that have been prepared in advance, and outputs the selected parameters as weighting information.

[0187] In this way, according to the encoding device 100 of Figure 21, during archiving processing, abnormalities in the envelope components that affect the tone can be detected at the same time as encoding, and if an abnormality is found, it can be corrected to the optimal envelope component of the tone.

[0188] (Editing Screen of Music Production Editing Tool) FIG. 22 is a diagram showing an example of the editing screen of the music production editing tool described above.

[0189] The music production and editing tool is configured to include the encoding device 100 of FIG. 21, and corrects and converts the tone color of the selected signal waveform.

[0190] 22, the main editing screen 500 on the left displays the signal waveforms of each track or object of the audio data being edited. When the editor selects a specific waveform portion WP in track (object) 02 as the track (object) they wish to edit and presses the Change Timbre button 501, a timbre selection dialog box 510 is displayed in the upper right corner of the figure.

[0191] The editor selects the desired timbre to be emphasized or attenuated in the timbre selection dialog 510. Here, the editor can select timbre from Fluctuation, Sharpness, and Roughness.

[0192] Alternatively, instead of the timbre selection dialog 510, a timbre processing dialog 520 may be displayed at the bottom right of the figure. In this case, the editor can adjust the intensity of the aforementioned fluctuation, sharpness, and roughness using sliders. Each intensity has a positive and negative value, and when it is set to 0, the corresponding timbre remains unchanged.

[0193] When the OK button 511 is pressed in the tone color selection dialog 510 or the OK button 521 is pressed in the tone color processing dialog 520, the focus returns to the main editing screen 500. In the main editing screen 500, processing is performed on the selected waveform portion WP of the audio data so as to emphasize or attenuate the tone color selected and adjusted in the tone color selection dialog 510 or the tone color processing dialog 520.

[0194] That is, in the encoding device 100 of Fig. 21, for a selected timbre, parameters corresponding to that timbre and content are obtained from the preset synthesis parameters 432, and the envelope correction unit 411 corrects and processes the timbre by changing the weighting coefficients of the envelope components (method of processing the envelope spectral coefficients) using those parameters. Note that when the intensity is 0, the corresponding envelope spectral coefficients are not processed. The range of the envelope spectral coefficients processed corresponding to the selected timbre is not specified, but the corresponding frequency bands and modulation frequency bands are shown in Fig. 23.

[0195] As shown in Figure 23, if you want to change the sense of fluctuation, the envelope spectrum coefficients corresponding to the modulation frequency band of 5 to 20 Hz for all frequency bands can be processed based on the intensity adjusted in the tone processing dialog 520.

[0196] Furthermore, if you want to change the roughness, for the frequency band above 100 Hz, the envelope spectrum coefficients corresponding to the modulation frequency band from 30 to 150 Hz can be processed based on the strength adjusted in the tone processing dialog 520.

[0197] Furthermore, if you want to change the sharpness, the envelope spectrum coefficients corresponding to the modulation frequency band from 200 to 1 kHz for the frequency band from 1 kHz to 3.5 kHz can be processed based on the intensity adjusted in the tone processing dialog 520.

[0198] When the above processing is completed, the editor can check the processed audio data by pressing the play button 502, and can save the processed audio data by pressing the OK button 503.

[0199] 8. Summary According to the encoding device 100 ( FIG. 1 ) of the present disclosure, an audio signal is separated into an envelope component and other components, the envelope component that is important for the timbre and atmosphere of the sound is frequency-converted and coded with high precision, and the other components are coded with a smaller number of bits. Therefore, even when transmitting a small amount of information, the original atmosphere, timbre, and sense of realism of the sound can be restored without loss.

[0200] Furthermore, depending on the tone and sound atmosphere corresponding to the sound category, a large number of bits are allocated to the quantization of the envelope components that are important for reproducing that tone, so even when transmitting a small amount of information, the signal can be reproduced with the tone that is optimal for that sound category.

[0201] Furthermore, according to the encoding device 100 (FIG. 1), transmission format (FIG. 6), and decoding device 200 (FIG. 8) of the present disclosure, it is possible to dynamically switch for each band between a configuration in which the band division signal is separated into an envelope component and other components and encoded, and a configuration in which the envelope component is not separated but is frequency-converted and encoded as is. This makes it possible to reproduce the envelope component of a band that is important for tone reproduction while suppressing the processing load according to circumstances such as the processing load of the playback terminal, and to reproduce a signal with the optimal tone.

[0202] According to the decoding device (FIGS. 8, 16, and 20) of the present disclosure, the timbre and sound atmosphere of an audio signal can be easily changed to desired characteristics during playback. In particular, playback can be performed with the optimal timbre depending on the type of video or audio content. Furthermore, even when network transmission conditions are poor or the processing load of the playback terminal is heavy, sound can be listened to with the optimal timbre with minimal sound dropouts or degradation in sound quality. Furthermore, the timbre can be dynamically switched to the optimal timbre for the user's hearing loss characteristics and viewing device.

[0203] Furthermore, by using a trained neural network in the process of multiplying the envelope components by weighting factors, it is possible to easily convert the audio signal into one that is closer to the desired tone color without having to tune the tone color conversion weighting factors.

[0204] The encoding device 100 ( FIG. 21 ) and the editing screen ( FIG. 22 ) of the music production and editing tool of the present disclosure enable easy and effective conversion to a sound with a desired tone and atmosphere during music production or editing, thereby reducing the time and effort required for music content production. Furthermore, during archiving processing, areas that cause abnormal sounds or discomfort can be automatically detected and corrected to the optimal tone, thereby reducing the effort required for archiving and improving the quality of archived content.

[0205] 9. Example of Computer Configuration The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware, a general-purpose personal computer, or the like.

[0206] 24 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes using a program. The encoding device 100 and the decoding device 200 are configured, for example, by a computer 800 having a configuration similar to that shown in FIG.

[0207] A CPU 801 , a ROM (Read Only Memory) 802 , and a RAM (Random Access Memory) 803 are interconnected by a bus 804 .

[0208] An input / output interface 805 is further connected to the bus 804. An input unit 806 including a keyboard, a mouse, etc., and an output unit 807 including a display, a speaker, etc. are connected to the input / output interface 805. Also connected to the input / output interface 805 are a storage unit 808 including a hard disk, a nonvolatile memory, etc., a communication unit 809 including a network interface, etc., and a drive 810 that drives removable media 811.

[0209] In the computer 800 configured as described above, the CPU 801 performs the above-described series of processes by, for example, loading a program stored in the memory unit 808 into the RAM 803 via the input / output interface 805 and the bus 804 and executing it.

[0210] The program executed by the CPU 801 is stored on a removable medium 811, or is provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting, and installed in the storage unit 808.

[0211] The program executed by computer 800 may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0212] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0213] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0214] The embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure.

[0215] For example, the embodiment of the present disclosure can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

[0216] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0217] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0218] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0219] Furthermore, the technology disclosed herein may have the following configuration: (1) A decoding device including: a demultiplexing unit that separates, for each band, from an encoded bit stream of a content signal, first encoded data obtained by frequency-converting an envelope component of the content signal and second encoded data obtained by encoding a flattened waveform of the content signal with a different number of bits than the envelope component; a band decoding unit that generates a band-divided signal by synthesizing the envelope component decoded from the first encoded data and the flattened waveform decoded from the second encoded data; and a band synthesizing unit that generates the content signal by synthesizing the band-divided signals for each band. (2) The decoding device described in (1), wherein the content signal is at least one of an audio signal, a video signal, and a haptic signal. (3) The decoding device described in (1), wherein the first encoded data is a frequency spectrum obtained by frequency-converting the envelope component, and the second encoded data is a parameter obtained by parameter-coding the flattened waveform. (4) The decoding device according to (1) or (2), wherein the demultiplexing unit separates, for each band, from the coded bitstream, third coded data which is a frequency spectrum obtained by frequency-transforming the band division signal, and the band decoding unit outputs the band division signal obtained by decoding the third coded data. (5) The decoding device according to (4), wherein the demultiplexing unit determines, for each band, whether to separate the first coded data and the second coded data or to separate only the third coded data, according to a flag included in the coded bitstream. (6) The decoding device according to any of (1) to (5), wherein the band decoding unit performs weighting processing on the first coded data for each band. (7) The decoding device according to (6), wherein the band decoding unit multiplies each frequency bin of the first coded data for each band by a different weighting coefficient. (8) The decoding device according to (7), wherein the band decoding unit performs the weighting processing using a trained neural network. (9) The decoding device according to any one of (1) to (5), wherein the band decoding unit performs weighting processing on a specific frequency range of the envelope component obtained by decoding the first encoded data for each band.(10) The decoding device according to (9), wherein the band decoding unit multiplies band signals obtained by dividing the envelope component using a first filter bank by weighting coefficients and then resynthesizes the band signals using a second filter bank. (11) The decoding device according to any of (1) to (10), further comprising a weight information determination unit that determines weight information for setting weight coefficients in weighting processing related to the first encoded data for each band using any of pre-prepared parameters. (12) The decoding device according to (11), wherein the weight information determination unit selects the parameter based on at least one of the content type of the content signal, user profile information, terminal information of the device itself, and network conditions over which the encoded bitstream is transmitted. (12-1) The decoding device according to (12), wherein the content type includes at least one of a video program type, video or music category information, instrument type, and timbre information. (12-2) The decoding device according to (12), wherein the profile information includes at least one of the hearing characteristics of the user, the preferences of the user, and the device characteristics of a hearable device worn by the user. (12-3) The decoding device according to (12), wherein the terminal information includes at least one of a remaining battery level, a CPU usage rate, and a memory usage rate. (13) The decoding device according to (11), wherein the coded bitstream is coded audio object data including audio object data and metadata of the audio objects, and further comprising a rendering unit that performs a rendering process of the audio objects on the band-divided signal for each band. (14) The decoding device according to (13), wherein the metadata includes at least one of attribute information, position information, and priority information of the audio objects. (15) The decoding device according to (13), wherein the rendering process is a rendering process using at least one of VBAP (Vector-based Amplitude Panning), HRTF (Head Related Transfer Function), RIR (Room Impulse Response), and HOA (Higher Order Ambisonics).(16) The decoding device according to (13), wherein the weight information determination unit selects the parameters based on the metadata. (17) The decoding device according to any of (13) to (16), wherein the rendering unit performs the rendering process based on the metadata. (18) A decoding method, wherein a decoding device separates, for each band, an encoded bit stream of a content signal into first encoded data obtained by encoding an envelope component of the content signal by frequency conversion and second encoded data obtained by encoding a flattened waveform of the content signal with a different number of bits from that of the envelope component, generates a band-divided signal by synthesizing the envelope component obtained by decoded the first encoded data and the flattened waveform obtained by decoded the second encoded data, and generates the content signal by synthesizing the band-divided signals of each band. (19) A program causing a computer to execute the following processes: separating, for each band, an encoded bit stream of a content signal into first encoded data obtained by encoding an envelope component of the content signal by frequency transform, and second encoded data obtained by encoding a flattened waveform of the content signal with a different number of bits from that of the envelope component, generating a band-divided signal by synthesizing the envelope component obtained by decoding the first encoded data and the flattened waveform obtained by decoding the second encoded data, and generating the content signal by synthesizing the band-divided signals for each band. (20) A coding device comprising: a band dividing unit that divides a content signal into band-divided signals for each band, a band coding unit that separates the band-divided signals into an envelope component and a flattened waveform, encodes the envelope component by frequency transform, and encodes the flattened waveform with a different number of bits from that of the envelope component, and a multiplexing unit that multiplexes the first encoded data obtained by encoding the envelope component and the second encoded data obtained by encoding the flattened waveform, to generate a coded bit stream. (21) The encoding device according to (20), wherein the first encoded data is a frequency spectrum obtained by frequency-transforming the envelope component, and the second encoded data is parameters obtained by parameter-encoding the flattened waveform.(22) The encoding device according to (21), wherein the parameters include either a predetermined signal waveform pattern or a sine wave parameter. (23) The encoding device according to (21), wherein the band encoding unit encodes the band divided signal directly by frequency transform, and the multiplexing unit generates the encoded bit stream by multiplexing the first encoded data, the second encoded data, and third encoded data, which is a frequency spectrum obtained by encoding the band divided signal by frequency transform. (24) The encoding device according to (23), wherein the band separation unit determines, for each band, an encoding method to separate the band divided signal into the envelope component and the flattened waveform and encode them, or to encode the band divided signal directly. (25) The encoding device according to (23), wherein the band separation unit determines the encoding method based on resource information of a playback terminal of the content signal or a network status for transmitting the encoded bit stream. (26) The encoding device according to (21), wherein the band encoding unit quantizes the first encoded data obtained by encoding the envelope component with a quantization bit rate or a quantization bandwidth according to at least one of the content type of the content signal and a network status over which the encoded bitstream is transmitted. (27) The encoding device according to (26), wherein the content type includes at least one of a type of content, a type of sound of the content, and a priority of an audio object. (28) The encoding device according to (21), wherein the band encoding unit performs a weighting process on the first encoded data for each band. (29) The encoding device according to (28), further comprising a weight information determination unit that determines weight information for setting a weight coefficient in the weighting process using any of pre-prepared parameters. (30) The encoding device according to (29), wherein the weight information determination unit selects the parameter in response to an input from a GUI (Graphical User Interface). (31) The encoding device according to (28), wherein the band encoding unit performs the weighting process in accordance with fluctuations in a specific frequency component in the first encoded data for each band.

[0220] 1 Audio signal transmission system, 100 Encoding device, 110 Band separation unit, 120, 120A, 120B Band coding unit, 130 Content type acquisition unit, 140 Network status detection unit, 150 Encoding / multiplexing unit, 161 Envelope separation unit, 162 Frequency conversion unit, 163 Envelope quantization unit, 164 Parameterization unit, 171 Frequency conversion unit, 172 Quantization unit, 200 Decoding device, 210 Decoding / demultiplexing unit, 220, 220A, 220B Band decoding unit, 230 Synthesis parameter determination unit, 240 Band synthesis unit, 251 Inverse quantization unit, 252 Envelope weighting unit, 253 Inverse frequency conversion unit, 254 Flattened waveform generation unit, 255 Envelope synthesis unit, 261 Inverse quantization unit, 262 inverse frequency conversion unit, 271 content type acquisition unit, 272 preset synthesis parameters, 273 envelope synthesis control unit, 311 envelope processing unit, 331 user profile acquisition unit, 341 terminal information acquisition unit, 351 rendering unit, 411 envelope correction unit, 420 synthesis parameter determination unit, 431 timbre selection unit, 432 preset synthesis parameters, 433 envelope synthesis control unit

Claims

1. A decoding device comprising: a demultiplexing unit that separates, for each band, from an encoded bit stream of a content signal, first encoded data obtained by encoding an envelope component of the content signal by frequency conversion, and second encoded data obtained by encoding a flattened waveform of the content signal with a different number of bits than the envelope component; a band decoding unit that generates a band division signal by synthesizing the envelope component decoded from the first encoded data and the flattened waveform decoded from the second encoded data; and a band synthesis unit that generates the content signal by synthesizing the band division signals of each band.

2. The decoding device according to claim 1, wherein the content signal is at least one of an audio signal, a video signal, and a haptic signal.

3. The decoding device according to claim 1, wherein the first encoded data is a frequency spectrum obtained by frequency-transforming the envelope component, and the second encoded data is parameters obtained by parameter-coding the flattened waveform.

4. The decoding device according to claim 1, wherein the demultiplexing unit separates third encoded data, which is a frequency spectrum obtained by frequency-converting the band division signal, for each band, from the encoded bit stream, and the band decoding unit outputs the band division signal decoded from the third encoded data.

5. The decoding device according to claim 4, wherein the demultiplexing unit determines, for each band, whether to separate the first encoded data and the second encoded data, or to separate only the third encoded data, depending on a flag included in the encoded bitstream.

6. The decoding device according to claim 1, wherein the band decoding unit performs weighting processing on the first encoded data for each band.

7. The decoding device according to claim 6, wherein the band decoding section multiplies each frequency bin of the first encoded data for each band by a different weighting coefficient.

8. The decoding device according to claim 7, wherein the band decoding unit performs the weighting process using a trained neural network.

9. The decoding device according to claim 1, wherein the band decoding section performs weighting processing on a specific frequency range of the envelope component obtained by decoding the first encoded data for each band.

10. The decoding device according to claim 9, wherein the band decoding unit multiplies the band signals obtained by dividing the envelope components using a first filter bank by weighting coefficients, and then resynthesizes the band signals using a second filter bank.

11. The decoding device according to claim 1, further comprising a weight information determination unit that determines weight information for setting a weight coefficient in the weighting process related to the first encoded data for each band using any of pre-prepared parameters.

12. The decoding device according to claim 11, wherein the weight information determination unit selects the parameters based on at least one of the content type of the content signal, user profile information, terminal information of the own device, and the network conditions over which the encoded bitstream is transmitted.

13. The decoding device according to claim 11, wherein the encoded bitstream is audio object encoded data including audio object data and audio object metadata, and further comprising a rendering unit that performs rendering processing of the audio object on the band division signal for each band.

14. The decoding device according to claim 13, wherein the metadata includes at least one of attribute information, position information, and priority information of the audio object.

15. The decoding device according to claim 13, wherein the rendering process uses at least one of VBAP (Vector-based Amplitude Panning), HRTF (Head Related Transfer Function), RIR (Room Impulse Response), and HOA (Higher Order Ambisonics).

16. The decoding device according to claim 13, wherein the weight information determination section selects the parameters based on the metadata.

17. The decoding device according to claim 13, wherein the rendering section performs the rendering process based on the metadata.

18. A decoding method in which a decoding device separates, for each band, an encoded bit stream of a content signal into first encoded data in which an envelope component of the content signal is encoded by frequency conversion, and second encoded data in which a flattened waveform of the content signal is encoded with a different number of bits from that of the envelope component, generates a band-split signal by synthesizing the envelope component decoded from the first encoded data and the flattened waveform decoded from the second encoded data, and generates the content signal by synthesizing the band-split signals of each band.

19. A program for causing a computer to execute the following processes: separate, for each band, an encoded bit stream of a content signal into first encoded data obtained by encoding an envelope component of the content signal by frequency conversion, and second encoded data obtained by encoding a flattened waveform of the content signal with a different number of bits from that of the envelope component; generate a band-split signal by synthesizing the envelope component obtained by decoded the first encoded data and the flattened waveform obtained by decoded the second encoded data; and generate the content signal by synthesizing the band-split signals of each band.

20. An encoding device comprising: a band splitting unit which splits a content signal into band split signals for each band; a band encoding unit which separates the band split signals into an envelope component and a flattened waveform, encodes the envelope component by frequency conversion, and encodes the flattened waveform with a different number of bits than the envelope component; and a multiplexing unit which generates an encoded bit stream by multiplexing first encoded data in which the envelope component is encoded and second encoded data in which the flattened waveform is encoded.

Images