Audio encoding device, audio decoding device, and program

Abstract

To provide a voice encoding device and a voice decoding device that prevent the sound quality from degrading within a range in which customization of a reproduced sound is permitted.SOLUTION: An acoustic metadata acquisition unit acquires acoustic metadata at least representing whether a sound signal is a priority signal or a non-priority signal. A classification unit classifies a plurality of input sound signals into priority signals or non-priority signals by referring to the acoustic metadata. A distribution unit is a distribution unit that distributes bit rates to the sound signals, and distributes a priority bit rate corresponding to the priority signal among all the bit rates as designated to the priority signal, and distributes a surplus bit rate based on a result obtained by subtracting a total sum of the priority bit rates distributed to the priority signals from all the bit rates to the non-priority signal. A sound signal encoding unit encodes each of the sound signals in accordance with the bit rates distributed by the distribution unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an audio encoding device, an audio decoding device, and a program. [Background technology]

[0002] Conventional multi-channel audio systems, such as the 22.2ch audio that began broadcasting in Japan in December 2018, are channel-based audio systems. Meanwhile, in recent years, the practical application of object-based audio systems and AR / VR audio, which combine audio signals and audio metadata, has been progressing. Object-based audio and AR / VR audio are mechanisms that provide a large number of audio signals to adjust, customize, and render sound to the listener's preferences, but they inevitably increase the amount of data transmitted. Therefore, the latest audio coding methods use technology that determines the importance of multiple audio signals based on predetermined indicators (e.g., levels, loudness values, and priorities at the time of rendering), and distributes the bit rate according to that importance to reduce the required bit rate for the entire program.

[0003] Object-based audio systems are described in Non-Patent Documents 1, 2, and 3. These documents describe a technique for determining the importance of an audio signal based on a predetermined index and allocating a bit rate according to the importance.

[0004] Regarding VR audio, the Moving Picture Experts Group (MPEG) is currently developing ISO / IEC 23090-4:202X, "MPEG-I Immersive Audio Coding (tentative name)" as a standard specification (reference URL: https: / / mpeg.chiariglione.org / sites / default / files / files / meetings / docs / w17904.docx).

[0005] Furthermore, Non-Patent Documents 4, 5, and 6 describe multi-channel audio technology in the production of broadcast programs. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] ISO / IEC 23008-3:2019, Information technology - High efficiency coding and media delivery in heterogeneous environments - Part 3: 3D audio, Second edition, International Organization for Standardization, February 2019. [Non-Patent Document 2] ETSI TS 103 190-2 V1.2.1 (2018-02), TECHNICAL SPECIFICATION, Digital Audio Compression (AC-4) Standard; Part 2: Immersive and personalized audio, ETSI, February 2018. [Non-Patent Document 3] ATSC Standard: A / 342 Part 3, MPEG-H System, March 3, 2017. [Non-Patent Document 4] ARIB Standard STD-B59 Version 2.0, 3D Multi-Channel Sound Studio Standard, ARIB, July 2016. [Non-Patent Document 5] Recommendation ITU-R BS.2051-2, "Advanced sound system for programme production", ITU-R, July 2018. [Non-Patent Document 6] SMPTE STANDARD SMPTE 2036-2-2008, "Ultra High Definition Television-Audio Characteristics and Audio Channel Mapping for Program Production", SMPTE, July 9, 2008. Summary of the Invention [Problem to be solved by the invention]

[0007] However, the above conventional technology uses an algorithm that optimizes the distribution of bit rates according to the initial state and initial value of the audio signal of a program. In other words, the conventional technology does not guarantee that the distributed bit rate is optimal even for the state of the program sound after the listener customizes it. In other words, depending on the content of the listener's customization, there is a possibility that sound quality deterioration beyond the expectation of the program producer may occur. For example, in a program that handles multiple audio signals, if a listener extracts and listens to only the audio signal with the smallest bit rate distribution amount (the audio signal with the greatest degradation), the sound quality deterioration will become significantly more noticeable due to the listener's customization, even though the sound quality deterioration was masked by the other audio signals in the initial state of the program.

[0008] In other words, in an audio system that allows the listener to customize the playback sound (for example, object-based audio, AR / VR audio, etc.), an audio encoding and decoding method that does not cause sound quality degradation within the range of allowable customization is required. In other words, an appropriate distribution of bit rate is required so that sound quality does not deteriorate within the range of allowable customization.

[0009] The present invention has been made based on the recognition of the above problems, and aims to provide an audio encoding device, an audio decoding device, and a program that do not cause degradation of sound quality within the range of allowable customization of the reproduced sound. [Means for solving the problem]

[0010] [1] In order to solve the above problem, an audio encoding device according to one embodiment of the present invention includes an audio metadata acquisition unit that acquires audio metadata indicating at least whether an audio signal is a priority signal or a non-priority signal; a classification unit that classifies multiple input audio signals as either priority signals or non-priority signals by referring to the audio metadata; a distribution unit that distributes a bit rate to the audio signals, the distribution unit distributing a priority bit rate corresponding to the priority signals from a specified total bit rate to the priority signals and distributing a surplus bit rate based on a result of subtracting the sum of the priority bit rates distributed to the priority signals from the total bit rate to the non-priority signals; and an audio signal encoding unit that encodes each of the audio signals in accordance with the bit rate distributed by the distribution unit.

[0011] [2] Furthermore, in one aspect of the present invention, in the above-mentioned audio encoding device, when it is determined that a specific priority signal among the priority signals is silent, the distribution unit cancels the distribution of the priority bit rate to the specific priority signal and instead distributes the priority bit rate to other audio signals.

[0012] [3] Moreover, in one aspect of the present invention, in the above-mentioned audio encoding device, the distribution unit distributes the priority bitrate to the priority signal, and then distributes to the non-priority signal a non-priority bitrate corresponding to the non-priority signal out of a post-subtraction bitrate resulting from subtracting the sum of the non-priority bitrates distributed to the priority signals from a total bitrate, and distributes the surplus bitrate resulting from subtracting the sum of the non-priority bitrates distributed to the non-priority signals from the post-subtraction bitrate to the non-priority signal.

[0013] [4] Moreover, in one aspect of the present invention, in the above-mentioned audio encoding device, the distribution unit distributes the surplus bit rate also to the priority signal.

[0014] [5] Moreover, in one aspect of the present invention, in the above-mentioned audio encoding device, the distribution unit distributes the priority bitrate to the priority signal, and then distributes to the non-priority signal a non-priority bitrate corresponding to the non-priority signal out of a post-subtraction bitrate resulting from subtracting the sum of the non-priority bitrates distributed to the priority signals from a total bitrate, and distributes the surplus bitrate resulting from subtracting the sum of the non-priority bitrates distributed to the non-priority signals from the post-subtraction bitrate to the priority signal.

[0015] [6] In another aspect of the present invention, the distributor distributes the surplus bit rate also to the non-priority signal.

[0016] [7] In another aspect of the present invention, in the above-mentioned audio encoding device, the device further includes a separation unit that acquires an input signal including the acoustic metadata and the plurality of audio signals, separates the input signal into the acoustic metadata and the audio signals, and passes the separated audio signals to the classification unit.

[0017] [8] In another aspect of the present invention, the above-mentioned audio encoding device further includes an audio metadata encoding unit that encodes the acquired audio metadata, and a multiplexing unit that multiplexes and outputs the encoded audio signal output by the audio signal encoding unit and the encoded audio metadata output by the audio metadata encoding unit.

[0018] [9] Also, one aspect of the present invention is that, in the above-mentioned audio encoding device, the priority signal is an audio signal of an audio object, and the non-priority signal is an audio signal of channel-based audio, which is an audio format such as 22.2ch audio.

[0019]

[10] Furthermore, in one aspect of the present invention, in the above-mentioned audio encoding device, the priority bit rate per channel of the priority signal is a fixed value according to the encoding method executed by the audio signal encoding unit, and is 32 kbits per second, 33 kbits per second, 34 kbits per second, 35 kbits per second, 36 kbits per second, 37 kbits per second, 38 kbits per second, 39 kbits per second, 40 kbits per second, 41 kbits per second, 42 kbits per second, 43 kbits per second, 44 kbits per second, , 45 kbits per second, 46 kbits per second, 47 kbits per second, 48 kbits per second, 49 kbits per second, 50 kbits per second, 51 kbits per second, 52 kbits per second, 53 kbits per second, 54 kbits per second, 55 kbits per second, 56 kbits per second, 57 kbits per second, 58 kbits per second, 59 kbits per second, 60 kbits per second, 61 kbits per second, 62 kbits per second, 63 kbits per second, or 64 kbits per second.

[0020]

[11] Another aspect of the present invention is a program for causing a computer to function as the audio encoding device described in any one of [1] to

[10] above.

[0021]

[12] Furthermore, an audio decoding device according to one embodiment of the present invention includes a separation unit that acquires a bitstream and separates the bitstream into an encoded audio signal and encoded audio metadata, an audio metadata decoding unit that decodes the encoded audio signal, and an audio rendering unit that performs audio rendering of the audio signal decoded by the audio signal decoding unit based on the acoustic metadata decoded by the audio metadata decoding unit, and outputs it as reproduced sound.

[0022]

[13] Also, one aspect of the present invention is that in the above-mentioned audio decoding device, the audio signal decoding unit decodes a priority signal and a non-priority signal contained in the encoded audio signal, and the audio rendering unit audio renders the decoded priority signal and non-priority signal based on the acoustic metadata decoded by the acoustic metadata decoding unit, and combines the audio rendering result of the priority signal and the audio rendering result of the non-priority signal to output the reproduced sound.

[0023]

[14] Moreover, one aspect of the present invention is that, in the above-mentioned audio decoding device, the device further includes an operation unit that generates instruction information based on a user's operation, and the audio rendering unit audio renders the decoded audio signal based also on the instruction information from the operation unit.

[0024]

[15] Another aspect of the present invention is a program for causing a computer to function as an audio decoding device according to any one of

[12] to

[14] above. Effect of the Invention

[0025] According to the present invention, when encoding audio content composed of multiple audio signals, transmission capacity (or recording capacity) can be preferentially allocated to audio signals for which sound quality degradation is not permitted. This makes it possible to maintain the overall quality of the content in line with the intentions of the content creator. [Brief description of the drawings]

[0026] [Figure 1] 1 is a functional block diagram showing a schematic functional configuration of a speech encoding device according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a schematic diagram showing an example of the arrangement of playback positions of each channel of multi-channel audio used in the first embodiment. [Diagram 3] 2 is a schematic diagram showing rules and procedures for distributing a bit rate by the audio encoding device according to the first embodiment. FIG. [Figure 4] FIG. 2 is a functional block diagram showing a first example of a schematic functional configuration of an audio decoding device according to a first embodiment. [Diagram 5] FIG. 4 is a functional block diagram showing a second example of a schematic functional configuration of the audio decoding device according to the first embodiment. [Figure 6] FIG. 11 is a functional block diagram showing a schematic functional configuration of a speech encoding device according to a second embodiment. [Figure 7] FIG. 11 is a schematic diagram showing rules and procedures for distributing a bit rate by the audio encoding device according to the second embodiment. [Figure 8] FIG. 11 is a functional block diagram showing a schematic functional configuration of an audio decoding device according to a second embodiment. [Figure 9] FIG. 11 is a functional block diagram showing a schematic functional configuration of a speech encoding device according to a third embodiment. [Figure 10] FIG. 11 is a schematic diagram (1 / 2) showing rules and procedures for distributing a bit rate by the audio encoding device according to the third embodiment. [Figure 11] FIG. 11 is a schematic diagram (2 / 2) showing rules and procedures for distributing a bit rate by the audio encoding device according to the third embodiment. [Figure 12] 13 is a flowchart (1 / 2) showing the procedure of an encoding process performed by a speech encoding device according to a third embodiment. [Figure 13] 13 is a flowchart (2 / 2) showing the procedure of the encoding process by the audio encoding device of the third embodiment. [Figure 14] FIG. 13 is a functional block diagram showing a schematic functional configuration of a speech encoding device according to a fourth embodiment. [Figure 15] FIG. 13 is a schematic diagram (1 / 2) showing rules and procedures for distributing a bit rate by an audio encoding device according to a fourth embodiment. [Figure 16] FIG. 13 is a schematic diagram (2 / 2) showing rules and procedures for distributing a bit rate by the audio encoding device according to the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] Next, a number of embodiments of the present invention will be described with reference to the drawings. In each embodiment, audio signals are classified into priority signals and non-priority signals for processing. A predetermined bit rate (e.g., a fixed bit rate according to the encoding method, etc.) is allocated to the priority signal. This ensures that the quality of the audio is guaranteed when the priority signal is encoded. The bit rate allocated to the priority signal is called the priority bit rate. The priority bit rate may also be called the mandatory bit rate, the bit rate for audio objects, the guaranteed bit rate, the fixed bit rate, etc. The non-priority signal is appropriately allocated the bit rate remaining after allocating the priority bit rate. The bit rate allocated to the non-priority signal is called the non-priority bit rate. The non-priority bit rate may be called an arbitrary bit rate, an optimal bit rate, a background sound bit rate, an unconstrained bit rate, a non-guaranteed bit rate, etc. Additionally, the surplus bit rate is also distributed appropriately.

[0028] In addition, in the embodiments described below, audio metadata is used, which may include information on constrained bitrate audio signal selection information (information indicating which channel's audio signal is a priority signal and which channel's audio signal is a non-priority signal), mandatory bitrate, arbitrary bitrate audio signal selection information, optimal bitrate, surplus bitrate, and total bitrate. That is, the audio metadata includes information for distinguishing whether each channel is a channel of a constrained bit-rate audio signal or a channel of an arbitrary bit-rate audio signal. The audio metadata also includes information regarding the content of the constraints for a channel of a constrained bit-rate audio signal. As a specific example, the audio metadata may include information regarding the required bit rate for each channel. That is, the audio metadata may have information indicating the value of the required bit rate for a channel of a constrained bit-rate audio signal. The audio metadata may further have information regarding the playback position and the playback volume.

[0029] The audio metadata in the existing technology is described in, for example, the following documents: Literature: Recommendation ITU-R BS.2076-1, “Audio Definition Model”, ITU-R, approved October 2019. Literature: Recommendation ITU-R BS.2125-0, “A serial representation of the Audio Definition Model”, ITU-R, approved January 21, 2019.

[0030] [First embodiment] An audio signal is input to the audio encoding device 1. This audio signal is, for example, an audio signal of a broadcast program. In other words, the audio encoding device 1 acquires and encodes audio output from an audio device used in producing the broadcast program. The audio signal input to the audio encoding device 1 may include audio metadata as a part of it. Note that the audio signal may be something other than the audio of a broadcast program.

[0031] FIG. 1 is a functional block diagram showing a schematic functional configuration of the audio encoding device according to the present embodiment. As shown in the figure, the audio encoding device 1 includes a separation unit 21, a classification unit 22, an audio metadata encoding unit 24, an audio signal encoding unit 31, a multiplexing unit 25, and an audio metadata acquisition unit 90. The audio signal encoding unit 31 includes a required bit rate distribution unit 33, a surplus bit rate distribution unit 34, an encoding unit 36, an encoding unit 37, and a combining unit 38. Each of these functional units is realized, for example, by using an electronic circuit. Each functional unit may include a storage means such as a semiconductor memory or a magnetic hard disk device, as necessary. Each function may be realized by a computer and software. An overview of each functional unit will be described below.

[0032] The separation unit 21 acquires an audio signal (e.g., an audio signal of a program) input to the audio encoding device 1, and separates it into audio metadata and the audio signal. The separation unit 21 passes the audio metadata obtained by separation to the classification unit 22, the audio signal encoding unit 31, and the audio metadata encoding unit 24. The separation unit 21 also passes the audio signal obtained by separation to the classification unit 22. Note that the separation unit 21 acquires audio metadata from the outside, and can also be considered to have the function of an audio metadata acquisition unit.

[0033] The acoustic metadata may not be separated from the input audio signal, but may be provided separately to the audio encoding device 1. In this case as well, the input acoustic metadata is passed to the classification unit 22, the audio signal encoding unit 31, and the acoustic metadata encoding unit 24.

[0034] The classification unit 22 classifies the audio signals passed from the separation unit 21 into constrained bit rate audio signals and arbitrary bit rate audio signals. Specifically, the classification unit 22 obtains information as to which channel's audio signals are constrained bit rate audio signals and which channel's audio signals are arbitrary bit rate audio signals by referring to the passed audio metadata. That is, the classification unit 22 classifies the input audio signals into constrained bit rate audio signals and arbitrary bit rate audio signals according to the referred audio metadata. The classification unit 22 may output constrained bit rate audio signals for a plurality of channels. The classification unit 22 may also output arbitrary bit rate audio signals for a plurality of channels.

[0035] The audio metadata encoder 24 encodes the audio metadata for transmission and passes the encoded audio metadata to the multiplexer 25.

[0036] The audio signal encoding unit 31 encodes and outputs the audio signal passed from the classification unit 22. The audio signal encoding unit 31 encodes the constrained bit rate audio signal and the arbitrary bit rate audio signal. The audio signal encoding unit 31 passes the encoded audio signal to the multiplexing unit 25. The audio signal encoding unit 31 has the configuration described above. The functional configuration of the audio signal encoding unit 31 in more detailed units will be described later.

[0037] The multiplexing unit 25 multiplexes the encoded audio metadata output from the audio metadata encoding unit 24 and the encoded audio data output from the audio signal encoding unit 31 for each time window (frame). The multiplexing unit 25 outputs one bit stream obtained by multiplexing as described above to the outside. The bit stream output by the multiplexing unit 25 is transmitted using a transmission medium as appropriate, and received by an audio decoding device described later.

[0038] A more detailed functional configuration of the audio signal encoding unit 31 is as follows: For encoding, the audio signal encoding unit 31 cuts out the received constrained bit-rate audio signal and arbitrary bit-rate audio signal using predetermined time windows (frames).

[0039] The mandatory bit rate distribution unit 33 distributes the mandatory bit rate to the constrained bit rate audio signal by referring to the received audio metadata. The mandatory bit rate distribution unit 33 passes information on the distributed bit rate to the encoding unit . The surplus bit rate distribution unit 34 distributes the surplus bit rate to the arbitrary bit rate audio signal while referring to the audio metadata in the same manner. The surplus bit rate is the bit rate remaining after the mandatory bit rate is distributed to the constrained bit rate audio signal, as described later. The surplus bit rate distribution unit 34 passes information on the distributed bit rate to the encoding unit 37. The manner in which the essential bitrate distribution unit 33 and the surplus bitrate distribution unit 34 distribute the bitrate will be described later with reference to another drawing.

[0040] The encoding unit 36 ​​encodes the constrained bit-rate audio signal classified by the classification unit 22, and outputs a code string. The encoding unit 36 ​​performs lossy encoding, and can appropriately adjust the bit rate (number of bits per unit time) of the code to be output. The encoding unit 36 ​​performs encoding for each channel according to the information on the bit rate (distributed bit rate) for each channel passed from the required bit-rate distribution unit 33. The encoding unit 36 ​​may perform encoding using any encoding method, such as MPEG-2 AAC (Advanced Audio Coding), MPEG-4 AAC, MPEG-D USAC (Unified Speech and Audio Coding), MPEG-H 3D Audio (3DA), MPEG-I IAC (Immersive Audio Coding), AC-3, AC-4, etc.

[0041] The encoding unit 37 encodes the arbitrary bit rate audio signal classified by the classification unit 22 and outputs a code string. The encoding unit 37 performs lossy encoding and can appropriately adjust the bit rate of the code to be output. The encoding unit 37 performs encoding for each channel according to the information on the bit rate (distributed bit rate) for each channel passed from the surplus bit rate distribution unit 34. The encoding unit 37 may perform encoding using any encoding method such as MPEG-2 AAC, MPEG-4 AAC, MPEG-D USAC, MPEG-H 3DA, MPEG-I IAC, AC-3, AC-4, etc.

[0042] The combining unit 38 combines the encoded constrained bit-rate audio signal (which may be for multiple channels) output from the encoding unit 36 ​​with the encoded arbitrary bit-rate audio signal (which may be for multiple channels) output from the encoding unit 37 to unify them. Here, the combining method is arbitrary. The combining unit 38 outputs the combined audio signal (for multiple channels). In other words, the combining unit 38 passes the combined audio signal (for multiple channels) to the multiplexing unit 25.

[0043] The acoustic metadata acquisition unit 90 acquires acoustic metadata from an external source. In other words, the acoustic metadata acquisition unit 90 can acquire acoustic metadata via a route other than the input signal acquired by the separation unit 21. The acoustic metadata acquisition unit 90 passes the acquired acoustic metadata to the classification unit 22, the audio signal encoding unit 31, and the acoustic metadata encoding unit 24.

[0044] 2 is a schematic diagram showing an example of an audio signal used in this embodiment, and showing the arrangement of channels included in the audio signal. In the illustrated example, the audio signal includes 24 channels of audio in the audio format of 22.2ch audio (Non-Patent Documents 4, 5, 6) and additional four audio objects (4 channels). In other words, this audio signal is an audio signal of a broadcast program that uses audio of a total of 28 channels.

[0045] 2(a), (b), and (c) show the arrangement of the upper, middle, and lower channels of a 22.2-channel audio system, respectively, and 2(d) shows four additional audio objects (objects A to D).

[0046] In the case of a 22.2ch audio format, the playback position in space of the audio signal of each channel is specified. As shown in (a), (b), and (c) of the same figure, the position of each channel is expressed, for example, in polar coordinates. For example, the elevation angle of the upper layer is 30 degrees, that of the middle layer is 0 degrees, and that of the lower layer is -30 degrees (depression angle 30 degrees). However, various standards (Non-Patent Documents 4 and 5) specify the allowable ranges for the azimuth angle and elevation / depression angles of each channel.

[0047] In the upper layer (Figure (a)), examples of the names and azimuth angles of each channel are as follows. Note that the azimuth angle is 0 degrees from the front, with the left side being positive and the right side being negative. The azimuth angle of TpFC (Top Front Center) is 0 degrees, the azimuth angle of TpFL (Top Front Left) is 45 degrees, the azimuth angle of TpSiL (Top Side Left) is 90 degrees, the azimuth angle of TpBL (Top Back Left) is 135 degrees, the azimuth angle of TpBC (Top Back Center) is 180 degrees, the azimuth angle of TpFR (Top Front Right) is -45 degrees, the azimuth angle of TpSiR (Top Side Right) is -90 degrees, and the azimuth angle of TpBR (Top Back Right) is -135 degrees. The position of TpC (Top Center) is the position of the center point when viewed in a plane within the upper layer.

[0048] In the middle layer (Figure (b)), examples of the names and azimuth angles of each channel are as follows: FC (Front Center) azimuth angle is 0 degrees, FLc (Front Left Center) azimuth angle is 30 degrees, FL (Front Left) azimuth angle is 60 degrees, SiL (Side Left) azimuth angle is 90 degrees, BL (Back Left) azimuth angle is 135 degrees, BC (Back Center) azimuth angle is 180 degrees, FRc (Front Right Center) azimuth angle is -30 degrees, FR (Front Right) azimuth angle is -60 degrees, SiR (Side Right) azimuth angle is -90 degrees, and BR (Back Right) azimuth angle is -135 degrees.

[0049] In the lower layer (Figure (c)), examples of the names and azimuth angles of each channel are as follows: BtFC (Bottom Front Center) has an azimuth angle of 0 degrees, BtFL (Bottom Front Left) has an azimuth angle of 45 degrees, and BtFR (Bottom Front Right) has an azimuth angle of -45 degrees. Also, the azimuth angles of LFE1 (Low Frequency Effects-1) and LFE2 (Low Frequency Effects-2) are 45 degrees and -45 degrees, respectively.

[0050] On the other hand, the playback positions of the audio objects (objects A to D) shown in Fig. 1(d) are arbitrarily defined by the audio metadata. In other words, the playback positions of the individual audio objects do not necessarily coincide with the positions of the channels included in the audio format. The playback positions of the audio objects may be, for example, positioned at an intermediate position between the channels in the audio format.

[0051] 2 are merely examples. In this embodiment, the number of channels included in the audio signal (the number of channels whose positions are defined as an audio format, and the number of audio objects whose positions can be arbitrarily defined) and their arrangement may be other than those shown in FIG.

[0052] Fig. 3 is a schematic diagram showing rules and procedures when the audio encoding device 1 distributes a bit rate. The configuration of the audio signal assumed here is the example configuration shown in Fig. 2. That is, the audio signal to be encoded has a 22.2ch audio signal (24 channels) and four audio objects (4 channels). In the example shown in Fig. 3, of the four audio objects, three audio objects are constrained bit rate audio signals and one audio object is an arbitrary bit rate audio signal. Specifically, audio objects A, B, and D are constrained bit rate audio signals, and audio object C is an arbitrary bit rate audio signal. In addition, all channels included in the 22.2ch audio signal are arbitrary bit rate audio signals.

[0053] Figure 3(a) shows the situation before the bitrate is distributed to the channels. Figure 3(b) shows the situation when the required bitrate has been distributed to the channels first. Figure 3(c) shows the situation after the surplus bitrate has been further distributed to the channels. In the figure, the 24 boxes (dotted lines) for 22.2ch audio and the four boxes (dotted lines) for audio objects correspond to the bitrates to be allocated to each channel. The box labeled "total bitrate" represents the overall transmission capacity used by the transmission medium to transmit the audio signal of a certain program.

[0054] In the situation in Fig. 3(a), the total bitrate has not yet been distributed anywhere. In other words, the bitrate of both the 22.2ch audio frame and the audio object frame is zero.

[0055] In the situation shown in FIG. 3(b), only the mandatory bit rate is allocated only to the channel of the constrained bit rate audio signal. That is, among the four audio objects, the mandatory bit rate is allocated to the audio objects A, B, and D, which are constrained bit rate audio signals. The hatched parts of the frames of the audio objects A, B, and D correspond to the allocated mandatory bit rate. And, the bit rate is not allocated to the audio object C, which is an arbitrary bit rate audio signal. Also, the bit rate is not allocated to each channel of the 22.2ch sound. Specific numerical examples are as follows. That is, when the audio coding method is MPEG-H 3DA, the mandatory bit rate per one audio object is, for example, 40 kbit / s (kilobits per second) or 48 kbit / s. When the audio coding method is MPEG-4 AAC or MPEG-2 AAC, the mandatory bit rate per one audio object is, for example, 64 kbit / s. It should be noted that "AAC" here is an abbreviation of "Advanced Audio Coding". In Fig. 3(b), the sum of the required bitrates allocated to the audio objects A, B, and D is subtracted from the total bitrate that was set in advance (the white part in the dotted line of the total bitrate frame is the subtracted amount). The bitrate remaining in this state is the surplus bitrate that will be allocated next (Fig. 3(c)).

[0056] The required bit rate per channel (priority bit rate per channel of the priority signal) may be a fixed value according to the encoding method executed by the audio signal encoding unit 31. For example, it is reasonable to set the required bit rate per channel to a fixed value of 32 kbits / sec or more and 64 kbits / sec or less. Also, the required bit rate may be, for example, any of the following values: 32 kbits / sec, 33 kbits / sec, 34 kbits / sec, 35 kbits / sec, 36 kbits / sec, 37 kbits / sec, 38 kbits / sec, 39 kbits / sec, 40 kbits / sec, 41 kbits / sec, 42 kbits / sec, 43 kbits / sec, 44 kbits / sec, 45 kbits / sec, 46 kbits / sec, 47 kbits / sec, 48 kbits / sec, 49 kbits / sec, 50 kbits / sec, 51 kbits / sec, 52 kbits / sec, 53 kbits / sec, 54 kbits / sec, 55 kbits / sec, 56 kbits / sec, 57 kbits / sec, 58 kbits / sec, 59 kbits / sec, 60 kbits / sec, 61 kbits / sec, 62 kbits / sec, 63 kbits / sec, or 64 kbits / sec. The required bit rate may also be some other value.

[0057] In the situation shown in Fig. 3(c), after the state of Fig. 3(b), the surplus bitrate is distributed to the arbitrary bitrate audio signal. That is, the surplus bitrate is distributed to the audio object C and each channel of the 22.2ch audio signal. As a result, no bitrate remains in the total bitrate frame. In the example shown here, the bitrate distributed to each channel of the arbitrary bitrate audio signal is different. However, the rule for distributing the bitrate to the arbitrary bitrate audio signal may differ depending on the audio encoding method used. As an example, the same amount of bitrate may be distributed to each channel of all arbitrary bitrate audio signals.

[0058] FIG. 4 is a functional block diagram showing a first example of a schematic functional configuration of the audio decoding device according to the present embodiment. The audio decoding device 51 has a function capable of handling the bit stream output by the audio encoding device 1. The audio decoding device 51 receives the bit stream output from the audio encoding device 1 and decodes the encoded audio. As shown in the figure, the audio decoding device 51 includes a separation unit 71, an audio metadata decoding unit 72, an audio signal decoding unit 75, an operation unit 81, and an audio rendering unit 82. Each functional unit of the audio decoding device 51 is also realized, for example, by using an electronic circuit. Each functional unit may include a storage means such as a semiconductor memory or a magnetic hard disk device, as necessary. Each function may be realized by a computer and software. The audio decoding device 51 may be realized as a part of a dedicated television receiver or an audio playback device. The audio decoding device 51 may also be realized by using a personal computer, a smartphone, or other information terminal device. The audio decoding device 51 having the above configuration receives a bit stream as an input, decodes an audio signal based on the bit stream, renders the audio signal, and outputs a reproduced sound. The functions of each functional unit are as follows.

[0059] The separation unit 71 obtains a bit stream input to the audio decoding device 51 and separates the bit stream. Specifically, the separation unit 71 separates the bit stream into an encoded audio signal and encoded audio metadata. The separation unit 71 passes the encoded audio signal after separation to the audio signal decoding unit 75 and passes the encoded audio metadata to the audio metadata decoding unit 72.

[0060] The audio metadata decoding unit 72 decodes the encoded audio metadata and passes the decoded audio metadata to an audio rendering unit 82 .

[0061] The audio signal decoding unit 75 decodes the encoded audio signal, and passes the audio signal resulting from the decoding to an audio rendering unit 82.

[0062] The operation unit 81 transmits instruction information related to rendering to the audio rendering unit 82. This instruction is based on, for example, a user operation (for example, an operation related to the output volume of the audio signal of each channel (including the case of an audio object), a playback position, etc.).

[0063] The audio rendering unit 82 renders the audio signal passed from the audio signal decoding unit 75 in accordance with the audio metadata received from the audio metadata decoding unit 72 and the instruction information received from the operation unit 81. Here, audio rendering is a process of adjusting the audio signal to be output for the actual output environment (the configuration and arrangement of speakers, etc.) so that the audio of each channel is played back at a predetermined volume and at a predetermined playback position. The audio rendering unit 82 outputs the playback sound, which is the result of the rendering, to the outside.

[0064] The audio decoding device shown in FIG. 5 may be configured. FIG. 5 is a functional block diagram showing a second example of a schematic functional configuration of the audio decoding device according to the present embodiment. This audio decoding device 52 has a function that can handle the bit stream output by the audio encoding device 1. The audio decoding device 52 includes a separation unit 71, an audio metadata decoding unit 73, an audio signal decoding unit 76, an operation unit 81, and an audio rendering unit 83. Each of these functional units is also realized by an electronic circuit or the like. Each functional unit may be realized by a computer and a program. The audio decoding device 52 having the above configuration obtains a bit stream as an input, decodes an audio signal based on the bit stream, renders the audio signal, and outputs a reproduced sound. The functions of each functional unit are as follows.

[0065] Similar to the first example of FIG. 4, the separator 71 separates the bit stream into an encoded audio signal and encoded audio metadata.

[0066] The audio metadata decoding unit 73 decodes the encoded audio metadata. The audio metadata decoding unit 73 performs the decoding in a manner corresponding to the encoding method of the input code string. The audio metadata decoding unit 73 passes the audio metadata resulting from the decoding to the audio rendering unit 83. The audio metadata decoding unit 73 also passes at least a portion of the audio metadata resulting from the decoding to the audio signal decoding unit 76.

[0067] The audio signal decoding unit 76 decodes the encoded audio signal. The audio signal decoding unit 76 performs the decoding by a method according to the encoding method of the input code string. More specifically, the audio signal decoding unit 76 refers to the audio metadata passed from the audio metadata decoding unit 73, and decodes the encoded constrained bit-rate audio signal and the encoded arbitrary bit-rate audio signal. The audio signal decoding unit 76 passes the decoded results, the constrained bit-rate audio signal and the arbitrary bit-rate audio signal, to the audio rendering unit 83.

[0068] The operation unit 81 transmits instruction information relating to rendering to the audio rendering unit 83, similarly to the first example of FIG.

[0069] The audio rendering unit 83 renders the audio signal according to the audio metadata passed from the audio metadata decoding unit 73 and the instruction information (instructions by user operation) passed from the operation unit 81. Specifically, the audio rendering unit 83 performs rendering using the constrained bit-rate audio signal and the arbitrary bit-rate audio signal. The audio rendering unit 83 combines the rendering results of the constrained bit-rate audio signal and the arbitrary bit-rate audio signal, and outputs the reproduced sound to the outside.

[0070] [Second embodiment] Next, a second embodiment of the present invention will be described. Note that the matters already described in the previous embodiment may not be described below. Here, the matters specific to this embodiment will be mainly described. In this second embodiment, the audio to be encoded is an audio composed of background sounds (also called "beds") and audio objects. As the background sound, for example, 22.2ch audio may be used as the audio format of the background sound.

[0071] Fig. 6 is a functional block diagram showing a schematic functional configuration of an audio encoding device according to this embodiment. As shown in the figure, the audio encoding device 2 includes a separation unit 21, a classification unit 23, an audio metadata encoding unit 24, an audio signal encoding unit 41, a multiplexing unit 25, and an audio metadata acquisition unit 90. The audio signal encoding unit 41 includes an audio object bit rate distribution unit 43, a background sound bit rate distribution unit 44, an encoding unit 46, an encoding unit 47, and a combining unit 48. The functions of each functional unit are as follows.

[0072] The separation unit 21 separates the input audio signal into audio metadata and an audio signal, as in the first embodiment. The separation unit 21 passes the separated audio signal to the classification unit 23. The separation unit 21 also passes the separated audio metadata to the classification unit 23, the audio signal encoding unit 41, and the audio metadata encoding unit 24. The separation unit 21 acquires audio metadata from the outside, and can also be considered to have the function of an audio metadata acquisition unit.

[0073] The classification unit 23 classifies the audio signals of the individual channels included in the audio signal passed from the separation unit 21 into audio signals of audio objects and audio signals of background sound while referring to the passed acoustic metadata. The classification unit 23 passes these audio signals to the audio signal encoding unit 41. Specifically, the classification unit 23 passes the audio signals of each audio object to the encoding unit 46, and passes the audio signals of each channel of the background sound to the encoding unit 47.

[0074] The audio metadata encoder 24 encodes the audio metadata for transmission in the same manner as in the first embodiment. The audio metadata encoder 24 passes the encoded audio metadata to the multiplexer 25.

[0075] The audio signal encoding unit 41 encodes and outputs the audio signal passed from the classification unit 23. The audio signal encoding unit 41 encodes the audio signal of the audio object and the audio signal of the background sound. The audio signal encoding unit 41 passes the encoded audio signal to the multiplexing unit 25. The functional configuration of the audio signal encoding unit 41 in more detailed units will be described later.

[0076] As in the first embodiment, the multiplexing unit 25 multiplexes, for each time window (frame), the encoded audio metadata output from the audio metadata encoding unit 24 and the encoded audio data output from the audio signal encoding unit 41. The multiplexing unit 25 outputs one bit stream obtained by multiplexing to the outside.

[0077] The acoustic metadata acquisition unit 90 acquires acoustic metadata from an external source. In other words, the acoustic metadata acquisition unit 90 can acquire acoustic metadata via a route other than the input signal acquired by the separation unit 21. The acoustic metadata acquisition unit 90 passes the acquired acoustic metadata to the classification unit 23, the audio signal encoding unit 41, and the acoustic metadata encoding unit 24.

[0078] A more detailed functional configuration of the audio signal encoding unit 41 is as follows: For encoding, the audio signal encoding unit 41 cuts out the received audio signal of an audio object and the audio signal of the background sound by using a predetermined time window (frame).

[0079] The audio object bit rate distribution unit 43 distributes the audio object bit rate to each audio signal of the audio object by referring to the received audio metadata. The audio object bit rate distribution unit 43 passes information on the distributed bit rate to the encoding unit 46. The background sound bit rate distribution unit 44 distributes the background sound bit rate to the audio signal of the background sound while similarly referring to the audio metadata. The background sound bit rate distributes the surplus bit rate after the audio object bit rate is distributed to the audio signal of the audio object, as will be described later (FIG. 7). The background sound bit rate distribution unit 44 passes information on the distributed bit rate to the encoding unit 47. The method of distributing the bit rate by the audio object bit rate distribution unit 43 and the background sound bit rate distribution unit 44 will be described later with reference to FIG.

[0080] The encoding unit 46 encodes the audio signal of the audio object classified by the classification unit 23, and outputs a code string. The encoding unit 46 performs lossy encoding, and can appropriately adjust the bit rate of the code to be output. The encoding unit 46 performs encoding for each channel according to the bit rate information for each channel passed from the audio object bit rate distribution unit 43. The encoding unit 46 may perform encoding using any encoding method, such as MPEG-2 AAC, MPEG-4 AAC, MPEG-D USAC, MPEG-H 3DA, MPEG-I IAC, AC-3, or AC-4.

[0081] The encoding unit 47 encodes the audio signal of the background sound classified by the classification unit 23, and outputs a code string. The encoding unit 47 performs lossy encoding, and can appropriately adjust the bit rate of the code to be output. The encoding unit 47 performs encoding for each channel according to the bit rate information for each channel passed from the background sound bit rate distribution unit 44. The encoding unit 47 may perform encoding using any encoding method, such as MPEG-2 AAC, MPEG-4 AAC, MPEG-D USAC, MPEG-H 3DA, MPEG-I IAC, AC-3, or AC-4.

[0082] The combining unit 48, like the combining unit 38 in the first embodiment, combines the coded audio signal of the audio object output from the encoding unit 46 and the coded audio signal of the background sound output from the encoding unit 47 to unify them. Here too, the combining unit 48 may combine the signals in any manner. The combining unit 48 outputs the combined audio signal. In other words, the combining unit 48 passes the combined audio signal to the multiplexing unit 25.

[0083] Fig. 7 is a schematic diagram showing rules and procedures for distributing bit rates by the audio encoding device 2. Bit rate distribution in this embodiment is similar to that in the first embodiment (Fig. 3). However, in this embodiment (Fig. 7), an audio object bit rate distribution unit 43 distributes the bit rate for audio objects. Also, a background sound bit rate distribution unit 44 distributes the bit rate for background sound (22.2ch background sound). The configuration of the audio signal assumed here is as follows. That is, the audio signal to be encoded has a 22.2ch audio signal (24 channels) for background sound and four audio objects (4 channels).

[0084] FIG. 7(a) shows the situation before the bitrate is distributed to the channels. FIG. 7(b) shows the situation when the bitrate for audio objects is first distributed to each of the four audio objects. FIG. 7(c) shows the state after that when the bitrate for background sound is distributed to the 22.2ch background sound (24 channels). Note that in FIG. 7 as well, the 24 boxes (dotted lines) for the 22.2ch background sound and the 4 boxes (dotted lines) for the audio objects correspond to the bitrates to be assigned to the respective channels. The box labeled "total bitrate" corresponds to the overall transmission capacity used by the transmission medium to transmit the audio signal of a certain program.

[0085] In the situation in Fig. 7(a), the total bitrate has not yet been distributed anywhere. In other words, the bitrate of both the 22.2ch background sound frame and the audio object frame is zero.

[0086] In the situation shown in FIG. 7(b), only the audio object bit rate is distributed to the channels of the audio objects. That is, the audio object bit rate is distributed to each of the four audio objects (objects A to D). The hatched parts of the frames of the audio objects A, B, C, and D correspond to the assigned audio object bit rate. And, the bit rate is not distributed to each channel of the 22.2ch background sound. Note that the specific numerical examples of the bit rate (audio object bit rate) per audio object according to the audio encoding method (audio encoding method is MPEG-H 3DA, MPEG-4 AAC, MPEG-2 AAC, etc.) are as shown in the explanation of FIG. 3 of the first embodiment. In this FIG. 7(b), the sum of the audio object bit rates distributed to the audio objects A, B, C, and D is subtracted from the preset total bit rate (the white part in the frame (dotted line) of the total bit rate is the subtracted amount). The remaining bitrate in this state is the background sound bitrate to be allocated next (FIG. 7(c)).

[0087] In the situation shown in FIG. 7(c), after the state in FIG. 7(b), the background sound bitrate is allocated to the background sound audio signal. That is, the background sound bitrate is allocated to each channel of the 22.2ch background sound. As a result, no bitrate remains in the total bitrate frame. In the example shown here, the bitrate allocated to each channel of the background sound audio signal is different. However, the rule for allocating the bitrate to the background sound audio signal may differ depending on the audio encoding method used. As an example, the same amount of bitrate may be allocated to all channels of the background sound audio signal.

[0088] 8 is a functional block diagram showing a schematic functional configuration of an audio decoding device according to this embodiment. The audio decoding device 61 is a device that performs a decoding process on a bit stream output by the audio encoding device 2 of this embodiment. The audio decoding device 61 includes a separation unit 71, an audio metadata decoding unit 74, an audio signal decoding unit 77, an operation unit 81, and an audio rendering unit 84.

[0089] As in the first embodiment, the separation unit 71 separates the bit stream input to the audio decoding device 61 into encoded audio metadata and an encoded audio signal. The separation unit 71 passes the encoded audio metadata obtained by separation to an audio metadata decoding unit 74. The separation unit 71 also passes the encoded audio signal obtained by separation to an audio signal decoding unit 77.

[0090] The audio metadata decoding unit 74 decodes the encoded audio metadata, similar to the audio metadata decoding unit 73 in the first embodiment. The audio metadata decoding unit 74 passes the audio metadata resulting from the decoding to an audio rendering unit 84. The audio metadata decoding unit 74 also passes the audio metadata to an audio signal decoding unit 77.

[0091] The audio signal decoding unit 77 decodes the audio signal while referring to the audio metadata decoded by the audio metadata decoding unit 74. Specifically, the audio signal decoding unit 77 decodes the audio signals of each channel of the audio object and the background sound. The audio signal decoding unit 77 passes the decoded audio signals of each channel of the audio object and the background sound to the audio rendering unit 84.

[0092] The operation unit 81 transmits instruction information relating to rendering to the audio rendering unit 84, as in the first embodiment.

[0093] Similar to the audio rendering units 82 and 83 in the first embodiment, the audio rendering unit 84 renders the audio signal (audio object and background sound) passed from the audio signal decoding unit 77 in accordance with the audio metadata received from the audio metadata decoding unit 74 and the instruction information received from the operation unit 81. The audio rendering unit 84 combines the rendering results of the audio object and the background sound, and outputs the reproduced sound to the outside.

[0094] [Third embodiment] Next, a third embodiment of the present invention will be described. Note that the description of the matters already described in the previous embodiments may be omitted below. Here, the description will focus on matters unique to this embodiment.

[0095] In the third embodiment, a modified method of bitrate distribution performed by the audio encoding device in the first embodiment is adopted. In the first embodiment, the surplus bitrate distribution unit 34 distributes all of the surplus bitrate after the mandatory bitrate distribution unit 33 distributes the mandatory bitrate to the arbitrary bitrate audio signal. In the third embodiment, the optimal bitrate distribution unit 35 distributes an optimal amount of the bitrate (for convenience, referred to as the subtraction bitrate) remaining after the mandatory bitrate distribution unit 33 distributes the mandatory bitrate to the arbitrary bitrate audio signal. Then, the surplus bitrate distribution unit 34 distributes the surplus bitrate to the arbitrary bitrate audio signal. Note that the surplus bitrate distribution unit 34 may distribute the surplus bitrate not only to the arbitrary bitrate audio signal but also to the mandatory bitrate audio signal.

[0096] 9 is a functional block diagram showing a schematic functional configuration of the audio encoding device according to this embodiment. As shown in the figure, the audio encoding device 3 includes a separation unit 21, a classification unit 22, an audio metadata encoding unit 24, an audio signal encoding unit 32, a multiplexing unit 25, and an audio metadata acquisition unit 90. The functions of each unit of the separation unit 21, the classification unit 22, the audio metadata encoding unit 24, the multiplexing unit 25, and the audio metadata acquisition unit 90 are the same as those described in the first embodiment. Therefore, the following description will be centered on the audio signal encoding unit 32, which has a function unique to this embodiment.

[0097] The audio signal encoding unit 32 includes a required bit rate distribution unit 33, a surplus bit rate distribution unit 34, an optimal bit rate distribution unit 35, an encoding unit 36, an encoding unit 37, and a combining unit 38. The audio signal encoding unit 32 extracts and encodes the constrained bit rate audio signal and the arbitrary bit rate audio signal using a predetermined time window (frame). The functions of each functional unit constituting the audio signal encoding unit 32 are as follows.

[0098] The mandatory bitrate distribution unit 33 distributes the mandatory bitrate to the constrained bitrate audio signal while referring to the audio metadata. The amount of the mandatory bitrate that the mandatory bitrate distribution unit 33 distributes to the constrained bitrate audio signal is as already described in the first embodiment. The mandatory bitrate distribution unit 33 transmits information on the distributed bitrate to the encoding unit 36 ​​for each channel.

[0099] The optimal bit rate distribution unit 35 distributes the optimal bit rate to the arbitrary bit rate audio signal. The amount of the optimal bit rate distributed by the optimal bit rate distribution unit 35 will be described later with reference to Fig. 11. The optimal bit rate distribution unit 35 transmits information on the distributed bit rate to the encoding unit 37 for each channel.

[0100] The surplus bitrate distribution unit 34 distributes the surplus bitrate to each channel (including audio objects) of the constrained bitrate audio signal and the arbitrary bitrate audio signal. In this embodiment, the surplus bitrate is the surplus remaining after subtracting the sum of the bitrates distributed by the required bitrate distribution unit 33 and the sum of the bitrates distributed by the optimal bitrate distribution unit 35 from the total bitrate. If the result of this subtraction is zero, there is no surplus bitrate to be distributed. As an example, the surplus bitrate distribution unit 34 may distribute the surplus bitrate equally to all channels (including audio objects). As another example of a distribution method, the surplus bitrate distribution unit 34 may distribute the surplus bitrate mainly to channels (including audio objects) that have already been distributed a small bitrate. Furthermore, the surplus bitrate distribution unit 34 may distribute the surplus bitrate according to yet another rule. The surplus bit rate distribution unit 34 transmits information about the distributed bit rate to the encoding unit 36 ​​and the encoding unit 37 for each channel.

[0101] The encoding unit 36 ​​encodes the constrained bit-rate audio signal for each channel according to the distributed bit rate, and outputs a code string. When encoding the constrained bit-rate audio signal, the encoding unit 36 ​​calculates the sum of the bit rate distributed by the required bit-rate distribution unit 33 and the bit rate distributed by the surplus bit-rate distribution unit 34 for each channel, and generates a code at the resulting bit rate.

[0102] The encoding unit 37 encodes the arbitrary bit-rate audio signal for each channel according to the distributed bit rate, and outputs a code string. When encoding the arbitrary bit-rate audio signal, the encoding unit 37 calculates the sum of the bit rate distributed by the optimal bit rate distribution unit 35 and the bit rate distributed by the surplus bit rate distribution unit 34 for each channel, and generates a code at that bit rate.

[0103] As in the first embodiment, the combining unit 38 combines the encoded constrained bit-rate audio signal output from the encoding unit 36 ​​and the encoded arbitrary bit-rate audio signal output from the encoding unit 37 to unify them.

[0104] 10 and 11 are schematic diagrams showing rules and procedures when the audio encoding device 3 of this embodiment distributes bit rates. The configuration of the audio signal assumed here is the same as that of the first embodiment. That is, the audio signal to be encoded has a 22.2ch audio signal (24 channels) and four audio objects (4 channels). Furthermore, of the four audio objects, three audio objects are constrained bit rate audio signals and one audio object is an arbitrary bit rate audio signal. That is, audio objects A, B, and D are constrained bit rate audio signals, and audio object C is an arbitrary bit rate audio signal. Furthermore, all channels included in the 22.2ch audio signal are arbitrary bit rate audio signals.

[0105] Fig. 10(a) shows the situation before the bitrates are distributed to the channels. Fig. 10(b) shows the situation when the required bitrates are first distributed to the channels. Fig. 11(c) shows the situation when the optimum bitrates are further distributed to the channels. And Fig. 11(d) shows the situation when the surplus bitrates are further distributed to the channels. In the figure, the 24 boxes (dotted lines) for 22.2ch audio and the 4 boxes (dotted lines) for audio objects correspond to the bitrates to be assigned to each channel. Moreover, the box labeled "total bitrate" represents the overall transmission capacity used by the transmission medium to transmit the audio signal of a certain program.

[0106] In the situation in Fig. 10(a), the entire bit rate has not yet been distributed anywhere. In other words, the bit rate of both the 22.2ch audio frame and the audio object frame is zero.

[0107] In the situation shown in Fig. 10(b), only the mandatory bitrate is distributed only to the channels of the constrained bitrate audio signal. In other words, of the four audio objects, the mandatory bitrate is allocated to audio objects A, B, and D, which are constrained bitrate audio signals. And no bitrate distribution is performed for audio object C, which is an arbitrary bitrate audio signal. Also, no bitrate distribution is performed for each channel of the 22.2ch audio. The bitrate distribution up to this point is the same as that at the point in time when the mandatory bitrate distribution in the first embodiment is completed, that is, the state shown in Fig. 3(b).

[0108] Next, in the situation shown in FIG. 11(c), after the state of FIG. 10(b), the optimum bit rate is distributed to the arbitrary bit rate audio signal. That is, the optimum bit rate is distributed to the audio object C and each channel of the 22.2ch audio signal. In this third embodiment, unlike the first embodiment, all surplus bit rates after the distribution of the mandatory bit rate are not distributed to the arbitrary bit rate audio signal at this stage. The amount of bit rate (transmission capacity) to be distributed to the arbitrary bit rate audio signal as the optimum bit rate is appropriately determined according to the arbitrary bit rate audio signal of each channel. The amount of the optimum bit rate may be calculated by a predetermined calculation procedure according to the properties of each channel, may be set in a parameter file or the like as a setting value for each channel, or may be determined by other methods. In addition, the bit rates distributed to each channel of the arbitrary bit rate audio signal may be different from each other or may be the same. The value of the optimum bit rate for each channel may be determined according to the audio encoding method used. In the state of Fig. 11(c), the amount allocated as the optimal bitrate is subtracted from the total bitrate amount. However, in this state, the total bitrate amount has not yet become zero, and there is still a surplus remaining.

[0109] Finally, in the situation shown in Fig. 11(d), after the state of Fig. 11(c), the surplus bitrate is further distributed to the audio signals of all channels. In the state of Fig. 11(d), the surplus bitrate (the bitrate that remained undistributed at the stage of Fig. 11(c)) is distributed equally to each of the constrained bitrate audio signal and the arbitrary bitrate audio signal. Also, the state is shown in which all bitrates have been distributed. The distributed surplus bitrate is the amount indicated by the small solid arrow line in Fig. 11(d). Also, as a result of distributing the surplus bitrate, no bitrate remains in the total bitrate frame.

[0110] In the example shown in Fig. 11(d), the surplus bitrate is distributed equally to each channel, but the amount of distribution may differ for each channel when distributing the surplus bitrate. For example, a different distribution rule may be set to distribute more bitrate to channels that have already received less bitrate.

[0111] 12 and 13 are flowcharts showing the procedure of the encoding process by the audio encoding device according to this embodiment. Figures 12 and 13 are connected by a flowchart connector. The following description will be given along with this flowchart.

[0112] First, in step S11 of Fig. 12, the audio encoding device 3 acquires a program audio signal. This program audio signal may include audio metadata. There is also a case where this program audio signal does not include audio metadata.

[0113] Next, in step S12, the separation unit 21 separates the audio signal from the program audio signal acquired in step S11 into an audio metadata signal and an audio signal. The separation unit 21 further passes the separated audio signal to the classification unit 22.

[0114] Next, in step S13, the audio encoding device 3 determines whether or not the signal acquired in step S11 includes audio metadata. If audio metadata is included (step S13: YES), the process proceeds to step S14. If audio metadata is not included (step S13: NO), the process proceeds to step S15.

[0115] If the process proceeds to step S14, in this step the audio encoding device 3 acquires acoustic metadata from the signal acquired in step S11. This acoustic metadata is passed to the classification unit 22, the audio signal encoding unit 31, and the acoustic metadata encoding unit 24. When the process of this step ends, the process proceeds to step S16.

[0116] When the process proceeds to step S15, in this step the audio encoding device 3 acquires audio metadata provided from an external source (outside the audio encoding device 3). This audio metadata is passed to the classification unit 22, the audio signal encoding unit 31, and the audio metadata encoding unit 24. When the process of this step ends, the process proceeds to step S16.

[0117] In this flowchart, step S13 branches to either step S14 or S15. As a result, the audio encoding device 3 uses either the audio metadata extracted from the signal acquired in step S11 or the audio metadata provided from an external source. However, the audio encoding device 3 may use both of the audio metadata acquired by these means.

[0118] Next, the process proceeds to step S16, where the classification unit 22 classifies the audio signal passed from the separation unit 21. Specifically, the classification unit 22 classifies the passed audio signal into an audio signal with a constrained bit rate and an audio signal with an arbitrary bit rate. The classification unit 22 passes both the audio signal with the constrained bit rate and the audio signal with the arbitrary bit rate to the audio signal encoding unit 32. More specifically, the classification unit 22 passes the audio signal with the constrained bit rate to an encoding unit 36 ​​in the audio signal encoding unit 32. In addition, the classification unit 22 passes the audio signal with the arbitrary bit rate to an encoding unit 37 in the audio signal encoding unit 32.

[0119] Next, in step S17, the mandatory bit rate distribution unit 33 distributes the bit rate (mandatory bit rate) for the constrained bit rate audio signal. The mandatory bit rate is as already explained.

[0120] Next, in step S18, the optimum bit rate distribution unit 35 distributes the bit rate (optimum bit rate) for the arbitrary bit rate audio signal. The optimum bit rate is as already explained.

[0121] Next, in step S19, the surplus bitrate distribution unit 34 judges whether or not there is a surplus bitrate. If there is a surplus bitrate (step S19: YES), the process proceeds to the next step S20 to execute a process of distributing the surplus bitrate. If there is no surplus bitrate (step S19: NO), the process skips steps S20 and S21 and proceeds to step S22. The surplus bitrate has already been described.

[0122] The steps that follow are shown in FIG. Next, when the process proceeds to step S20, in this step the surplus bit rate distribution unit 34 distributes the surplus bit rate to the constrained bit rate audio signal. Then, in step S21, the surplus bit rate distribution unit 34 distributes the surplus bit rate to the arbitrary bit rate audio signal.

[0123] The order of the process of step S21 and the process of step S22 may be reversed. In addition, when the surplus bit rate is allocated to the constrained bit rate audio signal first (when step S20 is executed first), if there is no surplus bit rate available for allocation to the arbitrary bit rate audio signal as a result of allocating the surplus bit rate to the constrained bit rate audio signal, the allocation of the surplus bit rate to the arbitrary bit rate audio signal may be canceled. Conversely, when the surplus bit rate is allocated to the arbitrary bit rate audio signal first (when step S21 is executed first), if there is no surplus bit rate available for allocation to the constrained bit rate audio signal after the surplus bit rate has been allocated to the arbitrary bit rate audio signal, the allocation of the surplus bit rate to the constrained bit rate audio signal may be canceled. Also, even when the surplus bit rate is allocated to the constrained bit rate audio signal first (when step S20 is executed first), the surplus bit rate may be allocated to the constrained bit rate audio signal only if there is a sufficient amount of surplus bit rate that can be allocated to the arbitrary bit rate audio signal. Conversely, even when the surplus bit rate is allocated to the arbitrary bit rate audio signal first (when step S21 is executed first), the surplus bit rate may be allocated to the arbitrary bit rate audio signal only if there is a sufficient amount of surplus bit rate that can be allocated to the constrained bit rate audio signal.

[0124] Regardless of whether the processes of steps S20 and S21 described above are performed or skipped, the bit rates of the constrained bit-rate audio signal and the arbitrary bit-rate audio signal have been determined by the time immediately before the process of step S22.

[0125] Then, in step S22, the encoding unit 36 ​​encodes the constrained bit-rate audio signal in accordance with the bit rate allocated above (the bit rate assigned to the constrained bit-rate audio signal).

[0126] Then, in step S23, the encoding unit 37 encodes the arbitrary bit-rate audio signal in accordance with the bit rate allocated above (the bit rate assigned to the arbitrary bit-rate audio signal).

[0127] Next, in step S24, the combining unit 38 combines the encoded constrained bit-rate audio signal (result of the process in step S22) and the encoded arbitrary bit-rate audio signal (result of the process in step S23) into a single signal.

[0128] Then, in step S25, the acoustic metadata encoding unit 24 encodes the acoustic metadata.

[0129] Next, in step S26, the multiplexing unit 25 multiplexes the encoded audio signal (the result of the combination in step S24) and the encoded acoustic metadata (the result of the process in step S25).

[0130] Then, in step S27, the multiplexing unit 25 outputs the bit stream obtained as a result of the multiplexing in step S26 to the outside. This bit stream can be transmitted via a transmission medium (broadcast signal, communication signal, etc.) as necessary.

[0131] The audio decoding device used in this embodiment may be the audio decoding device 51 (FIG. 4) or the audio decoding device 52 (FIG. 5) described in the first embodiment.

[0132] [Fourth embodiment] Next, a fourth embodiment of the present invention will be described. Note that the description of the matters already described in the previous embodiments may be omitted below. Here, the description will focus on matters unique to this embodiment.

[0133] The fourth embodiment employs a modified method of distributing bitrates performed by the audio encoding device of the second embodiment. In the second embodiment, the background sound bitrate distribution unit 44 distributes all of the surplus bitrates remaining after the audio object bitrate distribution unit 43 distributes the audio object bitrates to the background sound audio signal. In the fourth embodiment, the background sound bitrate distribution unit 44 distributes an optimal amount of the bitrates remaining after the audio object bitrate distribution unit 43 distributes the audio object bitrates (for convenience, referred to as the post-subtraction bitrates) to the background sound audio signal. Then, the surplus bitrate distribution unit 45 distributes the surplus bitrate to the background sound audio signal. Note that the surplus bitrate distribution unit 45 may distribute the surplus bitrate not only to the background sound audio signal but also to the audio object audio signal.

[0134] Fig. 14 is a functional block diagram showing a schematic functional configuration of the audio encoding device according to this embodiment. As shown in the figure, the audio encoding device 4 includes a separation unit 21, a classification unit 23, an audio metadata encoding unit 24, an audio signal encoding unit 42, a multiplexing unit 25, and an audio metadata acquisition unit 90. The functions of each unit of the separation unit 21, the classification unit 23, the audio metadata encoding unit 24, the multiplexing unit 25, and the audio metadata acquisition unit 90 are the same as those described in the second embodiment. Therefore, the following description will be focused on the audio signal encoding unit 42, which has a function unique to this embodiment.

[0135] The audio signal encoding unit 42 includes an audio object bit rate distribution unit 43, a background sound bit rate distribution unit 44, a surplus bit rate distribution unit 45, an encoding unit 46, an encoding unit 47, and a combining unit 48. The audio signal encoding unit 42 extracts and encodes the audio signals of each audio object and each channel of the 22.2ch background sound using a predetermined time window (frame). The functions of each functional unit constituting the audio signal encoding unit 42 are as follows.

[0136] The audio object bitrate distribution unit 43 distributes the mandatory bitrate to the audio signal of the audio object while referring to the audio metadata. The amount of the mandatory bitrate that the audio object bitrate distribution unit 43 distributes to the constrained bitrate audio signal is as already explained in the second embodiment. The audio object bitrate distribution unit 43 transmits information on the distributed bitrate to the encoding unit 46 for each channel.

[0137] The background sound bit rate distribution unit 44 distributes the background sound bit rate to the audio signal of each channel of the background sound. The amount of the background sound bit rate distributed by the background sound bit rate distribution unit 44 will be described later with reference to Fig. 16. The background sound bit rate distribution unit 44 transmits information on the distributed bit rate to the encoding unit 47 for each channel.

[0138] The surplus bitrate distribution unit 45 distributes the surplus bitrate to each of the audio signals of the audio object and the background sound. In this embodiment, the surplus bitrate is a surplus remaining after subtracting the sum of the bitrates distributed by the audio object bitrate distribution unit 43 and the sum of the bitrates distributed by the background sound bitrate distribution unit 44 from the total bitrate. If the result of this subtraction is zero, there is no surplus bitrate to be distributed by the surplus bitrate distribution unit 45. As an example of a distribution method, the surplus bitrate distribution unit 45 may distribute the surplus bitrate equally to all channels (including audio objects). As another example of a distribution method, the surplus bitrate distribution unit 45 may distribute the surplus bitrate mainly from channels (including audio objects) that have already been distributed a small bitrate. Furthermore, the surplus bitrate distribution unit 45 may distribute the surplus bitrate according to yet another rule. The surplus bit rate distribution unit 45 transmits information on the distributed bit rate to the encoding unit 46 and the encoding unit 47 for each channel.

[0139] The encoding unit 46 encodes the audio signal of the audio object for each channel according to the distributed bit rate and outputs a code string. When encoding the audio signal of the audio object, the encoding unit 46 calculates the sum of the bit rate distributed by the audio object bit rate distribution unit 43 and the bit rate distributed by the surplus bit rate distribution unit 45 for each channel, and generates a code at the resulting bit rate.

[0140] The encoding unit 47 encodes the background sound audio signal for each channel according to the distributed bit rate, and outputs a code string. When encoding the background sound audio signal, the encoding unit 37 calculates the sum of the bit rate distributed by the background sound bit rate distribution unit 44 and the bit rate distributed by the surplus bit rate distribution unit 45 for each channel, and generates a code at that bit rate.

[0141] As in the second embodiment, the combining unit 48 combines the encoded constrained bit-rate audio signal output from the encoding unit 46 and the encoded arbitrary bit-rate audio signal output from the encoding unit 47 to unify them.

[0142] 15 and 16 are schematic diagrams showing rules and procedures for distributing a bit rate by the audio encoding device 4 of this embodiment. The configuration of the audio signal assumed here is the same as that of the second embodiment. That is, the audio signal to be encoded has 22.2ch background sound (24 channels) and four audio objects (4 channels).

[0143] FIG. 15(a) shows the situation before the bitrate is distributed to the channels. FIG. 15(b) shows the situation when the bitrate for audio objects is first distributed to four audio objects. FIG. 16(c) shows the situation when the bitrate for background sound is further distributed to the 22.2ch background sound channel. FIG. 16(d) shows the situation when the surplus bitrate is further distributed to the channels. In the figure, the 24 boxes (dotted lines) for 22.2ch audio and the four boxes (dotted lines) for audio objects correspond to the bitrates to be assigned to the respective channels. The box labeled "total bitrate" represents the overall transmission capacity used by the transmission medium to transmit the audio signal of a certain program.

[0144] In the situation in Fig. 15(a), the entire bitrate has not yet been distributed anywhere. In other words, the bitrate of both the frame of the 22.2ch background sound and the frame of the audio object is zero.

[0145] In the situation shown in Fig. 15(b), only the audio object bitrate is distributed to the audio objects. In other words, the audio object bitrate is allocated to each of the four audio objects. Note that at this stage, no bitrate distribution has been performed for each channel of the 22.2ch background sound. The bitrate distribution up to this Fig. 15(b) is the same as that at the time when the audio object bitrate distribution in the second embodiment has been completed, i.e., the state in Fig. 7(b).

[0146] Next, in the situation shown in FIG. 16(c), after the state of FIG. 15(b), the background sound bit rate is distributed to the audio signal of each channel of the 22.2ch background sound. In this fourth embodiment, unlike the second embodiment, all surplus after the distribution of the audio object bit rate is not distributed to the audio signal of the 22.2ch background sound at this stage. The amount of bit rate (transmission capacity) distributed to each channel of the 22.2ch background sound as the background sound bit rate is appropriately determined according to the audio signal of each channel. The background sound bit rate may be determined, for example, by a predetermined calculation procedure according to the properties of each channel, or may be set in a parameter file or the like as a setting value for each channel, or may be determined by other methods. In addition, the bit rates distributed to each channel of the 22.2ch background sound may be different from each other or may be the same. The value for each channel may be determined according to the audio encoding method used. In this state of FIG. 16(c), the amount distributed as the background sound bit rate is subtracted from the amount of the total bit rate. However, in this state, the total bitrate amount has not yet reached zero and there is still some surplus remaining.

[0147] Finally, in the situation shown in Fig. 16(d), after the state of Fig. 16(c), the surplus bitrate is further distributed to the audio signals of all channels (for audio objects and background sounds). In the state of Fig. 16(d), the surplus bitrate (the bitrate that remained undistributed at the stage of Fig. 16(c)) is distributed equally to each of all channels. The distributed surplus bitrate is the amount indicated by the small solid arrow line in Fig. 16(d). Moreover, as a result of distributing the surplus bitrate, there is no bitrate remaining in the total bitrate frame.

[0148] In the example shown in Fig. 16(d), the surplus bitrate is distributed equally to each channel, but the amount of distribution may differ for each channel when distributing the surplus bitrate. For example, a different distribution rule may be set to distribute more bitrate to channels that have already received less bitrate.

[0149] The audio decoding device used in this embodiment may be the audio decoding device 61 (FIG. 8) described in the second embodiment.

[0150] In the third embodiment, the processing procedure of the audio encoding device has been described with reference to the flowcharts (FIGS. 12 and 13). In other embodiments, the processing procedure of the audio encoding device is as follows. The processing procedure of the audio encoding device in the first embodiment is similar to that in the third embodiment, except that in the first embodiment, after distributing the mandatory bit rate, all of the surplus bit rate is allocated to the arbitrary bit rate audio signal. The processing procedure of the audio encoding device in the fourth embodiment is similar to that of the third embodiment. However, with respect to the fourth embodiment, the following substitutions are made: A constrained bit-rate audio signal is replaced with an audio signal of an audio object; A free bit-rate audio signal is replaced with an audio signal of background sound; A required bit-rate is replaced with a bit-rate for an audio object; and An optimal bit-rate is replaced with a bit-rate for background sound. The processing procedure of the audio encoding device in the second embodiment is similar to that of the above-mentioned fourth embodiment, except that in the second embodiment, after the bit rate for the audio object is distributed, all of the surplus bit rate (the background sound bit rate) is allocated to the background sound bit rate audio signal.

[0151] The configuration common to the speech encoding devices of the first to fourth embodiments described above is as follows.

[0152] The separator 21 separates an input signal (for example, a program audio signal of a broadcast program) into audio signals (N signals) and audio metadata. The separator 21 also functions as an audio metadata acquisition unit. The audio metadata encoder 24 encodes the audio metadata, so that the audio metadata can be transmitted or recorded together with the encoded audio signal. The classification units (22, 23) refer to the audio metadata and classify (group) audio signals into l signals (priority signals, constrained bitrate audio signals, audio signals of audio objects) that are constrained by a priority bitrate (required bitrate) (p kilobits per second) and m signals (non-priority signals, arbitrary bitrate audio signals, background sound audio signals) (a total of N signals; N=l+m). The audio signal encoding units (31, 32, 41, 42) encode the audio signals. The distribution unit (mandatory bitrate distribution unit, audio object bitrate distribution unit) distributes predetermined priority bitrates (mandatory bitrate, audio object bitrate) (l bits, each p kbits / sec, total (p*l) kbits / sec) to priority signals from a predetermined total bitrate (T kbits / sec). The distribution section (optimum bit rate distribution section) distributes the remaining bit rate (subtracted bit rate) (Tp*l [kilobits per second]) obtained by subtracting the priority bit rate from the total bit rate to non-priority signals. (1) If T-pl ≦ q, distribute T-pl to non-priority signals. The method of distribution to each channel depends on the standard, etc. Here, q is the optimal bit rate determined in advance for each encoding method for the input background sound (m audio signals). (2-1) If T-pl>q, distribute q to the non-priority signal. Then, the distributor (surplus bitrate distributor) distributes the remaining bitrate (surplus), T-pl-q, to ​​the audio signal. For example, if the bitrate is to be distributed evenly, it is distributed in increments of (T-pl-q) / N. (2-2) When T-pl>q, T-pl may be distributed to priority signals in addition to non-priority signals. The multiplexing unit 25 multiplexes the encoded audio signal and the encoded audio metadata and outputs the multiplexed signal as a bit stream.

[0153] The audio decoding devices (51, 52, 61) have the following configuration. The separator 71 separates the acquired bit stream into an encoded audio signal and encoded audio metadata. The audio metadata decoding unit decodes and outputs the audio metadata. The audio signal decoding unit decodes and outputs the audio signal while referring to the decoded audio metadata. The audio rendering unit refers to the acoustic metadata and also refers to instruction information from the operation unit 81, performs appropriate audio rendering of the decoded priority signal and the decoded non-priority signal, and outputs the reproduced sound.

[0154] Other options (including the features described as embodiments) are as follows: (1) The audio metadata input to the audio encoding device does not necessarily have to be multiplexed with the audio signal. The audio signal and the audio metadata may be obtained through different routes (e.g., via broadcast waves, an operating device, program information, the Internet, manual input via a user interface, a recording medium in the audio encoding device, etc.). (2) The audio metadata may include selection information for constrained bit rate audio signals (information indicating which channel's audio signal is a priority signal and which channel's audio is a non-priority signal), required bit rate, selection information for optional bit rate audio signals, optimal bit rate, surplus bit rate, and total bit rate information. (3) If a specific channel in the priority signal (constrained bit-rate audio signal) is judged to be silent in a specific frame, it is possible to cancel the mandatory bit-rate allocation to that channel and allocate it to the optimum bit-rate, the surplus bit-rate, or a channel other than the specific channel in the priority signal (a channel that is not silent). For example, a situation may occur in which object C in Fig. 3(b) is judged to be silent and therefore the priority bit-rate is not allocated. (4) The remaining bit rate, which is obtained by subtracting the required bit rate from the total bit rate, may be distributed to the arbitrary bit rate audio signal without any case distinction (without judgment) based on a predetermined distribution method. (5) The required bit rate and the optimum bit rate may be preset for each encoding method and audio format. For example, in the case of MPEG-H 3D Audio and AC-4, the required bit rates are 32 kbits per second, 33 kbits per second, 34 kbits per second, 35 kbits per second, 36 kbits per second, 37 kbits per second, 38 kbits per second, 39 kbits per second, 40 kbits per second, 41 kbits per second, 42 kbits per second, 43 kbits per second, 44 kbits per second, 45 kbits per second, 46 kbits per second, 47 kbits per second, 48 kbits per second, 49 kbits per second, 50 kbits per second, 51 kbits per second, 52 kbits per second, 53 kbits per second, 54 kbits per second, 55 kbits per second, 56 kbits per second, 57 kbits per second, 58 kbits per second, 59 kbits per second, 60 kbits per second, 61 kbits per second, 62 kbits per second, 63 kbits per second, 64 kbits per second, 65 kbits per second, 66 kbits per second, 67 kbits per second, 68 kbits per second, 69 kbits per second, 70 kbits per second, 71 kbits per second, 72 kbits per second, 73 kbits per second, 74 kbits per second, 75 kbits per second, 76 kbits per second, 77 kbits per second, 78 kbits per second, 79 kbits per second, 80 kbits per second, 81 kbits per second, 82 kbits per second, 83 k The available bit rates are 528 or 768 kbit / s for 22.2ch, 192 or 180 kbit / s for 5.1 surround, 80 or 96 kbit / s for stereo, and 288 or 384 kbit / s for 7.1.4ch. (6) In the audio decoding device, the constrained bit-rate audio signal and the arbitrary bit-rate audio signal may be decoded and rendered separately, or may be decoded and rendered together without distinguishing between them.

[0155] In the above (3), an option was described to cancel the mandatory bit rate allocation to a specific channel in the priority signal (constrained bit rate audio signal) when the specific channel is judged to be silent in a specific frame. To realize this option, the audio coding device may be configured as follows.

[0156] For example, the audio encoding device may determine whether a particular channel (or an object) is silent in a particular frame by determining whether the signal level of the channel is equal to or lower than a predetermined threshold. That is, the audio encoding device may determine that a channel is silent if the signal level of the channel is equal to or lower than a predetermined threshold.

[0157] Alternatively, in order to determine whether a specific channel (or an object) is silent in a specific frame, the audio encoding device may determine whether a value obtained by adding a maximum value of a level amplification amount on the decoder side (or a receiver side including a decoder) to a signal level of the channel is equal to or less than a predetermined threshold. This "level amplification amount" is a value expressed in decibels (dB), for example. Also, the audio encoding device may determine whether a value obtained by multiplying an amplitude of a channel signal by a maximum value of an amplitude amplification factor on the decoder side is equal to or less than a predetermined threshold. Here, the amplification factor is a value obtained by dividing the amplitude of a signal after amplification by the amplitude of a signal before amplification. Note that the audio encoding device acquires information on the maximum value of the amplification amount or amplification factor on the decoder side by an appropriate means by the time of performing encoding. In other words, the audio encoding device may determine that a channel is silent if a value obtained by adding or multiplying a signal level of a channel by the maximum value of the amplification amount or amplification factor is equal to or less than a predetermined threshold.

[0158] In the above, the audio encoding device may gradually reduce the bit rate assigned to a specific channel (which may be an object) according to the signal level of the channel. That is, the audio encoding device can preset a plurality of threshold values of the signal level S as S0, S1, S2, ···, Sn (n ≧ 0). Then, the audio encoding device determines the bit rate R assigned to that channel as follows according to the value of S. That is, When S ≦ S0, R = 0, When S0 < S ≦ S1, R = R1, When S1 < S ≦ S2, R = R2, Similarly hereinafter, When Sn < S, R = Rn. However, in principle, 0 ≦ R1 ≦ R2 ≦ ··· ≦ Rn. In particular, 0 < R1 < R2 < ··· < Rn may also be used. Note that S0 is the threshold value for determining whether it is silent as described above. Even when the channel is not determined to be silent, when the audio encoding device determines to R = any of R1, R2, ···, R(n - 1), the remaining bit rate (the difference from Rn) can be circulated for distribution to other audio signals. Here, S is the signal level of the channel to be encoded, but it may also be the amplitude of the signal as described above, or the product of the amplitude of the signal and the maximum value of the amplification factor on the decoder side, or the sum of the signal level and the maximum value of the amplification amount on the decoder side.

[0159] Alternatively, the audio coding device may receive a control signal from an external source and follow the control signal to determine whether a particular channel (which may be an object) is silent in a particular frame. That is, the audio coding device receives a control signal from an external source, and the control signal is a signal indicating whether a particular channel is silent or not. If the control signal received for a channel indicates that the channel is silent, the audio coding device may determine that the channel is silent. Note that if the control signal indicates that the channel is silent, the audio coding device may determine that the channel is silent regardless of the actual level of the signal of the channel.

[0160] When the audio encoding device determines that a particular frame of a channel (or object) is silent by any of the determination methods described above, the audio signal of that frame of the channel may be replaced with a silent signal. In other words, in this case, the level of the audio signal in that frame of the channel output by the audio encoding device is the minimum value (-∞), that is, silent.

[0161] Alternatively, when the audio encoding device determines that a specific channel is silent in a specific frame, it may output (send) flag information (which may be called a "silence flag", for example) indicating that the channel is silent in the frame. Furthermore, when the audio encoding device outputs the above flag information, the corresponding audio decoding device may be configured as follows. That is, when the audio decoding device receives flag information (silence flag) indicating that a certain channel (object) is silent, for example, in a specific frame, it may cancel the process of decoding and reproducing the audio signal of the channel (object). Alternatively, when the audio decoding device receives flag information (silence flag) indicating that a certain channel (object) is silent, for example, in a specific frame, it may output and reproduce a silent audio signal for the channel (object).

[0162] Furthermore, the configurations in each embodiment can be summarized as follows.

[0163] The speech encoding devices (1, 2, 3, 4) are configured as follows. The audio metadata acquisition unit (audio metadata acquisition unit) acquires audio metadata that at least indicates whether an audio signal is a priority signal or a non-priority signal. The classification units (22, 23) classify the input multiple (channel) audio signals as either priority signals (constrained bit rate audio signals, audio signals of audio objects) or non-priority signals (arbitrary bit rate audio signals, background sound audio signals) by referring to the acoustic metadata. The distribution units (essential bitrate distribution unit 33, surplus bitrate distribution unit 34, optimal bitrate distribution unit 35, audio object bitrate distribution unit 43, background sound bitrate distribution unit 44, surplus bitrate distribution unit 45) distribute bitrates to the audio signals. These distribution units distribute a priority bitrate corresponding to the priority signal out of the total specified bitrate to the priority signal, and distribute a surplus bitrate based on the result of subtracting the sum of the priority bitrates distributed to the priority signals from the total bitrate to the non-priority signal. The audio signal encoding unit encodes each of the audio signals in accordance with the bit rate distributed by the distribution unit.

[0164] The third or fourth embodiment is as follows. After distributing the priority bit rate to the priority signal, the distribution unit distributes a non-priority bit rate corresponding to the non-priority signal out of a post-subtraction bit rate resulting from subtracting the sum of the non-priority bit rates distributed to the priority signals from a total bit rate, to the non-priority signal, and distributes the surplus bit rate resulting from subtracting the sum of the non-priority bit rates distributed to the non-priority signals from the post-subtraction bit rate to the non-priority signal.

[0165] In the third or fourth embodiment, the following may be performed. The distributor distributes the surplus bit rate to the priority signal as well.

[0166] The separation unit 21 receives an input signal including the metadata and the multiple audio signals, separates the input signal into the metadata and the audio signals, and passes the separated audio signals to the classification unit. However, the audio encoding devices (1, 2, 3, 4) may be configured not to have the separation unit 21.

[0167] An acoustic metadata encoding unit (acoustic metadata encoding unit 24) encodes the acquired acoustic metadata. A multiplexing unit 25 multiplexes the encoded audio signal output by the audio signal encoding unit and the encoded audio metadata output by the audio metadata encoding unit, and outputs the multiplexed signal. However, the audio encoding devices (1, 2, 3, 4) may be configured so as not to have the above-mentioned audio metadata encoding unit or multiplexing unit 25.

[0168] The priority signal may be the audio signal of an audio object. The non-priority signal may be an audio signal of channel-based audio, which is an audio format such as 22.2ch audio. The audio format of channel-based audio is, for example, as described in Non-Patent Documents 4, 5, and 6 listed as prior art documents. That is, it is as described in the following documents: Reference: ARIB Standard STD-B59 Version 2.0, 3D Multi-Channel Sound Studio Standard, ARIB, July 2016. Literature: Recommendation ITU-R BS.2051-2, "Advanced sound system for programme production", ITU-R, July 2018. Literature: SMPTE STANDARD SMPTE 2036-2-2008, "Ultra High Definition Television-Audio Characteristics and Audio Channel Mapping for Program Production", SMPTE, July 9, 2008.

[0169] The priority bit rate per channel of the priority signal is a fixed value according to the encoding method executed by the audio signal encoding unit, and is 32 kbits per second, 33 kbits per second, 34 kbits per second, 35 kbits per second, 36 kbits per second, 37 kbits per second, 38 kbits per second, 39 kbits per second, 40 kbits per second, 41 kbits per second, 42 kbits per second, 43 kbits per second, 44 kbits per second, 45 kbits per second, 46 kbits per second, 47 kbits per second, 48 kbits per second, 49 kbits per second, 50 kbits per second, 51 kbits per second, 52 kbits per second, 53 kbits per second, 54 kbits per second, 55 kbits per second, 56 kbits per second, 57 kbits per second, 58 kbits per second, 59 kbits per second, 60 kbits per second, 61 kbits per second, 62 kbits per second, 63 kbits per second, 64 kbits per second, 65 kbits per second, 66 kbits per second, 67 kbits per second, 68 kbits per second, 69 kbits per second, 70 kbits per second, 71 kbits per second, 72 kbits per second, 73 kbits per second, 74 kbits per second, 75 kbits per second, 76 kbits per second, 77 kbits per second, 78 kbits per second, 79 kbits per second, 80 kbits per second, 81 kbits per second, 82 kbits per second, 83 kbits per second, 84 kbits per second, 85 kbit The preferred bit rate may be any of 6 kbit / s, 47 kbit / s, 48 ​​kbit / s, 49 kbit / s, 50 kbit / s, 51 kbit / s, 52 kbit / s, 53 kbit / s, 54 kbit / s, 55 kbit / s, 56 kbit / s, 57 kbit / s, 58 kbit / s, 59 kbit / s, 60 kbit / s, 61 kbit / s, 62 kbit / s, 63 kbit / s, or 64 kbit / s, or other values.

[0170] The audio decoding devices (51, 52, 61) are configured as follows. The separator receives a bitstream and separates the bitstream into an encoded audio signal and encoded audio metadata. The audio metadata decoding unit decodes the encoded audio metadata. The audio signal decoding unit decodes the encoded audio signal. The audio rendering unit performs audio rendering of the audio signal decoded by the audio signal decoding unit based on the audio metadata decoded by the audio metadata decoding unit, and outputs the result as a reproduced sound.

[0171] The audio signal decoding unit may decode each of the priority signal and the non-priority signal included in the encoded audio signal. In this case, the audio rendering unit performs audio rendering of each of the decoded priority signal and non-priority signal based on the audio metadata decoded by the audio metadata decoding unit, and combines the audio rendering result of the priority signal and the audio rendering result of the non-priority signal to output as the reproduced sound. Furthermore, the audio decoding device (51, 52, 61) may further include an operation unit 81 that generates instruction information based on a user's operation, and the audio rendering unit may perform audio rendering of the decoded audio signal also based on the instruction information from the operation unit. The audio decoding device (51, 52, 61) may be configured not to have the operation unit 81.

[0172] At least some of the functions of the audio encoding device and audio decoding device in each of the above-mentioned embodiments can be realized by a computer. In that case, a program for realizing the functions may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the functions. The term "computer system" here includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB memories, and storage devices such as hard disks built into a computer system. The term "computer-readable recording medium" may also include a medium that temporarily and dynamically holds a program, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in such a case. The above program may be a program for realizing some of the above-mentioned functions, or may be a program that can be realized in combination with a program already recorded in the computer system.

[0173] Although a number of embodiments have been described above, the present invention can also be embodied in the following modified examples.

[0174] The bitstream output by the audio encoding device may be written to a recording medium (such as a CD, DVD, Blu-ray disc, hard disk drive, or semiconductor memory) instead of being transmitted to the audio decoding device. The audio decoding device can obtain the bitstream from the recording medium.

[0175] The techniques described in the embodiments may also be applied to multi-channel audio formats other than the 22.2ch audio format illustrated above, object-based audio formats, and AR / VR audio formats.

[0176] According to at least one of the above-described embodiments (and variations), when encoding audio content, etc., composed of multiple audio signals, it is possible to preferentially allocate transmission capacity (or recording capacity) to audio signals for which sound quality degradation is not permitted. This makes it possible to maintain the overall quality of the content, etc., in line with the intentions of the creator of the content, etc. Furthermore, there is no need to change the bitstream syntax of existing audio encoding standards in order to implement this form.

[0177] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Industrial Applicability]

[0178] The present invention can be used, for example, for broadcasting, distribution, recording, circulation, etc. of audio content, but the scope of use of the present invention is not limited to the examples given here. [Explanation of symbols]

[0179] 1,2,3,4 Audio Encoder 21 Separation unit (acoustic metadata acquisition unit) 22,23 Classification section 24 Audio Metadata Encoding Unit 25 Multiplexer 31,32 Audio signal encoding unit 33 Required Bitrate Distribution Section 34 Surplus bitrate distribution section 35 Optimal bitrate distribution section 36,37 Encoding section 37 Encoding section 38 Joint 41, 42 Audio signal encoding unit 43 Bitrate distribution unit for audio objects 44 Background sound bitrate distribution section 45 Surplus bitrate distribution section 46,47 Encoding section 48 Joint 51, 52, 61 Audio decoder 71 Separation part 72, 73, 74 Audio Metadata Decoding Unit 75, 76, 77 Audio signal decoding section 81 Operation section 82, 83, 84 Audio Rendering Section 90 Acoustic Metadata Acquisition Unit

Claims

1. an audio metadata acquisition unit that acquires audio metadata that at least indicates whether an audio signal is a priority signal or a non-priority signal; a classification unit that classifies a plurality of input audio signals as either a priority signal or a non-priority signal by referring to the audio metadata; a distribution unit that distributes a bit rate to the audio signal, the distribution unit distributing a priority bit rate corresponding to the priority signal out of a total specified bit rate to the priority signal, and distributing a surplus bit rate based on a result of subtracting the sum of the priority bit rates distributed to the priority signals from the total bit rate to the non-priority signal; an audio signal encoding unit that encodes each of the audio signals in accordance with the bit rate distributed by the distribution unit; Equipped with the distribution unit distributes the priority bit rate to the priority signal, and then distributes to the non-priority signal a non-priority bit rate corresponding to the non-priority signal out of a post-subtraction bit rate resulting from subtracting the sum of the priority bit rates distributed to the priority signals from a total bit rate, and distributes the surplus bit rate resulting from subtracting the sum of the non-priority bit rates distributed to the non-priority signals from the post-subtraction bit rate to the non-priority signal. Audio coding device.

2. The distributor distributes the surplus bit rate to the priority signal as well.

2. The audio encoding device according to claim 1.

3. An audio metadata acquisition unit that acquires audio metadata that at least indicates whether an audio signal is a priority signal or a non-priority signal; a classification unit that classifies a plurality of input audio signals as either a priority signal or a non-priority signal by referring to the audio metadata; a distribution unit that distributes a bit rate to the audio signal, the distribution unit distributing a priority bit rate corresponding to the priority signal out of a total specified bit rate to the priority signal, and distributing a surplus bit rate based on a result of subtracting the sum of the priority bit rates distributed to the priority signals from the total bit rate to the non-priority signal; an audio signal encoding unit that encodes each of the audio signals in accordance with the bit rate distributed by the distribution unit; Equipped with the distribution unit distributes the priority bitrate to the priority signal, and then distributes to the non-priority signal a non-priority bitrate corresponding to the non-priority signal out of a post-subtraction bitrate resulting from subtracting the sum of the priority bitrates distributed to the priority signals from a total bitrate, and distributes the surplus bitrate resulting from subtracting the sum of the non-priority bitrates distributed to the non-priority signals from the post-subtraction bitrate to the priority signal. Audio coding device.

4. The distributor distributes the surplus bit rate to the non-priority signal as well.

4. The audio coding apparatus according to claim 3.

5. A separation unit that receives an input signal including a plurality of audio signals and audio metadata obtained from an external source, the audio signals at least indicating whether the audio signals are priority signals or non-priority signals, and separates the input signal into the audio metadata and the audio signals; a classification unit that classifies a plurality of input audio signals as either a priority signal or a non-priority signal by referring to the audio metadata; a distribution unit that distributes a bit rate to the audio signal, the distribution unit distributing a priority bit rate corresponding to the priority signal out of a total specified bit rate to the priority signal, and distributing a surplus bit rate based on a result of subtracting the sum of the priority bit rates distributed to the priority signals from the total bit rate to the non-priority signal; an audio signal encoding unit that encodes each of the audio signals in accordance with the bit rate distributed by the distribution unit; A speech encoding device comprising:

6. when it is determined that a specific priority signal among the priority signals is silent, the distribution unit cancels distribution of the priority bit rate to the specific priority signal and allocates the priority bit rate to other audio signals.

6. A speech encoding device according to claim 5.

7. an audio metadata encoding unit that encodes the acquired audio metadata; a multiplexing unit that multiplexes and outputs the encoded audio signal output from the audio signal encoding unit and the encoded audio metadata output from the audio metadata encoding unit; The audio coding device according to claim 1 , further comprising:

8. the priority signal is an audio signal of an audio object, the non-prioritized signal is a channel-based audio signal; 8. A speech coding device according to any one of claims 1 to 7.

9. The priority bit rate per channel of the priority signal is a fixed value according to the encoding method executed by the audio signal encoding unit, and is 32 kbits per second, 33 kbits per second, 34 kbits per second, 35 kbits per second, 36 kbits per second, 37 kbits per second, 38 kbits per second, 39 kbits per second, 40 kbits per second, 41 kbits per second, 42 kbits per second, 43 kbits per second, 44 kbits per second, 45 kbits per second, , 46 kbits per second, 47 kbits per second, 48 kbits per second, 49 kbits per second, 50 kbits per second, 51 kbits per second, 52 kbits per second, 53 kbits per second, 54 kbits per second, 55 kbits per second, 56 kbits per second, 57 kbits per second, 58 kbits per second, 59 kbits per second, 60 kbits per second, 61 kbits per second, 62 kbits per second, 63 kbits per second, or 64 kbits per second, 9. A speech coding device according to any one of claims 1 to 8.

10. Computer, A speech coding device according to any one of claims 1 to 9, A program to function as a

11. a separator for receiving a bitstream and separating the bitstream into an encoded audio signal and encoded audio metadata; an audio metadata decoding unit that decodes the encoded audio metadata; an audio signal decoding unit that decodes the encoded audio signal; an audio rendering unit that performs audio rendering of the audio signal decoded by the audio signal decoding unit based on the audio metadata decoded by the audio metadata decoding unit and outputs the audio signal as a playback sound; Equipped with The audio metadata is obtained by a separation unit in an audio encoding device, which receives an input signal including a plurality of audio signals and audio metadata obtained from an external source, the audio metadata indicating at least whether the audio signals are priority signals or non-priority signals, and separates the input signal into the audio metadata and the audio signals. Audio decoder.

12. the audio signal decoding unit decodes the priority signal and the non-priority signal included in the encoded audio signal, the audio rendering unit performs audio rendering on each of the decoded priority signal and non-priority signal based on the audio metadata decoded by the audio metadata decoding unit, combines the audio rendering result of the priority signal and the audio rendering result of the non-priority signal, and outputs the combined result as the reproduced sound. The audio decoding device according to claim 11.

13. an operation unit that generates instruction information based on a user's operation; Further equipped with The audio rendering unit performs audio rendering of the decoded audio signal based on the instruction information from the operation unit. The audio decoding device according to claim 11 or 12.

14. Computer, A speech decoding device according to any one of claims 11 to 13, A program to function as a

Images