Information processing device, method, and program

Abstract

To enable acquiring higher-quality audio.SOLUTION: An audio signal decoding unit decodes encoded audio data to obtain an audio signal for each object. A metadata decoding unit decodes encoded metadata to obtain multiple metadata for each frame of the audio signal for each object. A gain calculation unit calculates a VBAP gain for the audio signal of each object for each speaker based on the metadata. An audio signal generation unit multiplies the VBAP gain by the audio signal of each object for each speaker, and adds them together to generate audio signals supplied to each speaker. This technology can be applied to decoding devices.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present technology relates to an information processing device, method, and program, and more particularly to an information processing device, method, and program that enable higher quality audio to be obtained. [Background technology]

[0002] Conventionally, the Moving Picture Experts Group (MPEG)-H 3D Audio standard is known, which compresses (encodes) audio signals of audio objects and metadata such as position information of the audio objects (see, for example, Non-Patent Document 1).

[0003] With this technology, the audio signal and metadata of an audio object are encoded and transmitted for each frame. At this time, a maximum of one piece of metadata is encoded and transmitted for each frame of the audio signal of the audio object. In other words, some frames may not contain any metadata.

[0004] The encoded audio signal and metadata are decoded in a decoding device, and rendering is performed based on the audio signal and metadata obtained by the decoding.

[0005] That is, in a decoding device, first, an audio signal and metadata are decoded. As a result of the decoding, for the audio signal, PCM (Pulse Code Modulation) sample values ​​for each sample in a frame are obtained. In other words, PCM data is obtained as the audio signal.

[0006] On the other hand, as for the metadata, the metadata of the representative sample in the frame, specifically the metadata of the last sample in the frame, is obtained.

[0007] Once the audio signal and metadata are obtained in this way, the renderer in the decoding device calculates a VBAP gain using VBAP (Vector Base Amplitude Panning) based on the position information as metadata of the representative sample in the frame so that the sound image of the audio object is localized at the position indicated by the position information. This VBAP gain is calculated for each speaker on the playback side.

[0008] However, as described above, the metadata of an audio object is the metadata of the representative sample in a frame, that is, the last sample in the frame. Therefore, the VBAP gain calculated by the renderer is the gain of the last sample in the frame, and the VBAP gains of the other samples in the frame are not calculated. Therefore, to play back the sound of an audio object, it is necessary to calculate the VBAP gains of samples other than the representative sample of the audio signal.

[0009] Therefore, the renderer calculates the VBAP gain of each sample by interpolation. Specifically, for each speaker, the VBAP gain of the sample in the current frame that is between the VBAP gain of the last sample in the current frame and the VBAP gain of the last sample in the frame immediately preceding the current frame is calculated by linear interpolation.

[0010] In this way, once the VBAP gain of each sample to be multiplied by the audio signal of the audio object is obtained for each speaker, the sound of the audio object can be reproduced.

[0011] That is, in the decoding device, the audio signal of the audio object is multiplied by the VBAP gain calculated for each speaker and supplied to each speaker, thereby reproducing the audio. [Prior art documents] [Non-patent literature]

[0012] [Non-Patent Document 1] ISO / IEC JTC1 / SC29 / WG11 N14747, August 2014, Sapporo, Japan, "Text of ISO / IEC 23008-3 / DIS, 3D Audio" Summary of the Invention [Problem to be solved by the invention]

[0013] However, with the above-mentioned techniques, it is difficult to obtain audio with sufficiently high quality.

[0014] For example, in VBAP, normalization is performed so that the sum of the squares of the calculated VBAP gains for each speaker becomes 1. This normalization causes the localization position of the sound image to be located on the surface of a sphere with a radius of 1, centered at a predetermined reference point in the playback space, for example, the head position of a virtual user viewing content such as a moving image with audio or music.

[0015] However, because the VBAP gains of samples other than the representative sample in a frame are calculated by interpolation, the sum of the squares of the VBAP gains of each speaker for such samples will not be 1. As a result, for samples whose VBAP gains have been calculated by interpolation, the position of the sound image will be shifted in the normal direction of the sphere described above or in the up, down, left, or right directions on the surface of the sphere when audio is played back, as seen from the virtual user. As a result, the sound image position of the audio object will fluctuate within the period of one frame when audio is played back, worsening the sense of positioning and degrading the sound quality.

[0016] In particular, the greater the number of samples that make up one frame, the longer the distance between the last sample position of the current frame and the last sample position of the frame immediately preceding it. This increases the difference between the sum of squares of the VBAP gains of each speaker calculated by interpolation and 1, resulting in greater degradation of sound quality.

[0017] Furthermore, when calculating the VBAP gains of samples other than the representative sample by interpolation, the faster the audio object moves, the greater the difference between the VBAP gain of the last sample in the current frame and the VBAP gain of the last sample in the frame immediately preceding the current frame becomes, making it impossible to accurately render the movement of the audio object, resulting in a deterioration in sound quality.

[0018] Furthermore, in actual content such as sports and movies, scenes change discontinuously. In such cases, audio objects move discontinuously at scene changes. However, if the VBAP gain is calculated by interpolation as described above, the audio object moves continuously for audio during the sample interval for which the VBAP gain is calculated by interpolation, i.e., between the last sample of the current frame and the last sample of the frame immediately preceding the current frame. This makes it impossible to express the discontinuous movement of the audio object through rendering, resulting in degradation of audio quality.

[0019] The present technology has been made in view of such circumstances, and makes it possible to obtain audio with higher sound quality. [Means for solving the problem]

[0020] A decoding device according to a first aspect of the present technology includes an acquisition unit that acquires encoded audio data obtained by encoding an audio signal of frames at a predetermined time interval of an audio object and a plurality of pieces of metadata for the frames, a decoding unit that decodes the encoded audio data, and a rendering unit that performs rendering based on the audio signal obtained by the decoding and the plurality of pieces of metadata.

[0021] A decoding method or program according to a first aspect of the present technology includes steps of obtaining encoded audio data obtained by encoding an audio signal of frames at a predetermined time interval of an audio object and a plurality of pieces of metadata for the frames, decoding the encoded audio data, and performing rendering based on the audio signal obtained by the decoding and the plurality of pieces of metadata.

[0022] In a first aspect of the present technology, encoded audio data obtained by encoding an audio signal of frames at a predetermined time interval of an audio object and multiple pieces of metadata for the frames are obtained, the encoded audio data is decoded, and rendering is performed based on the audio signal obtained by the decoding and the multiple pieces of metadata.

[0023] An encoding device according to a second aspect of the present technology includes an encoding unit that encodes an audio signal of frames at a predetermined time interval of an audio object, and a generating unit that generates a bitstream including encoded audio data obtained by the encoding and multiple pieces of metadata for the frames.

[0024] An encoding method or program according to a second aspect of the present technology includes a step of encoding an audio signal of frames at a predetermined time interval of an audio object, and generating a bitstream including encoded audio data obtained by the encoding and multiple pieces of metadata for the frames.

[0025] In a second aspect of the present technology, an audio signal of frames at a predetermined time interval of an audio object is encoded, and a bitstream is generated that includes encoded audio data obtained by the encoding and multiple pieces of metadata for the frames. [Effects of the Invention]

[0026] According to the first and second aspects of the present technology, audio with higher sound quality can be obtained.

[0027] The effects described here are not necessarily limited to those described herein, and may be any of the effects described in this disclosure. [Brief explanation of the drawings]

[0028] [Figure 1] FIG. 1 is a diagram illustrating a bit stream. [Figure 2] FIG. 1 illustrates an example of the configuration of an encoding device. [Figure 3] 10 is a flowchart illustrating an encoding process. [Figure 4] FIG. 10 is a diagram illustrating an example of the configuration of a decoding device. [Figure 5] 10 is a flowchart illustrating a decoding process. [Figure 6] FIG. 1 illustrates an example of the configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0029] Hereinafter, embodiments to which the present technology is applied will be described with reference to the drawings.

[0030] First Embodiment Overview of this technology The present technology makes it possible to obtain audio with higher sound quality when encoding and transmitting an audio signal of an audio object and metadata such as position information of the audio object, and when decoding the audio signal and metadata to play back audio on the decoding side. Note that hereinafter, an audio object will also be simply referred to as an object.

[0031] In this technology, multiple pieces of metadata, that is, two or more pieces of metadata, are encoded and transmitted for one frame of audio signal.

[0032] Here, the metadata is metadata of samples in a frame of an audio signal, i.e., metadata assigned to the samples. For example, the position of an audio object in space indicated by position information as metadata indicates the position in the playback timing of audio based on the sample to which the metadata is assigned.

[0033] The metadata can be transmitted using one of the following three methods: a number specification method, a sample specification method, and an automatic switching method. When transmitting the metadata, the metadata can be transmitted by switching between these three methods for each frame or object, which is a predetermined time interval.

[0034] (Quantity specification method) First, the number designation method will be described.

[0035] The number specification method is a method in which the number of metadata pieces to be transmitted for one frame is included in the bitstream syntax and a specified number of metadata pieces are transmitted. The information indicating the number of samples that make up one frame is stored in the bitstream header.

[0036] Furthermore, the metadata to be transmitted may be metadata for which sample in one frame is determined in advance, such as the position when one frame is equally divided.

[0037] For example, suppose one frame consists of 2048 samples, and four pieces of metadata are transmitted per frame. In this case, the section of one frame is divided equally by the number of pieces of metadata to be transmitted, and metadata is transmitted for the sample positions at the boundaries of the divided sections. In other words, metadata is transmitted for samples in the frame that are spaced at intervals of the number of samples obtained by dividing the number of samples in one frame by the number of pieces of metadata.

[0038] In this case, metadata is transmitted for the 512th sample, the 1024th sample, the 1536th sample, and the 2048th sample, respectively, from the beginning of the frame.

[0039] In addition, if the number of samples that make up one frame is S and the number of metadata sent per frame is A, then S / 2 (A-1) In other words, within a frame, metadata for a sample position determined by S / 2 (A-1) Metadata for some or all of the samples arranged at sample intervals may be transmitted. In this case, for example, when the number of metadata A is 1, the metadata for the last sample in the frame is transmitted.

[0040] Alternatively, metadata may be transmitted for each sample arranged at a predetermined interval, that is, for each predetermined number of samples.

[0041] (Sample specification method) Next, the sample designation method will be described.

[0042] In the sample specification method, in addition to the metadata number information transmitted in the number specification method described above, a sample index indicating the sample position of each piece of metadata is also stored in the bitstream and transmitted.

[0043] For example, suppose one frame consists of 2048 samples, four pieces of metadata are transmitted per frame, and metadata is transmitted for the 128th, 512th, 1536th, and 2048th samples, respectively, from the beginning of the frame.

[0044] In this case, the bitstream stores metadata number information indicating the number of metadata transmitted per frame ("4"), and sample indices indicating the positions of the 128th, 512th, 1536th, and 2048th samples from the beginning of the frame. For example, the value of the sample index indicating the position of the 128th sample from the beginning of the frame is 128.

[0045] The sample specification method allows metadata of any sample to be sent for each frame, so for example, metadata of samples before and after a scene change can be sent. In this case, discontinuous movement of objects can be expressed by rendering, resulting in high-quality audio.

[0046] (Automatic switching method) Furthermore, the automatic switching method will be described.

[0047] In the automatic switching method, the amount of metadata transmitted for each frame is automatically switched depending on the number of samples that make up one frame, i.e., the number of samples in one frame.

[0048] For example, if one frame contains 1024 samples, metadata for each sample that is spaced every 256 samples within the frame is transmitted. In this example, a total of four pieces of metadata are transmitted for the 256th, 512th, 768th, and 1024th samples, starting from the beginning of the frame.

[0049] For example, if one frame contains 2048 samples, metadata for each sample, spaced at 256 sample intervals within the frame, is transmitted. In this example, a total of eight pieces of metadata are transmitted.

[0050] In this way, by sending two or more pieces of metadata per frame using the number specification method, sample specification method, and automatic switching method, it is possible to send more metadata when the number of samples that make up a frame is large.

[0051] This shortens the length of the section in which samples for which the VBAP gain is calculated by linear interpolation are consecutively arranged, making it possible to obtain audio with higher sound quality.

[0052] For example, if the length of the section in which consecutive samples for which the VBAP gain is calculated by linear interpolation is shortened, the difference between the sum of the squares of the VBAP gains of each speaker and 1 will also become smaller, thereby improving the sense of positioning of the sound image of the object.

[0053] In addition, since the distance between samples with metadata is shortened, the difference in VBAP gain between those samples is also reduced, allowing for more accurate rendering of object movement. Furthermore, shortening the distance between samples with metadata can shorten the period in which an object appears to be moving continuously in terms of audio, even during periods in which the object actually moves discontinuously, such as during scene changes. In particular, the sample specification method can express discontinuous object movement by transmitting metadata for appropriate sample positions.

[0054] Note that metadata may be transmitted using only one of the three methods described above, the number specification method, the sample specification method, and the automatic switching method, or two or more of these three methods may be switched for each frame or object.

[0055] For example, if you want to switch between the three methods of number specification, sample specification, and automatic switching on a frame-by-frame or object-by-object basis, you can store a switching index in the bitstream that indicates which method was used to transmit the metadata.

[0056] In this case, for example, a switching index value of 0 indicates that the number specification method was selected, i.e., that the metadata was transmitted using the number specification method, a switching index value of 1 indicates that the sample specification method was selected, and a switching index value of 2 indicates that the automatic switching method was selected. In the following, the explanation will be continued assuming that the number specification method, sample specification method, and automatic switching method can be switched for each frame or each object.

[0057] Furthermore, in the method of transmitting audio signals and metadata defined in the MPEG-H 3D Audio standard, only the metadata of the last sample in a frame is transmitted, so when calculating the VBAP gain of each sample by interpolation, the VBAP gain of the last sample in the frame before the current frame is required.

[0058] Therefore, for example, even if the playback side (decoding side) attempts to perform random access to start playback from the audio signal of an arbitrary frame, the VBAP gain of frames prior to the randomly accessed frame has not been calculated, and therefore VBAP gain interpolation processing cannot be performed. For this reason, random access is not possible in the MPEG-H 3D Audio standard.

[0059] Therefore, in this technology, for each frame or frames at any interval, metadata required for interpolation processing is transmitted along with the metadata of those frames, thereby enabling calculation of the VBAP gain of a sample of a frame prior to the current frame or the first sample of the current frame. This enables random access. Note that hereinafter, the metadata for interpolation processing transmitted together with the normal metadata will also be referred to as additional metadata.

[0060] Here, the additional metadata transmitted together with the metadata of the current frame may be, for example, the metadata of the last sample of the frame immediately preceding the current frame, or the metadata of the first sample of the current frame.

[0061] In addition, to easily identify whether or not additional metadata is present for each frame, an additional metadata flag indicating the presence or absence of additional metadata for each object is stored in the bitstream for each frame. For example, if the additional metadata flag for a given frame has a value of 1, additional metadata is present in that frame, and if the additional metadata flag has a value of 0, no additional metadata is present in that frame.

[0062] Basically, the additional metadata flags of all objects in the same frame have the same value.

[0063] In this way, by transmitting the additional metadata flag for each frame and transmitting additional metadata as needed, random access can be performed for frames with additional metadata.

[0064] If a frame designated as the destination of random access does not contain additional metadata, the frame that is closest in time to the designated frame and that contains additional metadata can be used as the destination of random access. Therefore, by transmitting additional metadata at appropriate frame intervals, random access can be achieved without causing the user to feel unnatural.

[0065] Although the additional metadata has been explained above, it is also possible to perform VBAP gain interpolation processing for a frame designated as the destination of random access without using the additional metadata. In this case, random access becomes possible while suppressing an increase in the data amount (bit rate) of the bitstream due to the storage of the additional metadata.

[0066] Specifically, in a frame designated as the destination of random access, the VBAP gain value of the frame prior to the current frame is set to 0, and interpolation processing is performed with the VBAP gain value calculated for the current frame. Note that this method is not limited to this, and interpolation processing may be performed so that the VBAP gain values ​​of all samples of the current frame are the same as the VBAP gain calculated for the current frame. On the other hand, in frames not designated as the destination of random access, interpolation processing is performed as usual using the VBAP gain of the frame prior to the current frame.

[0067] In this way, by switching the VBAP gain interpolation process based on whether or not a file has been designated as a random access destination, random access becomes possible without using additional metadata.

[0068] In the above-mentioned MPEG-H 3D Audio standard, an independent flag (also referred to as indepFlag) is stored in the bitstream for each frame, indicating whether the current frame is a frame (referred to as an independent frame) that can be decoded and rendered using only the data of the current frame in the bitstream. If the value of the independent flag is 1, the decoding side can perform decoding and rendering without using data of frames in the bitstream prior to the current frame or any information obtained by decoding that data.

[0069] Therefore, when the value of the independent flag is 1, it is necessary to perform decoding and rendering without using the VBAP gain of the frame prior to the current frame.

[0070] Therefore, in a frame where the value of the independent flag is 1, the above-mentioned additional metadata may be stored in the bitstream, or the above-mentioned interpolation process may be switched.

[0071] In this way, by switching whether or not to store additional metadata in the bitstream and switching the VBAP gain interpolation process depending on the value of the independent flag, when the value of the independent flag is 1, it is possible to perform decoding and rendering without using the VBAP gain of a frame prior to the current frame.

[0072] Furthermore, as explained above, in the MPEG-H 3D Audio standard, the metadata obtained by decoding is limited to the representative sample in a frame, i.e., the metadata of the last sample. However, when encoding audio signals and metadata, the metadata before compression (encoding) input to the encoding device is rarely defined for all samples in a frame. In other words, many samples in an audio signal frame have no metadata even before encoding.

[0073] Currently, in most cases, only samples that are evenly spaced, such as the 0th sample, the 1024th sample, and the 2048th sample, have metadata, or only samples that are unevenly spaced, such as the 0th sample, the 138th sample, and the 2044th sample, have metadata.

[0074] In such a case, some frames may not have any samples with metadata, and no metadata will be transmitted for those frames. As a result, to calculate the VBAP gain for each sample in a frame without any samples with metadata, the decoder must calculate the VBAP gain for subsequent frames with metadata. This results in delays in decoding and rendering the metadata, making it impossible to perform decoding and rendering in real time.

[0075] Therefore, in this technology, the encoding side uses interpolation processing (sample interpolation) to obtain metadata for each sample between samples that have metadata as needed, and the decoding side can perform decoding and rendering in real time. In particular, in video games and the like, there is a demand for minimizing delays in audio playback. Therefore, it is highly significant that this technology can reduce delays in decoding and rendering, thereby improving interactivity in game operations and the like.

[0076] The metadata interpolation process may be any process, such as linear interpolation or non-linear interpolation using a higher-order function.

[0077] About Bitstream Next, a more specific embodiment to which the present technology described above is applied will be described.

[0078] An encoding device that encodes the audio signal and metadata of each object outputs a bitstream, for example, as shown in FIG.

[0079] The bitstream shown in Figure 1 has a header at the beginning, which stores information indicating the number of samples that make up one frame of the audio signal of each object, i.e., the number of samples in one frame (hereinafter also referred to as sample number information).

[0080] In the bitstream, the header is followed by data for each frame. Specifically, an independent flag indicating whether the current frame is an independent frame is placed in region R10. Region R11 contains encoded audio data obtained by encoding the audio signals of each object in the same frame.

[0081] In addition, in the portion of region R12 following region R11, encoded metadata obtained by encoding metadata of each object of the same frame is placed.

[0082] For example, in region R21 within region R12, encoded metadata for one frame of one object is arranged.

[0083] In this example, an additional metadata flag is placed at the beginning of the encoded metadata, and a switching index is placed after the additional metadata flag.

[0084] Furthermore, metadata number information and a sample index are arranged next to the switching index. Note that although only one sample index is illustrated here, more specifically, the same number of sample indexes as the number of metadata stored in the encoded metadata are stored in that encoded metadata.

[0085] In the encoded metadata, if the method indicated by the switching index is the number-specifying method, the metadata number information is arranged following the switching index, but no sample index is arranged.

[0086] If the method indicated by the switching index is the sample designation method, the switching index is followed by metadata number information and a sample index. If the method indicated by the switching index is the automatic switching method, neither metadata number information nor a sample index is placed after the switching index.

[0087] Additional metadata is placed following the metadata number information and sample index, which are placed as needed, and then following that additional metadata, the defined number of metadata for each sample is placed.

[0088] Here, the additional metadata is placed only when the value of the additional metadata flag is 1, and is not placed when the value of the additional metadata flag is 0.

[0089] In the region R12, encoded metadata similar to the encoded metadata arranged in the region R21 is arranged for each object.

[0090] In the bitstream, one frame of data is composed of an independent flag placed in region R10, encoded audio data of each object placed in region R11, and encoded metadata of each object placed in region R12.

[0091] <Configuration example of encoding device> Next, a description will be given of the configuration of an encoding device that outputs the bitstream shown in Fig. 1. Fig. 2 is a diagram showing an example of the configuration of an encoding device to which the present technology is applied.

[0092] The encoding device 11 includes an audio signal acquisition unit 21, an audio signal encoding unit 22, a metadata acquisition unit 23, an interpolation processing unit 24, a related information acquisition unit 25, a metadata encoding unit 26, a multiplexing unit 27, and an output unit 28.

[0093] The audio signal acquisition unit 21 acquires the audio signal of each object and supplies it to the audio signal encoding unit 22. The audio signal encoding unit 22 encodes the audio signal supplied from the audio signal acquisition unit 21 on a frame-by-frame basis, and supplies the resulting encoded audio data for each frame of each object to the multiplexing unit 27.

[0094] The metadata acquisition unit 23 acquires metadata for each frame of each object, more specifically, metadata for each sample within a frame, and supplies the metadata to the interpolation processing unit 24. Here, the metadata includes, for example, position information indicating the position of the object in space, importance information indicating the importance of the object, information indicating the degree of spread of the sound image of the object, etc. The metadata acquisition unit 23 acquires metadata for predetermined samples (PCM samples) of the audio signal of each object.

[0095] The interpolation processing unit 24 performs interpolation processing on the metadata supplied from the metadata acquisition unit 23 to generate metadata for all samples or some specific samples among the samples of the audio signal that do not have metadata. The interpolation processing unit 24 generates metadata for samples in a frame by interpolation processing so that the audio signal of one frame of one object has multiple pieces of metadata, that is, so that multiple samples in one frame have metadata.

[0096] The interpolation processing unit 24 supplies the metadata for each frame of each object obtained by the interpolation processing to the metadata encoding unit 26.

[0097] The related information acquisition unit 25 acquires, as related information, information related to metadata, such as information indicating whether the current frame is an independent frame for each frame (referred to as independent frame information), and, for each object, for each frame of the audio signal, sample number information, information indicating which method the metadata will be transmitted in, information indicating whether additional metadata will be transmitted, and information indicating which sample's metadata will be transmitted. Furthermore, based on the acquired related information, the related information acquisition unit 25 generates, for each object, necessary information from among an additional metadata flag, a switching index, metadata number information, and a sample index for each frame, and supplies the generated information to the metadata encoding unit 26.

[0098] The metadata encoding unit 26 encodes the metadata supplied from the interpolation processing unit 24 based on the information supplied from the related information acquisition unit 25, and supplies the resulting encoded metadata for each frame of each object and the independent frame information included in the information supplied from the related information acquisition unit 25 to the multiplexing unit 27.

[0099] The multiplexing unit 27 multiplexes the encoded audio data supplied from the audio signal encoding unit 22, the encoded metadata supplied from the metadata encoding unit 26, and the independent flag obtained based on the independent frame information supplied from the metadata encoding unit 26 to generate a bitstream, and supplies the bitstream to the output unit 28. The output unit 28 outputs the bitstream supplied from the multiplexing unit 27. In other words, the bitstream is transmitted.

[0100] <Description of the encoding process> When an audio signal of an object is supplied from an external source, the encoding device 11 performs encoding processing and outputs a bitstream. The encoding processing by the encoding device 11 will be described below with reference to the flowchart in Fig. 3. Note that this encoding processing is performed for each frame of the audio signal.

[0101] In step S11 , the audio signal acquisition unit 21 acquires one frame of audio signals for each object and supplies the acquired signals to the audio signal encoding unit 22 .

[0102] In step S12, the audio signal encoding unit 22 encodes the audio signal supplied from the audio signal acquisition unit 21, and supplies the resulting encoded audio data of one frame of each object to the multiplexing unit 27.

[0103] For example, the audio signal encoding unit 22 converts the audio signal from a time signal to a frequency signal by performing a Modified Discrete Cosine Transform (MDCT) on the audio signal, etc. Then, the audio signal encoding unit 22 encodes the MDCT coefficients obtained by the MDCT, and defines the resulting scale factors, side information, and quantized spectrum as encoded audio data obtained by encoding the audio signal.

[0104] As a result, for example, encoded audio data of each object stored in the region R11 of the bitstream shown in FIG. 1 is obtained.

[0105] In step S13, the metadata acquisition unit 23 acquires metadata for each frame of the audio signal for each object and supplies the metadata to the interpolation processing unit 24.

[0106] In step S14, the interpolation processing unit 24 performs interpolation processing on the metadata supplied from the metadata acquisition unit 23, and supplies the resulting metadata to the metadata encoding unit .

[0107] For example, for one audio signal, the interpolation processing unit 24 calculates the position information of each sample located between a predetermined sample by linear interpolation based on the position information of the predetermined sample as metadata and the position information of another sample located temporally before the predetermined sample. Similarly, interpolation processing such as linear interpolation is performed on metadata such as importance information and information indicating the extent of sound image spread, to generate metadata for each sample.

[0108] In the metadata interpolation process, the metadata may be calculated so that all samples of the audio signal of one frame of the object have the metadata, or so that only necessary samples out of all the samples have the metadata. Furthermore, the interpolation process is not limited to linear interpolation, and may be nonlinear interpolation.

[0109] In step S15, the related information acquisition unit 25 acquires related information related to the metadata for the frames of the audio signal of each object.

[0110] Then, based on the acquired related information, the related information acquisition unit 25 generates necessary information from among an additional metadata flag, a switching index, metadata number information, and a sample index for each object, and supplies the information to the metadata encoding unit 26.

[0111] It should be noted that the related information acquisition unit 25 may not generate the additional metadata flag, the switching index, and the like, but may instead acquire the additional metadata flag, the switching index, and the like from outside.

[0112] In step S16, the metadata encoding unit 26 encodes the metadata supplied from the interpolation processing unit 24 based on the additional metadata flag, switching index, metadata number information, sample index, etc. supplied from the related information acquisition unit 25.

[0113] When encoding the metadata, for each object, encoded metadata is generated so that, of the metadata for each sample in a frame of an audio signal, only the metadata for the sample position determined by the sample number information, the method indicated by the switching index, the metadata number information, the sample index, etc. Furthermore, the metadata for the first sample of a frame or the metadata for the last sample of the held immediately preceding frame is used as additional metadata as necessary.

[0114] The encoded metadata includes, in addition to the metadata, an additional metadata flag and a switching index, and also includes information on the number of metadata pieces, a sample index, additional metadata, etc. as necessary.

[0115] This provides the coded metadata for each object stored in region R12 of the bitstream shown in Fig. 1. For example, the coded metadata stored in region R21 is the coded metadata for one frame of one object.

[0116] In this case, for example, when the number specification method is selected for a frame to be processed for an object and additional metadata is transmitted, encoded metadata consisting of an additional metadata flag, a switching index, metadata number information, additional metadata, and metadata is generated.

[0117] Also, for example, when the sample designation method is selected for the frame to be processed for the object and additional metadata is not transmitted, encoded metadata consisting of an additional metadata flag, a switching index, metadata number information, a sample index, and metadata is generated.

[0118] Furthermore, for example, when an automatic switching method is selected in a frame in which an object is being processed and additional metadata is transmitted, coded metadata consisting of an additional metadata flag, a switching index, additional metadata, and metadata is generated.

[0119] The metadata encoding unit 26 supplies the encoded metadata of each object obtained by encoding the metadata and the independent frame information included in the information supplied from the related information acquisition unit 25 to the multiplexing unit 27.

[0120] In step S17, the multiplexing unit 27 multiplexes the encoded audio data supplied from the audio signal encoding unit 22, the encoded metadata supplied from the metadata encoding unit 26, and the independent flag obtained based on the independent frame information supplied from the metadata encoding unit 26, to generate a bitstream, and supplies the bitstream to the output unit 28.

[0121] As a result, a bitstream for one frame, for example, a bitstream consisting of the area R10 to the area R12 of the bitstream shown in FIG. 1, is generated.

[0122] In step S18, the output unit 28 outputs the bit stream supplied from the multiplexing unit 27, and the encoding process ends. When the leading portion of the bit stream is output, a header including sample number information and the like is also output, as shown in FIG.

[0123] In this manner, the encoding device 11 encodes the audio signal and also encodes the metadata, and outputs a bitstream consisting of the resulting encoded audio data and encoded metadata.

[0124] In this case, by transmitting multiple metadata for one frame, the length of the section of samples for which the VBAP gain is calculated by interpolation processing can be shortened on the decoding side, making it possible to obtain audio with higher quality.

[0125] In addition, by performing interpolation processing on the metadata, it is possible to always send one or more pieces of metadata per frame, enabling real-time decoding and rendering on the decoding side. Furthermore, by sending additional metadata as needed, random access can be achieved.

[0126] <Configuration example of a decoding device> Next, a description will be given of a decoding device that receives (obtains) and decodes the bitstream output from the encoding device 11. For example, a decoding device to which the present technology is applied is configured as shown in FIG.

[0127] A speaker system 52 consisting of multiple speakers arranged in a playback space is connected to this decoding device 51. The decoding device 51 supplies the audio signals of each channel obtained by decoding and rendering to the speakers of each channel that make up the speaker system 52, thereby reproducing the audio.

[0128] The decoding device 51 includes an acquisition unit 61 , a separation unit 62 , an audio signal decoding unit 63 , a metadata decoding unit 64 , a gain calculation unit 65 , and an audio signal generation unit 66 .

[0129] The acquisition unit 61 acquires the bitstream output from the encoding device 11 and supplies it to the separation unit 62. The separation unit 62 separates the bitstream supplied from the acquisition unit 61 into an independent flag, encoded audio data, and encoded metadata, and supplies the encoded audio data to an audio signal decoding unit 63 and the independent flag and encoded metadata to a metadata decoding unit 64.

[0130] The separator 62 reads various information such as sample number information from the header of the bit stream as needed, and supplies the information to the audio signal decoder 63 and the metadata decoder 64 .

[0131] The audio signal decoding unit 63 decodes the encoded audio data supplied from the separation unit 62 and supplies the resulting audio signals of each object to the audio signal generation unit 66 .

[0132] The metadata decoding unit 64 decodes the encoded metadata supplied from the separation unit 62 and supplies the resulting metadata for each frame of the audio signal for each object and the independent flag supplied from the separation unit 62 to the gain calculation unit 65.

[0133] The metadata decoding unit 64 has an additional metadata flag reading unit 71 that reads an additional metadata flag from the encoded metadata, and a switching index reading unit 72 that reads a switching index from the encoded metadata.

[0134] The gain calculation unit 65 calculates the VBAP gain of the samples in the frame of the audio signal for each object based on pre-stored placement position information indicating the spatial placement position of each speaker that constitutes the speaker system 52, and the metadata and independent flag for each object frame supplied from the metadata decoding unit 64.

[0135] The gain calculation unit 65 also has an interpolation processing unit 73 that calculates the VBAP gain of another sample by interpolation based on the VBAP gain of a predetermined sample.

[0136] The gain calculation unit 65 supplies the VBAP gain calculated for each object for each sample in a frame of the audio signal to the audio signal generation unit 66 .

[0137] The audio signal generation unit 66 generates an audio signal for each channel, i.e., an audio signal to be supplied to the speaker of each channel, based on the audio signal of each object supplied from the audio signal decoding unit 63 and the VBAP gain for each sample of each object supplied from the gain calculation unit 65.

[0138] The audio signal generating unit 66 supplies the generated audio signal to each speaker that constitutes the speaker system 52, and causes the speakers to output sound based on the audio signal.

[0139] In the decoding device 51, a block consisting of a gain calculation unit 65 and an audio signal generation unit 66 functions as a renderer (rendering unit) that performs rendering based on the audio signal and metadata obtained by decoding.

[0140] <Description of Decryption Process> When a bitstream is transmitted from the encoding device 11, the decoding device 51 receives (acquires) and decodes the bitstream. The decoding process by the decoding device 51 will be described below with reference to the flowchart in Fig. 5. Note that this decoding process is performed for each frame of the audio signal.

[0141] In step S 41 , the acquisition unit 61 acquires one frame of the bit stream output from the encoding device 11 and supplies it to the separation unit 62 .

[0142] In step S42, the separation unit 62 separates the bitstream supplied from the acquisition unit 61 into an independent flag, encoded audio data, and encoded metadata, and supplies the encoded audio data to the audio signal decoding unit 63, while supplying the independent flag and encoded metadata to the metadata decoding unit 64.

[0143] At this time, the demultiplexer 62 supplies the sample number information read from the header of the bit stream to the metadata decoder 64. The timing for supplying the sample number information may be the timing when the header of the bit stream is acquired.

[0144] In step S 43 , the audio signal decoding unit 63 decodes the encoded audio data supplied from the separation unit 62 , and supplies the audio signal generation unit 66 with the audio signal for one frame of each object obtained as a result.

[0145] For example, the audio signal decoding unit 63 decodes the encoded audio data to obtain MDCT coefficients. Specifically, the audio signal decoding unit 63 calculates the MDCT coefficients based on the scale factors, side information, and quantized spectrum supplied as the encoded audio data.

[0146] Furthermore, the audio signal decoding unit 63 performs an IMDCT (Inverse Modified Discrete Cosine Transform) based on the MDCT coefficients, and supplies the resulting PCM data to an audio signal generating unit 66 as an audio signal.

[0147] After the encoded audio data is decoded, the encoded metadata is then decoded. That is, in step S44, the additional metadata flag reading unit 71 of the metadata decoding unit 64 reads the additional metadata flag from the encoded metadata supplied from the separation unit 62.

[0148] For example, the metadata decoding unit 64 sequentially selects as processing target objects objects corresponding to the encoded metadata sequentially supplied from the separation unit 62. The additional metadata flag reading unit 71 reads the additional metadata flag from the encoded metadata of the object selected as the processing target.

[0149] In step S45, the switching index reading unit 72 of the metadata decoding unit 64 reads a switching index from the encoded metadata of the object to be processed, which is supplied from the separating unit 62.

[0150] In step S46, the switching index reading unit 72 determines whether the method indicated by the switching index read in step S45 is the number designation method.

[0151] If it is determined in step S46 that the number specification method is used, then in step S47 the metadata decoding unit 64 reads out metadata number information from the encoded metadata of the object to be processed, which is supplied from the separation unit 62.

[0152] The encoded metadata of the object to be processed contains metadata in the number indicated by the metadata number information read out in this way.

[0153] In step S48, the metadata decoding unit 64 determines the sample position of the transmitted metadata in the frame of the audio signal of the object to be processed based on the metadata number information read in step S47 and the sample number information supplied from the separation unit 62.

[0154] For example, a frame section consisting of the number of samples indicated by the sample number information is divided into sections with the number of metadata indicated by the metadata number information, and the position of the last sample in each divided section is taken as the sample position of the metadata, i.e., the position of the sample having the metadata. The sample positions thus obtained are taken as the sample positions of each piece of metadata included in the encoded metadata, i.e., the samples having that metadata.

[0155] Note that, although we have explained the case where a frame is divided into equal sections and the metadata of the last sample in each divided section is transmitted, the sample position of each piece of metadata is calculated from the sample number information and metadata number information depending on which sample's metadata is transmitted.

[0156] Once the number of metadata items included in the encoded metadata of the object to be processed and the sample positions of each metadata item have been identified in this way, the process proceeds to step S53.

[0157] On the other hand, if it is determined in step S46 that the method is not the number specification method, then in step S49 the switching index readout unit 72 determines whether the method indicated by the switching index read out in step S45 is the sample specification method.

[0158] If it is determined in step S49 that the method is the sample designation method, then in step S50 the metadata decoding unit 64 reads out metadata number information from the encoded metadata of the object to be processed, which is supplied from the separation unit 62.

[0159] In step S51, the metadata decoding unit 64 reads sample indexes from the encoded metadata of the object to be processed, supplied from the separation unit 62. At this time, the number of sample indexes read out is equal to the number indicated by the metadata number information.

[0160] From the metadata number information and sample indexes thus read out, it is possible to identify the number of metadata stored in the encoded metadata of the object to be processed and the sample positions of the metadata.

[0161] Once the number of metadata items included in the encoded metadata of the object to be processed and the sample positions of each metadata item have been identified, the process proceeds to step S53.

[0162] If it is determined in step S49 that the method is not the sample designation method, that is, if the method indicated by the switching index is the automatic switching method, the process proceeds to step S52.

[0163] In step S52, the metadata decoding unit 64 determines the number of metadata contained in the encoded metadata of the object to be processed and the sample position of each metadata based on the sample number information supplied from the separation unit 62, and the processing proceeds to step S53.

[0164] For example, in the automatic switching method, the number of metadata to be transmitted and the sample position of each metadata, that is, which sample's metadata is to be transmitted, are determined in advance for the number of samples that make up one frame.

[0165] Therefore, the metadata decoding section 64 can identify the number of metadata items stored in the encoded metadata of the object to be processed and the sample positions of the metadata items from the sample number information.

[0166] After the processing of step S48, step S51, or step S52 has been performed, in step S53 the metadata decoding section 64 determines whether or not there is additional metadata based on the value of the additional metadata flag read in step S44.

[0167] If it is determined in step S53 that there is additional metadata, in step S54 the metadata decoding unit 64 reads the additional metadata from the encoded metadata of the object being processed. After the additional metadata has been read, the process then proceeds to step S55.

[0168] On the other hand, if it is determined in step S53 that there is no additional metadata, the process of step S54 is skipped and the process proceeds to step S55.

[0169] If additional metadata is read in step S54 or if it is determined in step S53 that there is no additional metadata, then in step S55 the metadata decoding unit 64 reads metadata from the encoded metadata of the object to be processed.

[0170] At this time, the number of pieces of metadata specified by the above-described process is read from the encoded metadata.

[0171] Through the above processing, the metadata and additional metadata are read out for one frame of the audio signal of the object to be processed.

[0172] The metadata decoding unit 64 supplies each piece of read metadata to the gain calculation unit 65. At this time, the gain calculation unit 65 supplies the metadata so that it can be identified which metadata belongs to which sample of which object. Furthermore, when additional metadata is read, the metadata decoding unit 64 also supplies the read additional metadata to the gain calculation unit 65.

[0173] In step S56, the metadata decoding section 64 determines whether or not the metadata has been read for all objects.

[0174] If it is determined in step S56 that the metadata has not yet been read for all objects, the process returns to step S44, and the above-described process is repeated. In this case, an object that has not yet been processed is set as the new object to be processed, and metadata, etc. are read from the encoded metadata of that object.

[0175] On the other hand, if it is determined in step S56 that metadata has been read for all objects, the metadata decoding unit 64 supplies the independent flag supplied from the separation unit 62 to the gain calculation unit 65, and then the process proceeds to step S57, where rendering begins.

[0176] That is, in step S57, the gain calculation unit 65 calculates a VBAP gain based on the metadata, additional metadata, and independent flag supplied from the metadata decoding unit 64.

[0177] For example, the gain calculation unit 65 selects each object in turn as the object to be processed, and then selects samples with metadata within the frame of the audio signal of the object to be processed in turn as the samples to be processed.

[0178] The gain calculation unit 65 calculates the VBAP gain of each channel of the sample to be processed, i.e., the speaker of each channel, using VBAP, based on the spatial position of the object indicated by the position information as metadata of the sample and the spatial position of each speaker of the speaker system 52 indicated by the placement position information.

[0179] VBAP allows sound to be output at a specified gain from two or three speakers located around an object, thereby localizing a sound image at the object's position. VBAP is described in detail in, for example, "Ville Pulkki, 'Virtual Sound Source Positioning Using Vector Base Amplitude Panning,' Journal of AES, vol. 45, no. 6, pp. 456-466, 1997."

[0180] In step S58, the interpolation processing unit 73 performs interpolation processing to calculate the VBAP gain of each speaker for samples without metadata.

[0181] For example, the interpolation process uses the VBAP gain of the sample to be processed calculated in the immediately preceding step S57 and the VBAP gain of a sample (hereinafter also referred to as a reference sample) that precedes the sample to be processed in time and has metadata of the same frame of the object to be processed or the frame immediately preceding the sample to be processed. That is, for each speaker (channel) that makes up speaker system 52, the VBAP gain of the sample to be processed and the VBAP gain of the reference sample are used, and the VBAP gain of each sample between the sample to be processed and the reference sample is calculated by linear interpolation or the like.

[0182] For example, when random access is instructed, or when the value of the independent flag supplied from the metadata decoding unit 64 is 1 and there is additional metadata, the gain calculation unit 65 calculates the VBAP gain using the additional metadata.

[0183] Specifically, for example, within a frame of the audio signal of the object to be processed, the sample closest to the beginning of the frame and having metadata is set as the sample to be processed, and the VBAP gain of that sample is calculated. In this case, since the VBAP gain has not been calculated for frames prior to this frame, the gain calculation unit 65 uses the additional metadata to calculate the VBAP gain of the reference sample, using the first sample of that frame or the last sample of the frame immediately preceding that frame as a reference sample.

[0184] Then, the interpolation processing unit 73 calculates the VBAP gain of each sample between the sample to be processed and the reference sample by interpolation, using the VBAP gain of the sample to be processed and the VBAP gain of the reference sample.

[0185] On the other hand, for example, when random access is instructed, or when the value of the independent flag supplied from the metadata decoding unit 64 is 1 and there is no additional metadata, the VBAP gain is not calculated using the additional metadata, and the interpolation process is switched.

[0186] Specifically, for example, assume that within a frame of the audio signal of the object to be processed, the sample closest to the beginning of the frame and having metadata is set as the sample to be processed, and the VBAP gain of that sample is calculated. In this case, since the VBAP gain has not been calculated for frames prior to this frame, the gain calculation unit 65 uses the first sample of that frame or the last sample of the frame immediately preceding that frame as a reference sample and calculates the VBAP gain of the reference sample as 0.

[0187] Then, the interpolation processing unit 73 calculates the VBAP gain of each sample between the sample to be processed and the reference sample by interpolation, using the VBAP gain of the sample to be processed and the VBAP gain of the reference sample.

[0188] However, the present invention is not limited to this method. For example, the interpolation process may be performed so that the VBAP gains of all the samples to be interpolated are the same as the VBAP gain of the sample to be processed.

[0189] In this way, by switching the VBAP gain interpolation process, random access and decoding and rendering of independent frames are possible even for frames without additional metadata.

[0190] Although an example has been described in which the VBAP gain of a sample without metadata is obtained by interpolation, the metadata decoding unit 64 may be configured to obtain the metadata of the sample without metadata by interpolation. In this case, since the metadata of all samples of the audio signal is obtained, the interpolation processing unit 73 does not perform interpolation processing of the VBAP gain.

[0191] In step S59, the gain calculation unit 65 determines whether or not the VBAP gains of all samples within the frame of the audio signal of the object to be processed have been calculated.

[0192] If it is determined in step S59 that the VBAP gains have not yet been calculated for all samples, the process returns to step S57, and the above-described process is repeated: the next sample having metadata is selected as the sample to be processed, and the VBAP gain is calculated.

[0193] On the other hand, if it is determined in step S59 that the VBAP gains of all samples have been calculated, then in step S60 the gain calculation section 65 determines whether or not the VBAP gains of all objects have been calculated.

[0194] For example, when all objects are set as objects to be processed and the VBAP gain of each sample for each speaker is calculated for these objects, it is determined that the VBAP gains of all objects have been calculated.

[0195] If it is determined in step S60 that the VBAP gains of all objects have not yet been calculated, the process returns to step S57, and the above-described processes are repeated.

[0196] On the other hand, if it is determined in step S60 that the VBAP gains for all objects have been calculated, the gain calculation unit 65 supplies the calculated VBAP gains to the audio signal generation unit 66, and the process proceeds to step S61. In this case, the VBAP gains of each sample in the frame of the audio signal of each object, calculated for each speaker, are supplied to the audio signal generation unit 66.

[0197] In step S61, the audio signal generation unit 66 generates an audio signal for each speaker based on the audio signal of each object supplied from the audio signal decoding unit 63 and the VBAP gain for each sample of each object supplied from the gain calculation unit 65.

[0198] For example, the audio signal generation unit 66 generates an audio signal for each speaker by adding the signals obtained by multiplying each audio signal of each object by the VBAP gain of the same speaker obtained for each object for each sample.

[0199] Specifically, for example, suppose there are three objects, object OB1 to object OB3, and VBAP gains G1 to G3 are obtained as the VBAP gains of a predetermined speaker SP1 that constitutes the speaker system 52 for these objects. In this case, the audio signal of object OB1 multiplied by VBAP gain G1, the audio signal of object OB2 multiplied by VBAP gain G2, and the audio signal of object OB3 multiplied by VBAP gain G3 are added together, and the resulting audio signal is used as the audio signal to be supplied to speaker SP1.

[0200] In step S62, the audio signal generation unit 66 supplies the audio signals of each speaker obtained in the process of step S61 to each speaker of the speaker system 52, and reproduces sound based on those audio signals, after which the decoding process ends. As a result, the sound of each object is reproduced by the speaker system 52.

[0201] In this manner, the decoding device 51 decodes the encoded audio data and encoded metadata, performs rendering based on the audio signals and metadata obtained by decoding, and generates audio signals for each speaker.

[0202] In the decoding device 51, multiple pieces of metadata are obtained for each frame of the audio signal of an object when rendering, so the length of the section of samples for which the VBAP gain is calculated by interpolation can be shortened. This not only enables higher quality audio to be obtained, but also enables decoding and rendering to be performed in real time. Furthermore, because additional metadata is included in the encoded metadata for some frames, random access or decoding and rendering can be achieved for independent frames. Furthermore, even for frames that do not include additional metadata, random access or decoding and rendering can be achieved for independent frames by switching the VBAP gain interpolation process.

[0203] The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0204] FIG. 6 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes using a program.

[0205] In the computer, a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, and a RAM (Random Access Memory) 503 are interconnected by a bus 504.

[0206] An input / output interface 505 is further connected to the bus 504. An input unit 506, an output unit 507, a recording unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.

[0207] The input unit 506 includes a keyboard, a mouse, a microphone, an image sensor, etc. The output unit 507 includes a display, a speaker, etc. The recording unit 508 includes a hard disk, a non-volatile memory, etc. The communication unit 509 includes a network interface, etc. The drive 510 drives a removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0208] In a computer configured as described above, the CPU 501 performs the above-described series of processes by, for example, loading a program recorded in the recording unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executing it.

[0209] The program executed by the computer (CPU 501) can be provided by being recorded on a removable recording medium 511 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0210] In a computer, a program can be installed in the recording unit 508 via the input / output interface 505 by inserting a removable recording medium 511 into the drive 510. The program can also be received by the communication unit 509 via a wired or wireless transmission medium and installed in the recording unit 508. Alternatively, the program can be installed in the ROM 502 or the recording unit 508 in advance.

[0211] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0212] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0213] For example, this technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0214] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by multiple devices.

[0215] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0216] Furthermore, the present technology can also be configured as follows.

[0217] (1) an acquisition unit that acquires encoded audio data obtained by encoding an audio signal of frames at a predetermined time interval of an audio object and a plurality of metadata of the frames; a decoding unit that decodes the encoded audio data; a rendering unit that performs rendering based on the audio signal obtained by the decoding and the plurality of metadata; A decoding device comprising: (2) The metadata includes location information indicating the location of the audio object. The decoding device according to (1). (3) Each of the plurality of metadata is metadata for each of a plurality of samples within the frame of the audio signal. A decoding device according to (1) or (2). (4) Each of the plurality of metadata is metadata of a plurality of samples arranged at intervals of the number of samples obtained by dividing the number of samples constituting the frame by the number of the plurality of metadata. (3) A decoding device according to the present invention. (5) Each of the plurality of metadata is metadata of each of a plurality of samples indicated by each of a plurality of sample indexes. (3) A decoding device according to the present invention. (6) Each of the plurality of metadata is metadata of a plurality of samples arranged at intervals of a predetermined number of samples in the frame. (3) A decoding device according to the present invention. (7) The plurality of metadata include metadata for performing an interpolation process on gains of the samples of the audio signal calculated based on the metadata. A decoding device according to any one of (1) to (6). (8) Obtaining encoded audio data obtained by encoding an audio signal of frames at a predetermined time interval of an audio object and a plurality of metadata of the frames; decoding the encoded audio data; Rendering is performed based on the audio signal obtained by the decoding and the plurality of metadata. A decoding method including steps. (9) Obtaining encoded audio data obtained by encoding an audio signal of frames at a predetermined time interval of an audio object and a plurality of metadata of the frames; decoding the encoded audio data; Rendering is performed based on the audio signal obtained by the decoding and the plurality of metadata. A program that causes a computer to execute a process that includes steps. (10) an encoding unit that encodes an audio signal of frames of an audio object at a predetermined time interval; a generating unit that generates a bitstream including the encoded audio data obtained by the encoding and a plurality of pieces of metadata for the frames; An encoding device comprising: (11) The metadata includes location information indicating the location of the audio object. The encoding device according to (10). (12) Each of the plurality of metadata is metadata for each of a plurality of samples within the frame of the audio signal. The encoding device according to (10) or (11). (13) Each of the plurality of metadata is metadata of a plurality of samples arranged at intervals of the number of samples obtained by dividing the number of samples constituting the frame by the number of the plurality of metadata. The encoding device according to (12). (14) Each of the plurality of metadata is metadata of each of a plurality of samples indicated by each of a plurality of sample indexes. The encoding device according to (12). (15) Each of the plurality of metadata is metadata of a plurality of samples arranged at intervals of a predetermined number of samples in the frame. The encoding device according to (12). (16) The plurality of metadata include metadata for performing an interpolation process on gains of the samples of the audio signal calculated based on the metadata. An encoding device according to any one of (10) to (15). (17) The interpolation processing unit performs an interpolation process on the metadata. An encoding device according to any one of (10) to (16). (18) encoding an audio signal of frames of a predetermined time interval of an audio object; generating a bitstream including the encoded audio data obtained by the encoding and a plurality of metadata of the frames; An encoding method comprising the steps: (19) encoding an audio signal of frames of a predetermined time interval of an audio object; generating a bitstream including the encoded audio data obtained by the encoding and a plurality of metadata of the frames; A program that causes a computer to execute a process that includes steps. [Explanation of symbols]

[0218] 11 encoding device, 22 audio signal encoding unit, 24 interpolation processing unit, 25 related information acquisition unit, 26 metadata encoding unit, 27 multiplexing unit, 28 output unit, 51 decoding device, 62 separation unit, 63 audio signal decoding unit, 64 metadata decoding unit, 65 gain calculation unit, 66 audio signal generation unit, 71 additional metadata flag reading unit, 72 switching index reading unit, 73 interpolation processing unit

Claims

1. an acquisition unit that acquires encoded audio data including an independent flag obtained by encoding an audio signal of frames at a predetermined time interval, and encoded metadata corresponding to each of the frames; a decoding unit that decodes the encoded audio data to obtain an audio signal and decodes the encoded metadata to obtain a plurality of pieces of metadata corresponding to each of the frames; a rendering unit that performs rendering based on the audio signal obtained by the decoding, a plurality of pieces of metadata corresponding to each of the frames, and the independent flag in the frame; An information processing device comprising:

2. The metadata is data including position information indicating the position of an audio object. The information processing device according to claim 1 .

3. The rendering unit determines, for each of the frames, whether or not the frame corresponding to the independent flag is an independent frame based on the independent flag, and performs rendering based on the audio signal obtained by decoding and the plurality of metadata in accordance with the determination result. The information processing device according to claim 2 .

4. The rendering unit performs rendering using VBAP based on the audio signal obtained by the decoding and a plurality of pieces of metadata corresponding to each of the frames. The information processing device according to claim 3 .

5. Each of the plurality of metadata is metadata of a plurality of samples arranged at intervals of the number of samples obtained by dividing the number of samples constituting the frame by the number of the plurality of metadata. The information processing device according to claim 1 .

6. Each of the plurality of metadata is metadata of each of a plurality of samples indicated by each of a plurality of sample indexes. The information processing device according to claim 1 .

7. Each of the plurality of metadata is metadata of a plurality of samples arranged at intervals of a predetermined number of samples in the frame. The information processing device according to claim 1 .

8. The plurality of metadata include metadata for performing interpolation processing on VBAP gains of the samples of the audio signal calculated based on the metadata. The information processing device according to claim 1 .

9. If it is determined based on the independent flag that the frame corresponding to the independent flag is an independent frame, rendering is performed without using the VBAP gain of a frame prior to the current frame. The information processing device according to claim 1 .

10. If it is determined based on the independent flag that the frame corresponding to the independent flag is an independent frame, rendering is performed using additional metadata. The information processing device according to claim 1 .

11. Obtaining encoded audio data including an independent flag obtained by encoding an audio signal of frames at a predetermined time interval, and encoded metadata corresponding to each of the frames; decoding the encoded audio data to obtain an audio signal, and decoding the encoded metadata to obtain a plurality of metadata corresponding to each of the frames; performing rendering based on the audio signal obtained by the decoding, a plurality of pieces of metadata corresponding to each of the frames, and the independent flag in each of the frames; An information processing method including:

12. Obtaining encoded audio data including an independent flag obtained by encoding an audio signal of frames at a predetermined time interval, and encoded metadata corresponding to each of the frames; decoding the encoded audio data to obtain an audio signal, and decoding the encoded metadata to obtain a plurality of metadata corresponding to each of the frames; performing rendering based on the audio signal obtained by the decoding, a plurality of pieces of metadata corresponding to each of the frames, and the independent flag in each of the frames; A program that causes a computer to execute a process including the above.

Images