Encoding device and method, decoding device and method, and program
The encoding device and method allow content creators to control the sense of distance in object-based audio by encoding audio data with customizable distance control information, addressing the limitation of predetermined processing in existing technologies and improving distance reproduction.
Patent Information
- Application Number
- JP2021570021
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-01-10
- Filing Date
- 2020-12-25
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-12-25
AI Technical Summary
Existing object-based audio technologies lack the ability for content creators to control the sense of distance based on their intentions, as gain control and filter processing are predetermined, limiting the reproduction of frequency characteristics and volume changes.
An encoding device and method that includes an object encoding unit, metadata encoding unit, and a multiplexing unit to encode audio data and metadata with distance control information, allowing for customizable sense of distance control processing on the decoding side.
Enables content creators to realize distance control based on their intentions by encoding and decoding audio data with customizable distance control information, enhancing the sense of distance reproduction in object-based audio.
Smart Images

Figure 0007593333000001 
Figure 0007593333000002 
Figure 0007593333000003
Abstract
Description
[Technical field]
[0001] The present technology relates to an encoding device and method, a decoding device and method, and a program, and in particular to an encoding device and method, a decoding device and method, and a program that enable control of sense of distance based on the intention of a content creator. [Background technology]
[0002] In recent years, object-based audio technology has been attracting attention.
[0003] In object-based audio, object audio data is made up of a waveform signal for an audio object and metadata indicating localization information of the audio object represented by a relative position from a predetermined reference listening position.
[0004] Then, the waveform signal of the audio object is rendered into a signal with a desired number of channels by, for example, VBAP (Vector Based Amplitude Panning) based on the metadata and played back (for example, see Non-Patent Documents 1 and 2).
[0005] Furthermore, as a technology relating to object-based audio, a technology has been proposed that realizes audio reproduction with a higher degree of freedom, for example, by allowing a user to specify an arbitrary listening position (see, for example, Patent Document 1).
[0006] This technology corrects the position information of the audio object according to the listening position, and performs gain control and filter processing according to changes in the distance from the listening position to the audio object, thereby reproducing changes in frequency characteristics and volume that accompany changes in the user's listening position, i.e., the sense of distance to the audio object. [Prior art documents] [Non-patent literature]
[0007] [Non-Patent Document 1] ISO / IEC 23008-3 Information technology - High efficiency coding and media delivery in heterogeneous environments - Part 3: 3D audio [Non-Patent Document 2] Ville Pulkki, “Virtual Sound Source Positioning Using Vector Base Amplitude Panning”, Journal of AES, vol.45, no.6, pp.456-466, 1997 [Patent documents]
[0008] [Patent Document 1] International Publication No. 2015 / 107926 Summary of the Invention [Problem to be solved by the invention]
[0009] However, in the above-mentioned techniques, the gain control and filtering for reproducing the change in frequency characteristics and volume according to the distance from the listening position to the audio object are predetermined.
[0010] Therefore, even if a content creator wanted to reproduce a sense of distance by changing the frequency characteristics or volume differently, it was not possible to do so, meaning that it was not possible to realize distance control based on the content creator's intention.
[0011] The present technology has been made in consideration of such circumstances, and makes it possible to realize sense of distance control based on the intentions of the content creator. [Means for solving the problem]
[0012] The encoding device according to a first aspect of the present technology includes an object encoding unit that encodes audio data of an object, a metadata encoding unit that encodes metadata including position information of the object, a distance control information determination unit that determines distance control information for distance control processing performed on the audio data, a distance control information encoding unit that encodes the distance control information, and a multiplexing unit that multiplexes the encoded audio data, the encoded metadata, and the encoded distance control information to generate encoded data.
[0013] The encoding method or program of a first aspect of the present technology includes the steps of encoding audio data of an object, encoding metadata including position information of the object, determining distance control information for a distance control process to be performed on the audio data, encoding the distance control information, multiplexing the encoded audio data, the encoded metadata, and the encoded distance control information, and generating encoded data.
[0014] In a first aspect of the present technology, audio data of an object is encoded, metadata including position information of the object is encoded, distance control information for a distance control process to be performed on the audio data is determined, the distance control information is encoded, and the encoded audio data, the encoded metadata, and the encoded distance control information are multiplexed to generate encoded data.
[0015] A decoding device according to a second aspect of the present technology includes a demultiplexing unit that demultiplexes encoded data and extracts encoded audio data of an object, encoded metadata including position information of the object, and encoded distance control information for distance control processing to be performed on the audio data, an object decoding unit that decodes the encoded audio data, a metadata decoding unit that decodes the encoded metadata, a distance control information decoding unit that decodes the encoded distance control information, a distance control processing unit that performs the distance control processing on the audio data of the object based on the distance control information, and a rendering processing unit that performs rendering processing based on the audio data obtained by the distance control processing and the metadata, and generates playback audio data for reproducing the sound of the object.
[0016] A decoding method or program according to a second aspect of the present technology includes the steps of demultiplexing encoded data, extracting encoded audio data of an object, encoded metadata including position information of the object, and encoded distance control information for a distance control process to be performed on the audio data, decoding the encoded audio data, decoding the encoded metadata, decoding the encoded distance control information, performing the distance control process on the audio data of the object based on the distance control information, performing a rendering process based on the audio data obtained by the distance control process and the metadata, and generating playback audio data for reproducing the sound of the object.
[0017] In a second aspect of the present technology, the encoded data is demultiplexed to extract encoded audio data of an object, encoded metadata including position information of the object, and encoded distance control information for a distance control process performed on the audio data, the encoded audio data is decoded, the encoded metadata is decoded, the encoded distance control information is decoded, the distance control process is performed on the audio data of the object based on the distance control information, a rendering process is performed based on the audio data obtained by the distance control process and the metadata, and playback audio data for reproducing the sound of the object is generated. [Brief description of the drawings]
[0018] [Figure 1] FIG. 1 illustrates an example of the configuration of an encoding device. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of a decoding device. [Diagram 3] FIG. 4 is a diagram illustrating an example of the configuration of a sense of distance control processing unit. [Figure 4] FIG. 2 is a diagram illustrating an example of the configuration of a reverb processing unit. [Diagram 5] 5A and 5B are diagrams illustrating an example of a control rule for a gain control process. [Figure 6] 11A and 11B are diagrams illustrating an example of a control rule for filtering by a high shelf filter. [Figure 7] 11A and 11B are diagrams illustrating an example of a control rule for filtering by a low-shelf filter. [Figure 8] 11A and 11B are diagrams illustrating examples of control rules for reverb processing. [Figure 9] FIG. 13 is a diagram illustrating the generation of a wet component. [Figure 10] FIG. 13 is a diagram illustrating the generation of a wet component. [Figure 11] FIG. 11 is a diagram showing an example of sense of distance control information. [Figure 12] FIG. 13 is a diagram illustrating an example of parameter configuration information for gain control. [Figure 13] 13 is a diagram illustrating an example of parameter configuration information for filter processing. FIG. [Figure 14] 13 is a diagram showing an example of parameter configuration information for reverb processing. FIG. [Figure 15] 11 is a flowchart illustrating an encoding process. [Figure 16] 13 is a flowchart illustrating a decoding process. [Figure 17] 11A and 11B are diagrams illustrating examples of tables and functions for obtaining gain values. [Figure 18] FIG. 13 is a diagram illustrating an example of parameter configuration information for gain control. [Figure 19] FIG. 11 is a diagram showing an example of sense of distance control information. [Figure 20] FIG. 11 is a diagram showing an example of sense of distance control information. [Figure 21] FIG. 4 is a diagram illustrating an example of the configuration of a sense of distance control processing unit. [Figure 22] FIG. 11 is a diagram showing an example of sense of distance control information. [Figure 23] FIG. 1 illustrates an example of the configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0019] Hereinafter, embodiments to which the present technology is applied will be described with reference to the drawings.
[0020] First embodiment <Example of the configuration of the encoding device> The present technology relates to the playback of object-based audio content, which consists of the sounds of one or more audio objects.
[0021] Hereinafter, audio objects will also be simply referred to as objects, and audio contents will also be simply referred to as contents.
[0022] With this technology, distance control information for the distance control process that reproduces the sense of distance from the listening position to the object set by the content creator is transmitted to the decoding side along with the audio data of the object, making it possible to realize distance control based on the intentions of the content creator.
[0023] Here, distance control processing refers to processing for reproducing the sense of distance from the listening position to an object when playing the sound of the object, i.e., processing for adding a sense of distance to the sound of the object, and is signal processing that is realized by executing any one or a combination of multiple processes.
[0024] Specifically, for example, the distance perception control process includes gain control process for audio data, filter process for adding frequency characteristics and various sound effects, reverb process, and the like.
[0025] The information for enabling such distance control processing to be reconstructed on the decoding side is distance control information, which includes configuration information and control rule information. In other words, the distance control information is made up of configuration information and control rule information.
[0026] For example, the configuration information that constitutes the sense of distance control information is information that indicates one or more signal processing operations that are combined to realize the sense of distance control processing, which is obtained by parameterizing the configuration of the sense of distance control processing set by the content creator.
[0027] More specifically, the configuration information indicates how many signal processes the sense of distance control process is made up of, what kind of signal processes these are, and in what order they are executed.
[0028] Note that when one or more signal processes constituting the sense of distance control process and the order in which these signal processes are performed are determined in advance, the sense of distance control information does not necessarily need to include configuration information.
[0029] In addition, the control rule information is information for obtaining parameters used in each signal processing that constitutes the distance perception control processing, which is obtained by parameterizing the control rules for each signal processing that constitutes the distance perception control processing set by the content creator.
[0030] More specifically, the control rule information indicates what parameters are used for each signal process that constitutes the distance perception control processing, and by what control rules these parameters change depending on the distance from the listening position to the object.
[0031] On the encoding side, such distance control information and the audio data of each object are encoded and transmitted to the decoding side.
[0032] Moreover, on the decoding side, the distance control process is reconstructed based on the distance control information, and the distance control process is performed on the audio data of each object.
[0033] At this time, parameters according to the distance from the listening position to the object are determined based on the control rule information included in the sense of distance control information, and signal processing constituting the sense of distance control processing is carried out based on the parameters.
[0034] Then, 3D audio rendering processing is performed based on the audio data obtained by the distance control processing, and playback audio data for playing back the sound of the content, that is, the sound of the object, is generated.
[0035] Now, a more specific embodiment to which the present technology is applied will be described below.
[0036] For example, a content playback system to which the present technology is applied comprises an encoding device that encodes audio data and distance control information of one or more objects that constitute the content to generate encoded data, and a decoding device that receives the encoded data and generates playback audio data.
[0037] An encoding device constituting such a content playback system is configured, for example, as shown in FIG.
[0038] The encoding device 11 shown in FIG. 1 includes an object encoding section 21, a metadata encoding section 22, a sense of distance control information determination section 23, a sense of distance control information encoding section 24, and a multiplexing section 25.
[0039] Audio data of one or more objects constituting the content is supplied to the object coding unit 21. This audio data is a waveform signal (audio signal) for reproducing the sound of the object.
[0040] The object coding unit 21 codes the audio data of each object supplied thereto, and supplies the resulting coded audio data to a multiplexing unit 25 .
[0041] The metadata encoding unit 22 is supplied with metadata of the audio data of each object.
[0042] The metadata includes at least position information indicating the absolute position of the object in space. This position information is expressed as coordinates indicating the position of the object in an absolute coordinate system, i.e., a three-dimensional orthogonal coordinate system based on a predetermined position in space. The metadata may also include gain information for performing gain control (gain correction) on the audio data of the object.
[0043] The metadata encoding unit 22 encodes the metadata of each object supplied thereto, and supplies the encoded metadata obtained as a result to the multiplexing unit 25 .
[0044] The sense-of-distance control information determination section 23 determines sense-of-distance control information in response to a designation operation by the user, and supplies the determined sense-of-distance control information to a sense-of-distance control information coding section 24 .
[0045] For example, the sense of distance control information determination unit 23 acquires configuration information and control rule information designated by the user in response to a designation operation by the user, and determines sense of distance control information consisting of the configuration information and control rule information.
[0046] Also, for example, the sense of distance control information determination unit 23 may determine the sense of distance control information based on audio data of each object of the content, information about the content such as the genre of the content, information about the reproduction space of the content, and the like.
[0047] Note that, when the signal processing that constitutes the sense of distance control processing and the processing order of these signal processing are known on the decoding side, the sense of distance control information does not need to include configuration information.
[0048] The sense of distance control information encoder 24 encodes the sense of distance control information supplied from the sense of distance control information determiner 23 , and supplies the encoded sense of distance control information obtained as a result to a multiplexer 25 .
[0049] The multiplexing unit 25 multiplexes the encoded audio data supplied from the object encoding unit 21, the encoded metadata supplied from the metadata encoding unit 22, and the encoded distance control information supplied from the distance control information encoding unit 24, to generate encoded data (code string). The multiplexing unit 25 transmits (transmits) the encoded data obtained by multiplexing to a decoding device via a communication network or the like.
[0050] <Configuration Example of Decoding Device> Moreover, the decoding device constituting the content reproduction system is configured, for example, as shown in FIG.
[0051] The decoding device 51 shown in Figure 2 has a demultiplexing unit 61, an object decoding unit 62, a metadata decoding unit 63, a distance control information decoding unit 64, a user interface 65, a distance calculation unit 66, a distance control processing unit 67, and a 3D audio rendering processing unit 68.
[0052] The demultiplexing unit 61 receives the encoded data transmitted from the encoding device 11 and demultiplexes the received encoded data to extract encoded audio data, encoded metadata, and encoded distance control information from the encoded data.
[0053] The demultiplexer 61 supplies the encoded audio data to an object decoder 62 , supplies the encoded metadata to a metadata decoder 63 , and supplies the encoded sense of distance control information to a sense of distance control information decoder 64 .
[0054] The object decoding unit 62 decodes the encoded audio data supplied from the demultiplexing unit 61 , and supplies the resulting audio data to a distance control processing unit 67 .
[0055] The metadata decoding unit 63 decodes the encoded metadata supplied from the demultiplexing unit 61 , and supplies the resulting metadata to a sense of distance control processing unit 67 and a distance calculation unit 66 .
[0056] The sense of distance control information decoder 64 decodes the encoded sense of distance control information supplied from the demultiplexer 61 , and supplies the sense of distance control information obtained as a result to a sense of distance control processor 67 .
[0057] The user interface 65 supplies listening position information indicating the listening position designated by the user to a distance calculation unit 66, a sense of distance control processing unit 67, and a 3D audio rendering processing unit 68 in response to, for example, a user operation.
[0058] Here, the listening position indicated by the listening position information is the absolute position of the listener who listens to the sound of the content within the reproduction space. For example, the listening position information may be coordinates indicating the listening position in the same absolute coordinate system as the position information of the object included in the metadata.
[0059] The distance calculation unit 66 calculates the distance from the listening position to the object for each object based on the metadata supplied from the metadata decoding unit 63 and the listening position information supplied from the user interface 65, and supplies distance information indicating the calculation results to the distance control processing unit 67.
[0060] The distance control processing unit 67 performs distance control processing on the audio data supplied from the object decoding unit 62 based on the metadata supplied from the metadata decoding unit 63, the distance control information supplied from the distance control information decoding unit 64, the listening position information supplied from the user interface 65, and the distance information supplied from the distance calculation unit 66.
[0061] At this time, the distance control processing unit 67 obtains parameters based on the control rule information and distance information, and performs distance control processing on the audio data based on the obtained parameters.
[0062] By such a distance perception control process, audio data of the dry component and audio data of the wet component of the object are generated.
[0063] Here, the audio data of the dry component is audio data such as a direct sound component of an object, obtained by performing one or more processes on the audio data of the original object.
[0064] As metadata for this dry component audio data, the metadata for the original object, that is, the metadata output from the metadata decoding unit 63, is used.
[0065] The wet component audio data is audio data such as a reverberation component of the sound of an object, obtained by performing one or more processes on the original audio data of the object.
[0066] Therefore, generating audio data of the wet component can be said to be generating audio data of a new object related to the original object.
[0067] In the distance control processing unit 67, necessary information from among the metadata of the original object, the control rule information, the distance information, and the listening position information is appropriately used to generate metadata of the wet component audio data.
[0068] This metadata includes at least position information indicating the location of the object in the wet component.
[0069] For example, the position information of an object for the wet component may be expressed as polar coordinates expressed by a horizontal angle (horizontal angle) indicating the position of the object as seen by the listener in the playback space, a vertical angle (vertical angle), and a radius indicating the distance from the listening position to the object.
[0070] The distance control processor 67 supplies the audio data and metadata of the dry component and the audio data and metadata of the wet component to a 3D audio rendering processor 68.
[0071] The 3D audio rendering processing unit 68 performs 3D audio rendering processing based on the audio data and metadata supplied from the distance control processing unit 67 and the listening position information supplied from the user interface 65, and generates playback audio data.
[0072] For example, the 3D audio rendering processor 68 performs 3D audio rendering such as VBAP, which is a rendering process in a polar coordinate system.
[0073] In this case, for the dry component audio data, the 3D audio rendering processor 68 generates position information expressed in polar coordinates based on the position information included in the metadata of the dry component object and the listening position information, and uses the obtained position information for rendering processing. This position information is polar coordinates expressed by horizontal and vertical angles indicating the relative position of the object as seen by the listener, and a radius indicating the distance from the listening position to the object.
[0074] By such a rendering process, for example, multi-channel playback audio data is generated that is made up of audio data for channels corresponding to each of a plurality of speakers that make up a speaker system that is the output destination.
[0075] The 3D audio rendering processor 68 outputs the reproduced audio data obtained by the rendering process to the subsequent stage.
[0076] <Configuration example of distance sense control processing unit> Next, a specific example of the configuration of the distance control processing unit 67 of the decoding device 51 will be described.
[0077] Note that an example will be described here in which the configuration of the sense of distance control processing unit 67, that is, one or more processes that constitute the sense of distance control process, and the order of these processes are predetermined.
[0078] In such a case, the sense of distance control processing section 67 is configured as shown in FIG.
[0079] The distance control processing section 67 shown in FIG. 3 has a gain control section 101 , a high-shelf filter processing section 102 , a low-shelf filter processing section 103 , and a reverb processing section 104 .
[0080] In this example, the sense of distance control process includes a gain control process, a filtering process using a high-shelf filter, a filtering process using a low-shelf filter, and a reverb process, which are executed in this order.
[0081] The gain control unit 101 performs gain control on the audio data of the object supplied from the object decoding unit 62 using parameters (gain values) corresponding to the control rule information and distance information, and supplies the resulting audio data to the high shelf filter processing unit 102.
[0082] The high shelf filter processing unit 102 performs filtering on the audio data supplied from the gain control unit 101 using a high shelf filter determined by parameters corresponding to the control rule information and distance information, and supplies the resulting audio data to the low shelf filter processing unit 103.
[0083] In filtering using a high shelf filter, the gain of the high frequencies of the audio data is suppressed according to the distance from the listening position to the object.
[0084] The low-shelf filter processing unit 103 performs filtering on the audio data supplied from the high-shelf filter processing unit 102 using a low-shelf filter determined by parameters corresponding to the control rule information and distance information.
[0085] In filtering using a low-shelf filter, the low frequencies of the audio data are boosted (emphasized) according to the distance from the listening position to the object.
[0086] The low-shelf filter processing unit 103 supplies the audio data obtained by the filtering process to the 3D audio rendering processing unit 68 and the reverb processing unit 104 .
[0087] Here, the audio data output from the low-shelf filter processor 103 is the audio data of the original object described above, that is, the audio data of the dry component of the object.
[0088] The reverb processing unit 104 performs reverb processing on the audio data supplied from the low-shelf filter processing unit 103 using parameters (gain) according to the control rule information and distance information, and supplies the resulting audio data to the 3D audio rendering processing unit 68.
[0089] Here, the audio data output from the reverberation processor 104 is audio data of wet components, which are reverberation components of the original object described above, or in other words, audio data of the wet component object.
[0090] <Example of reverb processing section configuration> More specifically, the reverb processing section 104 is configured as shown in FIG.
[0091] In the example shown in FIG. 4, the reverb processing unit 104 has a gain control unit 141, a delay generation unit 142, a comb filter group 143, an all-pass filter group 144, an adder unit 145, an adder unit 146, a delay generation unit 147, a comb filter group 148, an all-pass filter group 149, an adder unit 150, and an adder unit 151.
[0092] In this example, reverb processing generates stereo reverberation components from mono audio data, that is, audio data of two wet components located on the left and right of the original object.
[0093] The gain control unit 141 performs gain control processing (gain correction processing) on the dry component audio data supplied from the low shelf filter processing unit 103 based on a wet gain value obtained from control rule information and distance information, and supplies the resulting audio data to the delay generation unit 142 and the delay generation unit 147.
[0094] The delay generation unit 142 delays the audio data supplied from the gain control unit 141 by holding it for a certain period of time, and supplies the delayed audio data to the comb filter group 143 .
[0095] In addition, the delay generation unit 142 supplies two pieces of audio data to the addition unit 145, the audio data being obtained by delaying the audio data supplied from the gain control unit 141 and having a different delay amount from the audio data supplied to the comb filter group 143 and having different delay amounts from each other.
[0096] Comb filter group 143 is made up of a plurality of comb filters, and performs filtering on the audio data supplied from delay generation unit 142 using the plurality of comb filters, and supplies the resulting audio data to all-pass filter group 144.
[0097] All-pass filter group 144 is made up of a plurality of all-pass filters, and performs filtering on the audio data supplied from comb filter group 143 using the plurality of all-pass filters, and supplies the resulting audio data to addition section 146.
[0098] The addition unit 145 adds the two pieces of audio data supplied from the delay generation unit 142 and supplies the result to the addition unit 146 .
[0099] The adder unit 146 adds the audio data supplied from the all-pass filter group 144 and the audio data supplied from the adder unit 145, and supplies the resulting wet component audio data to the 3D audio rendering processing unit 68.
[0100] The delay generation unit 147 delays the audio data supplied from the gain control unit 141 by holding it for a certain period of time, and supplies the delayed audio data to the comb filter group 148 .
[0101] In addition, the delay generation unit 147 supplies two pieces of audio data to the addition unit 150, the audio data being obtained by delaying the audio data supplied from the gain control unit 141 and having a different delay amount from the audio data supplied to the comb filter group 148 and having different delay amounts from each other.
[0102] Comb filter group 148 is made up of a plurality of comb filters, and performs filtering on the audio data supplied from delay generation unit 147 using the plurality of comb filters, and supplies the resulting audio data to all-pass filter group 149 .
[0103] All-pass filter group 149 is made up of a plurality of all-pass filters, and performs filtering on the audio data supplied from comb filter group 148 using the plurality of all-pass filters, and supplies the resulting audio data to addition section 151.
[0104] The addition unit 150 adds the two pieces of audio data supplied from the delay generation unit 147 and supplies the result to an addition unit 151 .
[0105] The adder unit 151 adds the audio data supplied from the all-pass filter group 149 and the audio data supplied from the adder unit 150, and supplies the resulting wet component audio data to the 3D audio rendering processing unit 68.
[0106] Although an example in which stereo (two) wet components are generated for one object has been described here, one wet component may be generated for one object, or three or more wet components may be generated. Also, the configuration of the reverb processing unit 104 is not limited to the configuration shown in Fig. 4, and may be any other configuration.
[0107] <Parameter control rules> As described above, in each processing block constituting the sense of distance control processor 67, the parameters used in the processing in that processing block, that is, the processing characteristics, change according to the distance from the listening position to the object.
[0108] Here, an example of parameters according to the distance from the listening position to the object, that is, an example of parameter control rules, will be described.
[0109] For example, in the gain control section 101, a gain value to be used in the gain control process is determined as a parameter according to the distance from the listening position to an object.
[0110] In this case, the gain value changes according to the distance from the listening position to the object, for example as shown in FIG.
[0111] For example, the portion indicated by the arrow Q11 shows the change in gain value according to distance. That is, the vertical axis shows the gain value as a parameter, and the horizontal axis shows the distance from the listening position to the object.
[0112] As shown by the broken line L11, when the distance d from the listening position to the object is between the predetermined minimum value Min and D0, the gain value is 0.0 dB, and when the distance d is between D0 and D1, the gain value decreases linearly as the distance d increases. Also, when the distance d is between D1 and the predetermined maximum value Max, the gain value is -40.0 dB.
[0113] From this, it can be seen that in the example shown in FIG. 5, as the distance d increases, control is performed to suppress the gain of the audio data.
[0114] As a specific example, when the distance d is 1 m (= D0) or less, the gain value is set to 0.0 dB, and when the distance d is between 1 m and 100 m (= D1), the gain value can be changed linearly to -40.0 dB as the distance d increases.
[0115] Here, if a point at which a parameter changes is called a control change point, in the example of FIG. 5, the point (position) on the broken line L11 where the distance d=D0 and the point where the distance d=D1 are the control change points.
[0116] In this case, for example, by transmitting a gain value of "0.0" at distance d = D0 corresponding to the control change point and a gain value of "-40.0" at distance d = D1 to decoding device 51 as shown by arrow Q12, decoding device 51 can obtain the gain value at any distance d.
[0117] Moreover, in the high shelf filter processing unit 102, as indicated by an arrow Q21 in FIG. 6, for example, filtering is performed to suppress the gain in the high frequency range as the distance d from the listening position to the object increases.
[0118] In the portion indicated by the arrow Q21, the vertical axis indicates the gain value as a parameter, and the horizontal axis indicates the distance d from the listening position to the object.
[0119] In particular, in this example, the high shelf filter realized by the high shelf filter processing unit 102 is determined by a cutoff frequency Fc, a Q value indicating sharpness, and a gain value at the cutoff frequency Fc.
[0120] In other words, in the high shelf filter processing unit 102, filtering is performed by a high shelf filter determined by the parameters of the cutoff frequency Fc, the Q value, and the gain value.
[0121] A broken line L21 in the portion indicated by the arrow Q21 indicates the gain value at the cutoff frequency Fc determined for the distance d.
[0122] In this example, when the distance d is between the minimum value Min and D0, the gain value is 0.0 dB, and when the distance d is between D0 and D1, the gain value decreases linearly as the distance d increases.
[0123] Similarly, when the distance d is between D2 and D3 and when the distance d is between D3 and D4, the gain value decreases linearly as the distance d increases. Furthermore, when the distance d is between D4 and the maximum value Max, the gain value is -12.0 dB.
[0124] From this, it can be seen that in the example shown in FIG. 6, as the distance d increases, control is performed to suppress the gain of the frequency components in the vicinity of the cutoff frequency Fc in the audio data.
[0125] As a specific example, when the distance d is 1 m (= D0) or less, frequency components above 6 kHz, which is the cutoff frequency Fc, are passed through, and when the distance d is between 1 m and 100 m (= D4), the frequency components above 6 kHz are changed to -12.0 dB as the distance d increases.
[0126] Furthermore, to realize such a high shelf filter in the decoding device 51, it is sufficient to transmit the parameters, that is, the cutoff frequency Fc, the Q value, and the gain value, only for the five control change points of distances d=D0, D1, D2, D3, and D4, as shown by the arrow Q22.
[0127] Although an example will be described here in which the cutoff frequency Fc is 6 kHz and the Q value is 2.0 regardless of the distance d, these cutoff frequency Fc and Q value may also be changed according to the distance d.
[0128] Furthermore, in the low-shelf filter processing unit 103, as indicated by an arrow Q31 in FIG. 7, for example, filtering is performed to amplify the low-frequency gain as the distance d from the listening position to the object becomes smaller.
[0129] In the portion indicated by arrow Q31, the vertical axis indicates the gain value as a parameter, and the horizontal axis indicates the distance d from the listening position to the object.
[0130] In particular, in this example, the low-shelf filter realized by the low-shelf filter processing unit 103 is determined by a cut-off frequency Fc, a Q value indicating sharpness, and a gain value at the cut-off frequency Fc.
[0131] In other words, the low-shelf filter processor 103 performs filtering using a low-shelf filter determined by the parameters of the cutoff frequency Fc, the Q value, and the gain value.
[0132] A broken line L31 in the portion indicated by the arrow Q31 indicates a gain value at the cutoff frequency Fc determined for the distance d.
[0133] In this example, the gain value is 3.0 dB when the distance d is between the minimum value Min and D0, and the gain value decreases linearly as the distance d increases when the distance d is between D0 and D1. Also, the gain value is 0.0 dB when the distance d is between D1 and the maximum value Max.
[0134] From this, it can be seen that in the example shown in FIG. 7, as the distance d becomes smaller, control is performed such that the gain of the frequency components in the vicinity of the cutoff frequency Fc in the audio data is amplified.
[0135] As a specific example, when the distance d is 3 m (= D1) or more, frequency components below 200 Hz, which is the cutoff frequency Fc, are passed through, and when the distance d is between 3 m and 10 cm (= D0), the frequency components below 200 Hz are changed up to +3.0 dB as the distance d decreases.
[0136] To realize such a low-shelf filter in the decoding device 51, for example, it is sufficient to transmit the parameters, cutoff frequency Fc, Q value, and gain value, only for two control change points at distances d=D0 and D1, as indicated by arrow Q32.
[0137] Although an example will be described here in which the cutoff frequency Fc is 200 Hz and the Q value is 2.0, regardless of the distance d, these cutoff frequency Fc and Q value may also be changed according to the distance d.
[0138] Furthermore, in the reverb processing section 104, as indicated by an arrow Q41 in FIG. 8, reverb processing is performed in which the gain of the wet component (wet gain value) increases as the distance d from the listening position to the object increases.
[0139] In other words, as the distance d increases, the ratio of the wet components (reverberation components) generated by the reverb process to the dry components increases. Note that the wet gain value referred to here is the gain value used in the gain control in the gain control section 141 shown in FIG. 4, for example.
[0140] In the portion indicated by the arrow Q41, the vertical axis indicates the wet gain value as a parameter, and the horizontal axis indicates the distance d from the listening position to the object. Also, the broken line L41 indicates the wet gain value determined with respect to the distance d.
[0141] As shown by the broken line L41, when the distance d from the listening position to the object is between the minimum value Min and D0, the wet gain value is minus infinity (-InfdB), and when the distance d is between D0 and D1, the wet gain value increases linearly as the distance d increases. Also, when the distance d is between D1 and the maximum value Max, the wet gain value is -3.0dB.
[0142] From this, it can be seen that in the example shown in FIG. 8, control is performed such that the wet component becomes larger as the distance d becomes larger.
[0143] As a specific example, when the distance d is 1 m (= D0) or less, the gain of the wet component (wet gain value) is -InfdB, and when the distance d is between 1 m and 50 m (= D1), the gain can be changed linearly to -3.0 dB as the distance d increases.
[0144] Furthermore, to realize such reverb processing in the decoding device 51, it is only necessary to transmit the wet gain values, which are parameters, for only two control change points at distances d=D0 and D1, as indicated by arrow Q42, for example.
[0145] Furthermore, in reverb processing, audio data for any number of wet components (reverberation components) can be generated.
[0146] Specifically, for example, as shown in FIG. 9, audio data of a single object, that is, monaural audio data, can be used to generate audio data of stereo reverberation components.
[0147] In this example, the listening position is the origin O of an XYZ coordinate system, which is a three-dimensional orthogonal coordinate system in the reproduction space, and one object OB11 is placed in the reproduction space.
[0148] Now, the position of any object in the playback space is expressed by a horizontal angle indicating its horizontal position as seen from the origin O, and a vertical angle indicating its vertical position as seen from the origin O, and the position of object OB11 is expressed as (az, el) in terms of horizontal angle az and vertical angle el.
[0149] The horizontal angle az is the angle between the line LN' and the Z axis when the line LN is the line connecting the origin O and the object OB11 and the line LN' is the line obtained by projecting the line LN onto the XZ plane. The vertical angle el is the angle between the line LN and the XZ plane.
[0150] In the example of FIG. 9, two objects OB12 and OB13 are generated as wet component objects for an object OB11.
[0151] In particular, here, the objects OB12 and OB13 are disposed at positions symmetrical to the left and right of the object OB11 when viewed from the origin O.
[0152] That is, the objects OB12 and OB13 are disposed at positions shifted 60 degrees to the left and right relative to the object OB11.
[0153] Therefore, the position of object OB12 is the position (az+60, el) represented by the horizontal angle (az+60) and the vertical angle el, and the position of object OB13 is the position (az-60, el) represented by the horizontal angle (az-60) and the vertical angle el.
[0154] In this way, when generating wet components at positions symmetrical to the left and right of the object OB11, the positions of those wet components can be specified by offset angles with respect to the position of the object OB11. For example, in this example, a horizontal offset angle of ±60 degrees can be specified.
[0155] Note that, although an example has been described here in which two wet components, one on the left side and one on the right side, are generated for one object, any number of wet components may be generated for one object, such as generating wet components at each of the top, bottom, left, and right positions.
[0156] Furthermore, for example, when generating a symmetrical wet component as shown in FIG. 9, the offset angle that specifies the position of the wet component may be changed depending on the distance from the listening position to the object as shown in FIG. 10.
[0157] 10, the offset angle of the horizontal angle between the objects OB12 and OB13, which are the wet components shown in FIG. 9, is shown.
[0158] That is, in the portion indicated by arrow Q51, the vertical axis indicates the horizontal offset angle, and the horizontal axis indicates the distance d from the listening position to object OB11.
[0159] Moreover, the broken line L51 indicates the offset angle of the object OB12, which is the left wet component, determined for each distance d. In this example, the smaller the distance d, the larger the offset angle becomes, and the object OB12 is placed at a position farther away from the original object OB11.
[0160] On the other hand, the broken line L52 indicates the offset angle of the object OB13, which is the wet component on the right side, determined for each distance d. In this example, the offset angle decreases as the distance d decreases, and the object OB13 is placed at a position farther away from the original object OB11.
[0161] In this manner, when the offset angle changes according to the distance d, for example, as shown by arrow Q52, by transmitting the offset angle to the decoding device 51 only for the control change point at distance d=D0, the wet component can be generated at the position intended by the content creator.
[0162] In this way, by performing distance perception control processing with a configuration and parameters according to the distance d from the listening position to the object, it is possible to reproduce a proper sense of distance, i.e., to allow the listener to sense the distance to the object.
[0163] In this case, if the content creator freely determines the parameters for each distance d, it is possible to realize distance control based on the content creator's intention.
[0164] Note that the control rules for parameters according to distance d described above are merely examples, and content creators can freely specify control rules to change the way in which the distance to an object is perceived.
[0165] For example, because the way sound changes with distance is different outdoors and indoors, the control rules need to be changed depending on whether the space you want to reproduce is outdoors or indoors.
[0166] Therefore, for example, by determining (specifying) control rules according to the space that a content creator wants to reproduce in the content, it is possible to realize distance control based on the content creator's intentions, and to reproduce content with a higher sense of realism.
[0167] Furthermore, the sense of distance control processing unit 67 can further adjust parameters used in the sense of distance control processing in accordance with the playback environment of the content (playback audio data).
[0168] Specifically, for example, the gain of the wet component used in the reverb process, that is, the above-mentioned wet gain value, can be adjusted according to the playback environment of the content.
[0169] When content is actually played back in a real space using speakers, etc., reverberation occurs in the real space from the sound output from the speakers, etc. At this time, the level of reverberation that occurs varies depending on the real space in which the content is played back, that is, the playback environment.
[0170] For example, if content is played in a reverberant environment, the sound of the played content will have more reverberation added to it, which may cause the listener to feel a greater sense of distance than the sense of distance achieved by the distance control process, i.e., the sense of distance intended by the content creator, when the content is actually played.
[0171] Therefore, when there is little reverberation in the playback environment, distance control processing is performed according to a preset control rule, i.e., control rule information, but when there is relatively much reverberation in the playback environment, fine adjustment of the wet gain value determined according to the control rule may be performed.
[0172] Specifically, for example, assume that a user or the like operates the user interface 65 to input information related to the reverberation of the playback environment, such as information on the type of playback environment (e.g., outdoors or indoors), information indicating whether the playback environment has a lot of reverberation, etc. In such a case, the user interface 65 supplies the information related to the reverberation of the playback environment input by the user or the like to the distance control processing unit 67.
[0173] The distance control processor 67 then calculates a wet gain value based on the control rule information, the distance information, and the information on the reverberation of the reproduction environment supplied from the user interface 65 .
[0174] Specifically, the distance control processing unit 67 calculates a wet gain value based on the control rule information and distance information, and performs a process of determining whether or not the playback environment has a lot of reverberation based on information about the reverberation of the playback environment.
[0175] Here, for example, when information indicating that the playback environment has a lot of reverberation or type information indicating that the playback environment has a lot of reverberation is supplied as information regarding the reverberation of the playback environment, it is determined that the playback environment has a lot of reverberation.
[0176] Then, if the distance control processing unit 67 determines that the playback environment is not one with a lot of reverberation, that is, the playback environment has little reverberation, it supplies the calculated wet gain value to the reverb processing unit 104 as the final wet gain value.
[0177] In contrast, when the distance control processing unit 67 determines that the playback environment has a lot of reverberation, it corrects (adjusts) the calculated wet gain value with a predetermined correction value such as -6 dB, and supplies the corrected wet gain value to the reverb processing unit 104 as the final wet gain value.
[0178] The correction value for the wet gain value may be a predetermined value, or may be calculated by the distance control processing unit 67 based on information regarding the reverberation in the playback environment, i.e., the degree of reverberation in the playback environment.
[0179] By adjusting the wet gain value in this way depending on the playback environment, it is possible to improve the sense of distance that differs from that intended by the content creator, which is caused by the playback environment of the content.
[0180] Regarding transmission of distance control information Next, a method for transmitting the sense of distance control information described above will be described.
[0181] The sense of distance control information encoded by the sense of distance control information encoding unit 24 may have a structure shown in FIG.
[0182] In FIG. 11, “DistanceRender_Attn( )” indicates parameter configuration information indicating the control rule of the parameters used in the gain control unit 101.
[0183] Further, “DistanceRender_Filt( )” indicates parameter configuration information indicating a control rule of the parameters used in the high-shelf filter processing unit 102 or the low-shelf filter processing unit 103 .
[0184] Here, the high-shelf filter and the low-shelf filter can be expressed with the same parameter configuration, and therefore are described with the same parameter configuration information Syntax, DistanceRender_Filt(). Therefore, the distance control information includes parameter configuration information DistanceRender_Filt() of the high-shelf filter processing unit 102 and parameter configuration information DistanceRender_Filt() of the low-shelf filter processing unit 103.
[0185] Furthermore, “DistanceRender_Revb( )” indicates parameter configuration information that indicates the control rules of the parameters used in the reverb processing unit 104 .
[0186] The parameter configuration information DistanceRender_Attn(), the parameter configuration information DistanceRender_Filt(), and the parameter configuration information DistanceRender_Revb() included in the sense of distance control information correspond to the control rule information.
[0187] In the sense of distance control information shown in FIG. 11, parameter configuration information for the four processes constituting the sense of distance control process is stored in the order in which the processes are performed.
[0188] Therefore, based on the sense of distance control information, the decoding device 51 can identify the configuration of the sense of distance control processing unit 67 shown in Fig. 3. In other words, it is possible to identify from the sense of distance control information shown in Fig. 11 how many processes the sense of distance control process consists of, what kind of processes these processes are, and in what order they are performed. Therefore, in this example, it can be said that the sense of distance control information essentially includes configuration information.
[0189] Furthermore, the parameter configuration information DistanceRender_Attn(), the parameter configuration information DistanceRender_Filt(), and the parameter configuration information DistanceRender_Revb() shown in FIG. 11 are configured, for example, as shown in FIGS.
[0190] FIG. 12 is a diagram showing an example of the configuration of parameter configuration information DistanceRender_Attn() for gain control processing, that is, an example of Syntax.
[0191] In Fig. 12, "num_points" indicates the number of control change points of the parameters of the gain control process. For example, in the example shown in Fig. 5, the point (position) where the distance d = D0 and the point where the distance d = D1 are the control change points.
[0192] In the example of Fig. 12, the data includes "distance[i]" indicating the distance d corresponding to each control change point and a gain value "gain[i]" as a parameter at the distance d, for each control change point. By transmitting the distance distance[i] and gain value gain[i] of each control change point in this manner, the gain control shown in Fig. 5 can be realized in the decoding device 51.
[0193] FIG. 13 is a diagram showing an example of the configuration of parameter configuration information DistanceRender_Filt() for filtering, that is, an example of Syntax.
[0194] In FIG. 13, "filt_type" indicates an index indicating the filter type.
[0195] For example, the index filt_type "0" indicates a low-shelf filter, the index filt_type "1" indicates a high-shelf filter, and the index filt_type "2" indicates a peak filter.
[0196] Moreover, the index filt_type "3" indicates a low-pass filter, and the index filt_type "4" indicates a high-pass filter.
[0197] Therefore, for example, if the value of the index filt_type is "0", it is understood that this parameter configuration information DistanceRender_Filt() includes information on parameters for specifying the configuration of a low-shelf filter.
[0198] In the example shown in FIG. 3, a high-shelf filter and a low-shelf filter have been described as examples of filters in the filter processing that constitutes the distance perception control processing.
[0199] In contrast to this, in the example shown in FIG. 13, a peak filter, a low-pass filter, a high-pass filter, and the like can also be used.
[0200] In addition, the filters for the filter processing that constitutes the distance control process may be made to use only some of a low shelf filter, a high shelf filter, a peak filter, a low pass filter, and a high pass filter, or other filters may also be used.
[0201] In the parameter configuration information DistanceRender_Filt() shown in FIG. 13, the area following the index filt_type includes parameters for specifying the configuration of the filter indicated by the index filt_type.
[0202] That is, "num_points" indicates the number of control change points of the parameters of the filter processing.
[0203] It also includes, for each control change point indicated by "num_points", a "distance[i]" indicating the distance d corresponding to the control change point, a frequency "freq[i]", a Q value "Q[i]", and a gain value "gain[i]" as parameters at that distance d.
[0204] For example, if the index filt_type is “0”, which indicates a low-shelf filter, the parameters frequency “freq[i]”, Q value “Q[i]”, and gain value “gain[i]” correspond to the cutoff frequency Fc, Q value, and gain value shown in FIG. 7.
[0205] Note that the frequency freq[i] is the cutoff frequency when the filter type is a low-shelf filter, a high-shelf filter, a low-pass filter, or a high-pass filter, but is the center frequency when the filter type is a peak filter.
[0206] By transmitting the distance distance[i], frequency “freq[i]”, Q value “Q[i]”, and gain value “gain[i]” of each control change point as described above, it is possible to realize the high shelf filter shown in FIG. 6 and the low shelf filter shown in FIG. 7 in the decoding device 51.
[0207] FIG. 14 is a diagram showing an example of the configuration of parameter configuration information DistanceRender_Revb() for reverb processing, that is, an example of Syntax.
[0208] 14, "num_points" indicates the number of control change points of the reverb processing parameters, and in this example, "distance[i]" indicating the distance d corresponding to each control change point and the wet gain value "wet_gain[i]" as the parameter at that distance d are included for each control change point. This wet gain value wet_gain[i] corresponds to the wet gain value shown in FIG. 8, for example.
[0209] Also, in FIG. 14, "num_wetobjs" indicates the number of wet components to be generated, that is, the number of wet component objects, and offset angles indicating the positions of the wet components are stored for the number of wet components.
[0210] That is, "wet_azimuth_offset[i][j]" indicates the horizontal angle offset angle of the jth wet component (object) at the distance distance[i] corresponding to the ith control change point. This offset angle wet_azimuth_offset[i][j] corresponds to the horizontal angle offset angle shown in FIG. 10, for example.
[0211] Similarly, "wet_elevation_offset[i][j]" indicates the offset angle of the vertical angle of the jth wet component at the distance distance[i] corresponding to the i-th control change point.
[0212] The number of wet components to be generated, num_wetobjs, is determined by the reverb processing to be performed by the decoding device 51, and for example, the number of wet components, num_wetobjs, is assumed to be given from the outside.
[0213] Thus, in the example of Figure 14, the distance distance[i] and wet gain value wet_gain[i] at each control change point, and the offset angle wet_azimuth_offset[i][j] and offset angle wet_elevation_offset[i][j] of each wet component are transmitted to the decoding device 51.
[0214] In this way, the decoding device 51 can realize, for example, the reverb processing unit 104 shown in FIG. 4, and can obtain audio data of the dry component and audio data and metadata of each wet component.
[0215] <Description of Encoding Process> Next, the operation of the content reproduction system will be described.
[0216] First, the encoding process performed by the encoding device 11 will be described with reference to the flowchart in FIG.
[0217] In step S 11 , the object coding unit 21 codes the audio data of each object supplied thereto, and supplies the resulting coded audio data to the multiplexing unit 25 .
[0218] In step S 12 , the metadata encoding unit 22 encodes the metadata of each object supplied thereto, and supplies the resulting encoded metadata to the multiplexing unit 25 .
[0219] In step S13, the sense-of-distance control information determination section 23 determines sense-of-distance control information in response to a designation operation by the user, and supplies the determined sense-of-distance control information to the sense-of-distance control information coding section 24.
[0220] In step S14, the sense of distance control information encoder 24 encodes the sense of distance control information supplied from the sense of distance control information determiner 23, and supplies the obtained encoded sense of distance control information to the multiplexer 25. In this way, the sense of distance control information (encoded sense of distance control information) shown in FIG.
[0221] In step S15, the multiplexing unit 25 multiplexes the encoded audio data from the object encoding unit 21, the encoded metadata from the metadata encoding unit 22, and the encoded distance control information from the distance control information encoding unit 24 to generate encoded data.
[0222] In step S16, the multiplexing unit 25 transmits the encoded data obtained by multiplexing to the decoding device 51 via a communication network or the like, and the encoding process ends.
[0223] In this manner, the encoding device 11 generates encoded data including the sense of distance control information, and transmits it to the decoding device 51.
[0224] In this way, by transmitting the sense of distance control information in addition to the audio data and metadata of each object to the decoding device 51, it becomes possible to realize the sense of distance control based on the intention of the content creator on the decoding device 51 side.
[0225] <Description of Decryption Process> 15 is performed in the encoding device 11, and then a decoding process is performed in the decoding device 51. The decoding process by the decoding device 51 will be described below with reference to the flowchart in FIG.
[0226] In step S 41 , the demultiplexer 61 receives the encoded data transmitted from the encoding device 11 .
[0227] In step S42, the demultiplexer 61 demultiplexes the received encoded data, and extracts encoded audio data, encoded metadata, and encoded sense of distance control information from the encoded data.
[0228] The demultiplexer 61 supplies the encoded audio data to an object decoder 62 , supplies the encoded metadata to a metadata decoder 63 , and supplies the encoded sense of distance control information to a sense of distance control information decoder 64 .
[0229] In step S 43 , the object decoding unit 62 decodes the encoded audio data supplied from the demultiplexing unit 61 , and supplies the obtained audio data to the distance control processing unit 67 .
[0230] In step S 44 , the metadata decoding unit 63 decodes the encoded metadata supplied from the demultiplexing unit 61 , and supplies the obtained metadata to the sense of distance control processing unit 67 and the distance calculation unit 66 .
[0231] In step S 45 , the sense of distance control information decoding unit 64 decodes the encoded sense of distance control information supplied from the demultiplexing unit 61 , and supplies the obtained sense of distance control information to the sense of distance control processing unit 67 .
[0232] In step S46, the distance calculation unit 66 calculates the distance from the listening position to the object based on the metadata supplied from the metadata decoding unit 63 and the listening position information supplied from the user interface 65, and supplies distance information indicating the calculation result to the sense of distance control processing unit 67. In step S46, distance information is obtained for each object.
[0233] In step S47, the distance control processing unit 67 performs distance control processing based on the audio data supplied from the object decoding unit 62, the metadata supplied from the metadata decoding unit 63, the distance control information supplied from the distance control information decoding unit 64, the listening position information supplied from the user interface 65, and the distance information supplied from the distance calculation unit 66.
[0234] For example, if the sense of distance control processing unit 67 has the configuration shown in FIG. 3 and is supplied with the sense of distance control information shown in FIG. 11, the sense of distance control processing unit 67 calculates parameters to be used in each process based on the sense of distance control information and distance information.
[0235] Specifically, for example, the sense of distance control processing unit 67 calculates a gain value at a distance d indicated by the distance information based on the distance distance[i] and gain value gain[i] of each control change point, and supplies the calculated gain value to the gain control unit 101 .
[0236] In addition, the distance control processing unit 67 determines the cutoff frequency, Q value, and gain value at the distance d indicated by the distance information based on the distance distance[i], frequency freq[i], Q value Q[i], and gain value gain[i] of each control change point of the high shelf filter, and supplies them to the high shelf filter processing unit 102.
[0237] This enables the high shelf filter processing unit 102 to construct a high shelf filter according to the distance d indicated by the distance information.
[0238] Similar to the case of the high-shelf filter, the distance control processing unit 67 determines the cutoff frequency, Q value, and gain value of the low-shelf filter at the distance d indicated by the distance information, and supplies them to the low-shelf filter processing unit 103. This enables the low-shelf filter processing unit 103 to construct a low-shelf filter according to the distance d indicated by the distance information.
[0239] Furthermore, the distance control processor 67 calculates a wet gain value at the distance d indicated by the distance information based on the distance distance[i] and the wet gain value wet_gain[i] of each control change point, and supplies the wet gain value to the reverb processor 104 .
[0240] In this way, the sense of distance control processing unit 67 shown in FIG. 3 is constructed from the sense of distance control information.
[0241] The distance control processor 67 also supplies the horizontal offset angle wet_azimuth_offset[i][j] and the vertical offset angle wet_elevation_offset[i][j], object metadata, and listening position information to the reverb processor 104.
[0242] The gain control section 101 performs gain control processing on the audio data of the object based on the gain value supplied from the distance control processing section 67, and supplies the audio data obtained as a result to a high shelf filter processing section .
[0243] The high shelf filter processing unit 102 performs filtering on the audio data supplied from the gain control unit 101 using a high shelf filter determined by the cutoff frequency, Q value, and gain value supplied from the distance control processing unit 67, and supplies the resulting audio data to the low shelf filter processing unit 103.
[0244] The low-shelf filter processing unit 103 performs filtering on the audio data supplied from the high-shelf filter processing unit 102 using a low-shelf filter determined by the cutoff frequency, Q value, and gain value supplied from the distance control processing unit 67 .
[0245] The distance control processing unit 67 supplies the audio data obtained by the filtering process in the low shelf filter processing unit 103 as audio data of the dry component together with metadata of the object of the dry component to the 3D audio rendering processing unit 68. This metadata of the dry component is metadata supplied from the metadata decoding unit 63.
[0246] Furthermore, the low-shelf filter processing unit 103 supplies the audio data obtained by the filtering process to the reverb processing unit 104 .
[0247] The reverb processing unit 104 then performs gain control based on the wet gain value on the dry component audio data, delay processing on the audio data, and filtering using a comb filter or all-pass filter, as described with reference to Figure 4, to generate wet component audio data.
[0248] In addition, the reverb processing unit 104 calculates position information of the wet component based on the offset angle wet_azimuth_offset[i][j] and the offset angle wet_elevation_offset[i][j], the metadata of the object (dry component), and the listening position information, and generates metadata of the wet component including the position information.
[0249] The reverberation processor 104 supplies the audio data and metadata of each wet component thus generated to the 3D audio rendering processor 68.
[0250] In step S48, the 3D audio rendering processor 68 performs rendering processing based on the audio data and metadata supplied from the distance control processor 67 and the listening position information supplied from the user interface 65, to generate playback audio data. For example, VBAP or the like is performed as the rendering processing in step S48.
[0251] When the playback audio data is generated, the 3D audio rendering processor 68 outputs the generated playback audio data to the subsequent stage, and the decoding process ends.
[0252] In this manner, the decoding device 51 performs the sense of distance control process based on the sense of distance control information included in the encoded data, and generates playback audio data. In this manner, it is possible to realize the sense of distance control based on the intention of the content creator.
[0253] First Modification of the First Embodiment <Other examples of parameter configuration information> In the above, the parameter configuration information has been described using the examples shown in Figures 12, 13, and 14, but the parameter configuration information is not limited to these and may be any information that can obtain parameters for the sense of distance control processing.
[0254] For example, for one or more processes constituting the distance control process, a table or a function (mathematical formula) for obtaining a parameter for the distance d from the listening position to the object may be prepared in advance, and an index indicating the table or function may be included in the parameter configuration information. In this case, the index indicating the table or function becomes control rule information indicating the control rule for the parameter.
[0255] In this way, when the control rule information is an index indicating a table or function for obtaining parameters, it is possible to prepare multiple tables or functions for obtaining gain values of the gain control processing as parameters, for example, as shown in FIG. 17.
[0256] In this example, for the index value "1", the function "20log10(1 / d)" is used to obtain the gain value of the gain control process. 2 " is prepared, and by substituting the distance d into this function, the gain value of the gain control process according to the distance d can be obtained.
[0257] Also, for example, a table is prepared for obtaining a gain value for the gain control process for an index value of "2." When this table is used, the greater the distance d, the smaller the gain value as a parameter.
[0258] The distance control processing unit 67 of the decoding device 51 holds in advance tables and functions associated with each of these indexes.
[0259] In such a case, for example, the parameter configuration information DistanceRender_Attn() shown in FIG. 11 has the configuration shown in FIG.
[0260] In the example of FIG. 18, the parameter configuration information DistanceRender_Attn() includes an index "index" that indicates a function or table specified by the content creator.
[0261] Therefore, in the distance control processing unit 67, a table or function stored in association with this index is read out, and a gain value is calculated as a parameter based on the read out table or function and the distance d from the listening position to the object.
[0262] In this way, by defining multiple patterns, i.e., multiple tables and functions, for obtaining parameters according to the distance d in advance, content creators can specify (select) the desired pattern from among those patterns, thereby enabling distance control processing to be performed in accordance with their own intentions.
[0263] In the above description, an example has been described in which a table or function for obtaining parameters for gain control processing is specified by an index. However, the present invention is not limited to this example, and parameter control rules can be specified by an index in the same manner in filter processing such as a high shelf filter or reverb processing.
[0264] Second Modification of the First Embodiment Other examples of distance control information Furthermore, in the above description, an example has been described in which parameters are determined for all objects according to the distance d using the same control rule, but it is also possible to set (specify) parameter control rules for each object.
[0265] In such a case, the sense of distance control information has a structure shown in FIG.
[0266] In the example shown in FIG. 19, "num_objs" indicates the number of objects that make up the content, and for example, the number of objects, num_objs, is given to the sense of distance control information determination unit 23 from outside.
[0267] The sense of distance control information includes flags "isDistanceRenderFlg" each for the number of objects num_objs, which indicate whether the object is subject to sense of distance control.
[0268] For example, if the value of the flag isDistanceRenderFlg of the i-th object is "1", the object is considered to be a target of the sense of distance control, and the sense of distance control process is performed on the audio data of the object.
[0269] When the value of the flag isDistanceRenderFlg of the i-th object is "1", the sense of distance control information includes the parameter configuration information DistanceRender_Attn(), two pieces of parameter configuration information DistanceRender_Filt(), and parameter configuration information DistanceRender_Revb() of that object.
[0270] Therefore, in this case, as described above, the distance control processing unit 67 performs distance control processing on the audio data of the targeted object, and the resulting audio data and metadata of the dry and wet components are output.
[0271] On the other hand, if the value of the flag isDistanceRenderFlg of the i-th object is "0", the object is not subject to distance control, i.e., is not a target, and distance control processing is not performed on the audio data of that object.
[0272] Therefore, for such objects, the audio data and metadata of the object are supplied as is from the distance control processing unit 67 to the 3D audio rendering processing unit 68.
[0273] When the value of the flag isDistanceRenderFlg of the i-th object is "0", the sense of distance control information does not include the parameter configuration information DistanceRender_Attn(), the parameter configuration information DistanceRender_Filt(), and the parameter configuration information DistanceRender_Revb() of that object.
[0274] 19, the parameter configuration information is encoded for each object in the sense of distance control information encoding unit 24. In other words, the sense of distance control information is encoded for each object. This makes it possible to realize sense of distance control for each object based on the intention of the content creator, and to play back content with a higher sense of realism.
[0275] In particular, in this example, by storing a flag isDistanceRenderFlg in the sense of distance control information, it is possible to set whether or not to perform sense of distance control for each object, and then perform different sense of distance control for each object.
[0276] For example, for a human voice object, different control rules can be set for that object than for other objects, or distance control can be avoided altogether, so that the sense of distance is not felt as much, meaning that sounds that are always easy for the listener to hear (easy-to-listen-to sounds) are reproduced.
[0277] Third Modification of the First Embodiment Other examples of distance control information Furthermore, it may be possible to set (specify) parameter control rules for each object group consisting of one or more objects, rather than for each object.
[0278] In such a case, the sense of distance control information has a structure shown in FIG.
[0279] In the example shown in FIG. 20, "num_obj_groups" indicates the number of object groups that make up the content, and for example, the number of object groups, num_obj_groups, is given to the sense of distance control information determination unit 23 from outside.
[0280] The sense of distance control information includes a flag "isDistanceRenderFlg" indicating whether or not an object group, or more specifically, an object belonging to an object group, is a target of sense of distance control, for the number of object groups, num_obj_groups.
[0281] For example, if the value of the flag isDistanceRenderFlg of the i-th object group is "1", that object group is considered to be subject to distance perception control, and distance perception control processing is performed on the audio data of the objects belonging to that object group.
[0282] When the value of the flag isDistanceRenderFlg of the i-th object group is "1", the sense of distance control information includes the parameter configuration information DistanceRender_Attn(), two pieces of parameter configuration information DistanceRender_Filt(), and parameter configuration information DistanceRender_Revb() for that object group.
[0283] Therefore, in this case, as described above, the distance control processing section 67 performs distance control processing on the audio data of the objects belonging to the targeted object group.
[0284] On the other hand, if the value of the flag isDistanceRenderFlg of the i-th object group is "0", then the object group is not considered to be subject to distance perception control, and distance perception control processing is not performed on the audio data of the objects in the object group.
[0285] Therefore, for objects in such an object group, the audio data and metadata of the object are supplied as is from the distance control processing unit 67 to the 3D audio rendering processing unit 68.
[0286] When the value of the flag isDistanceRenderFlg of the i-th object group is "0", the sense of distance control information does not include the parameter configuration information DistanceRender_Attn(), the parameter configuration information DistanceRender_Filt(), and the parameter configuration information DistanceRender_Revb() of that object group.
[0287] 20, the parameter configuration information is encoded for each object group in the distance control information encoding unit 24. In other words, the distance control information is encoded for each object group. This makes it possible to realize distance control for each object group based on the intention of the content creator, and to play back content with a higher sense of realism.
[0288] In particular, in this example, by storing a flag isDistanceRenderFlg in the sense of distance control information, it is possible to set whether or not to perform sense of distance control for each object group, and then perform different sense of distance control for each object group.
[0289] For example, when setting the same control rules for multiple percussion instruments that make up a drum set, such as a snare drum, bass drum, tom-toms, and cymbals, the content creator can group the objects of those multiple percussion instruments together into a single object group.
[0290] In this way, the same control rule can be set for each object corresponding to each of the multiple percussion instruments constituting a drum set that belong to the same object group. In other words, the same control rule information can be given to each of the multiple objects. Furthermore, as in the example shown in Fig. 20, by transmitting parameter configuration information for each object group, the amount of information such as parameters to be transmitted to the decoding side, i.e., the amount of information on sense of distance control, can be reduced.
[0291] Second embodiment <Configuration example of distance sense control processing unit> Also, an example has been described above in which the configuration of the sense of distance control processing unit 67 provided in the decoding device 51 is predetermined. That is, an example has been described in which one or more processes constituting the sense of distance control process, which are indicated by the configuration information of the sense of distance control information, and the order of these processes are predetermined.
[0292] However, the present invention is not limited to this and the configuration of the sense of distance control processing unit 67 may be freely changed depending on the configuration information of the sense of distance control information.
[0293] In such a case, the sense of distance control processing section 67 is configured as shown in FIG.
[0294] In the example shown in FIG. 21, the distance control processing unit 67 executes a program in response to the distance control information, and realizes some of the processing blocks among the signal processing units 201-1 to 201-3 and the reverb processing units 202-1 to 202-4.
[0295] The signal processing unit 201-1 performs signal processing on the audio data of the object supplied from the object decoding unit 62 based on the distance information supplied from the distance calculation unit 66 and the distance control information supplied from the distance control information decoding unit 64, and supplies the resulting audio data to the signal processing unit 201-2.
[0296] At this time, if the reverb processing unit 202-2 is functioning, that is, if the reverb processing unit 202-2 is realized, the signal processing unit 201-1 also supplies the audio data obtained by the signal processing to the reverb processing unit 202-2.
[0297] The signal processing unit 201-2 performs signal processing on the audio data supplied from the signal processing unit 201-1 based on the distance information supplied from the distance calculation unit 66 and the distance control information supplied from the distance control information decoding unit 64, and supplies the resulting audio data to the signal processing unit 201-3. At this time, if the reverb processing unit 202-3 is functioning, the signal processing unit 201-2 also supplies the audio data obtained by the signal processing to the reverb processing unit 202-3.
[0298] The signal processing unit 201-3 performs signal processing on the audio data supplied from the signal processing unit 201-2 based on the distance information supplied from the distance calculation unit 66 and the distance control information supplied from the distance control information decoding unit 64, and supplies the resulting audio data to the 3D audio rendering processing unit 68. At this time, if the reverb processing unit 202-4 is functioning, the signal processing unit 201-3 also supplies the audio data obtained by the signal processing to the reverb processing unit 202-4.
[0299] In the following description, when there is no need to particularly distinguish between the signal processing units 201-1 to 201-3, they will also be simply referred to as the signal processing unit 201.
[0300] The signal processing performed by signal processing unit 201-1, signal processing unit 201-2, and signal processing unit 201-3 is processing indicated by configuration information of the sense of distance control information.
[0301] Specifically, the signal processing performed by the signal processor 201 includes, for example, gain control processing, and filtering using a high-shelf filter, a low-shelf filter, or the like.
[0302] The reverb processing unit 202-1 generates wet component audio data by applying reverb processing to the audio data of the object supplied from the object decoding unit 62 based on the distance information supplied from the distance calculation unit 66 and the distance control information supplied from the distance control information decoding unit 64.
[0303] The reverb processor 202-1 also generates metadata including position information of the wet component based on the distance control information supplied from the distance control information decoder 64, the metadata supplied from the metadata decoder 63, and the listening position information supplied from the user interface 65. The reverb processor 202-1 also uses distance information as necessary to generate metadata of the wet component.
[0304] The reverb processing unit 202-1 supplies the wet component metadata and audio data generated in this manner to the 3D audio rendering processing unit 68.
[0305] The reverb processing unit 202-2 generates wet component metadata and audio data based on the distance information from the distance calculation unit 66, the distance control information from the distance control information decoding unit 64, the audio data from the signal processing unit 201-1, the metadata from the metadata decoding unit 63, and the listening position information from the user interface 65, and supplies them to the 3D audio rendering processing unit 68.
[0306] The reverb processing unit 202-3 generates wet component metadata and audio data based on the distance information from the distance calculation unit 66, the distance control information from the distance control information decoding unit 64, the audio data from the signal processing unit 201-2, the metadata from the metadata decoding unit 63, and the listening position information from the user interface 65, and supplies them to the 3D audio rendering processing unit 68.
[0307] The reverb processing unit 202-4 generates wet component metadata and audio data based on the distance information from the distance calculation unit 66, the distance control information from the distance control information decoding unit 64, the audio data from the signal processing unit 201-3, the metadata from the metadata decoding unit 63, and the listening position information from the user interface 65, and supplies them to the 3D audio rendering processing unit 68.
[0308] In the reverb processing units 202-2, 202-3, and 202-4, the same processing as in the reverb processing unit 202-1 is performed, and wet component metadata and audio data are generated.
[0309] In the following description, when there is no need to particularly distinguish between the reverb processing units 202-1 to 202-4, they will also be simply referred to as the reverb processing unit 202.
[0310] The distance control processing section 67 may be configured so that none of the reverb processing sections 202 function, or so that one or more reverb processing sections 202 function.
[0311] Therefore, for example, the distance control processing unit 67 may be configured to have a reverb processing unit 202 that generates wet components located to the left and right of the object (dry components), and a reverb processing unit 202 that generates wet components located above and below the object.
[0312] In this way, the content creator can freely specify each signal processing that constitutes the sense of distance control processing and the order in which these signal processing are performed, thereby realizing sense of distance control based on the intention of the content creator.
[0313] Other examples of distance control information Furthermore, when the configuration of the sense of distance control processing unit 67 can be freely changed (specified) as shown in FIG. 21, the sense of distance control information has a configuration shown in FIG. 22, for example.
[0314] In the example shown in FIG. 22, "num_objs" indicates the number of objects that make up the content, and the distance control information includes a flag "isDistanceRenderFlg" indicating whether or not the object is subject to distance control, for each of the objects, num_objs.
[0315] It should be noted that the number of objects num_objs and the flag isDistanceRenderFlg are the same as those in the example shown in FIG. 19, and therefore a description thereof will be omitted.
[0316] When the value of the flag isDistanceRenderFlg for the i-th object is “1”, the sense of distance control information includes ID information “proc_id” indicating the signal processing and parameter configuration information for each signal processing that constitutes the sense of distance control processing performed on that object.
[0317] That is, for example, according to the ID information "proc_id" indicating the jth (where 0≦j<4) signal processing, the distance control information includes parameter configuration information "DistanceRender_Attn()" for gain control processing, parameter configuration information "DistanceRender_Filt()" for filter processing, parameter configuration information "DistanceRender_Revb()" for reverb processing, or parameter configuration information "DistanceRender_UserDefine()" for user-defined processing.
[0318] Specifically, for example, when the ID information "proc_id" is "ATTN" indicating a gain control process, the sense of distance control information includes parameter configuration information "DistanceRender_Attn( )" for the gain control process.
[0319] Note that the parameter configuration information "DistanceRender_Attn()", "DistanceRender_Filt()", and "DistanceRender_Revb()" are the same as those in FIG. 11, and therefore the description thereof will be omitted.
[0320] The parameter configuration information "DistanceRender_UserDefine()" indicates parameter configuration information indicating the control rules of parameters used in user-defined processing, which is signal processing arbitrarily defined by the user.
[0321] Therefore, in this example, in addition to gain control processing, filter processing, and reverb processing, user-defined processing separately defined by the user can be added as signal processing that constitutes the distance perception control processing.
[0322] Although the sense of distance control process is configured by four signal processes here, the sense of distance control process may be configured by any number of signal processes.
[0323] In the distance control information shown in Figure 22, for example, if the 0th signal processing that constitutes the distance control processing is a gain control processing, the 1st signal processing is a filtering processing using a high shelf filter, the 2nd signal processing is a filtering processing using a low shelf filter, and the 3rd signal processing is a reverb processing, then a distance control processing unit 67 having the same configuration as that shown in Figure 3 will be realized.
[0324] In such a case, in the distance control processing unit 67 shown in FIG. 21, signal processing units 201-1 to 201-3 and reverb processing unit 202-4 are realized, but reverb processing units 202-1 to 202-3 are not realized (do not function).
[0325] 3. Signal processing units 201-1 to 201-3 and reverb processing unit 202-4 function as gain control unit 101, high shelf filter processing unit 102, low shelf filter processing unit 103, and reverb processing unit 104 shown in FIG.
[0326] In this way, even when the distance perception control information has the configuration shown in Figure 22, basically, the encoding process described with reference to Figure 15 is performed in the encoding device 11, and the decoding process described with reference to Figure 16 is performed in the decoding device 51.
[0327] However, in the encoding process, for example in step S13, it is determined for each object whether or not it is to be subject to distance control processing, the configuration of the distance control processing, etc., and in step S14, the distance control information with the configuration shown in Figure 22 is encoded.
[0328] On the other hand, in the decoding process, in step S47, the configuration of the sense of distance control processing unit 67 is determined for each object based on the sense of distance control information having the configuration shown in FIG. 22, and the sense of distance control process is performed appropriately.
[0329] As described above, according to the present technology, by transmitting distance control information to the decoding side together with the audio data of an object according to the settings of the content creator, etc., it is possible to realize distance control based on the intentions of the content creator in object-based audio.
[0330] Example of computer configuration The above-mentioned series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed in a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, for example, capable of executing various functions by installing various programs.
[0331] FIG. 23 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes by a program.
[0332] In the computer, a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, and a RAM (Random Access Memory) 503 are interconnected via a bus 504.
[0333] An input / output interface 505 is further connected to the bus 504. An input unit 506, an output unit 507, a recording unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.
[0334] The input unit 506 includes a keyboard, a mouse, a microphone, an image sensor, etc. The output unit 507 includes a display, a speaker, etc. The recording unit 508 includes a hard disk, a non-volatile memory, etc. The communication unit 509 includes a network interface, etc. The drive 510 drives a removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.
[0335] In a computer configured as described above, the CPU 501 loads, for example, a program recorded in the recording unit 508 into the RAM 503 via the input / output interface 505 and the bus 504, and executes the program, thereby performing the series of processes described above.
[0336] The program executed by the computer (CPU 501) can be provided by being recorded on a removable recording medium 511 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.
[0337] In the computer, the program can be installed in the recording unit 508 via the input / output interface 505 by mounting the removable recording medium 511 in the drive 510. The program can also be received by the communication unit 509 via a wired or wireless transmission medium and installed in the recording unit 508. Alternatively, the program can be installed in the ROM 502 or the recording unit 508 in advance.
[0338] In addition, the program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or it may be a program in which processing is performed in parallel or at the required timing, such as when called.
[0339] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.
[0340] For example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.
[0341] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.
[0342] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices.
[0343] Furthermore, the present technology can also be configured as follows.
[0344] (1) an object coding unit for coding audio data of an object; a metadata encoding unit that encodes metadata including position information of the object; a distance control information determination unit that determines distance control information for a distance control process performed on the audio data; a distance control information encoding unit that encodes the distance control information; a multiplexing unit that multiplexes the encoded audio data, the encoded metadata, and the encoded sense of distance control information to generate encoded data; An encoding device comprising: (2) The sense of distance control information includes control rule information for obtaining parameters used in the sense of distance control process. An encoding device as described in (1). (3) The parameter varies depending on the distance from the listening position to the object. An encoding device as described in (2). (4) The control rule information is an index indicating a function or a table for obtaining the parameter. An encoding device according to (2) or (3). (5) The sense of distance control information includes configuration information indicating one or more processes to be performed in combination to realize the sense of distance control process. An encoding device according to any one of (2) to (4). (6) The configuration information is information indicating the one or more processes and an order in which the one or more processes are to be performed. An encoding device as described in (5). (7) The processing is a gain control processing, a filter processing, or a reverb processing. An encoding device according to (5) or (6). (8) The sense of distance control information encoding unit encodes the sense of distance control information for each of the plurality of objects. An encoding device according to any one of (1) to (7). (9) The sense of distance control information encoding unit encodes the sense of distance control information for each object group consisting of one or more of the objects. An encoding device according to any one of (1) to (7). (10) The encoding device, Encode the audio data of the object, encoding metadata including location information of the object; determining distance control information for a distance control process to be performed on the audio data; encoding the sense of distance control information; The encoded audio data, the encoded metadata, and the encoded distance perception control information are multiplexed to generate encoded data. Encoding method. (11) Encode the audio data of the object, encoding metadata including location information of the object; determining distance control information for a distance control process to be performed on the audio data; encoding the sense of distance control information; The encoded audio data, the encoded metadata, and the encoded distance perception control information are multiplexed to generate encoded data. A program that causes a computer to execute a process including steps. (12) a demultiplexing unit for demultiplexing the encoded data and extracting encoded audio data of an object, encoded metadata including position information of the object, and encoded distance control information for a distance control process to be performed on the audio data; an object decoding unit that decodes the encoded audio data; a metadata decoding unit for decoding the encoded metadata; a distance control information decoding unit that decodes the encoded distance control information; a distance control processing unit that performs the distance control processing on the audio data of the object based on the distance control information; a rendering processing unit that performs a rendering process based on the audio data obtained by the distance control process and the metadata, and generates playback audio data for playing back the sound of the object; A decoding device comprising: (13) The sense of distance control processing unit performs the sense of distance control processing based on parameters obtained from control rule information included in the sense of distance control information and a listening position. A decoding device according to (12). (14) The parameter varies depending on the distance from the listening position to the object. A decoding device according to (13). (15) The distance control processing unit adjusts the parameters in accordance with a reproduction environment of the reproduced audio data. A decoding device according to (13) or (14). (16) The sense of distance control processing unit performs the sense of distance control processing by combining one or a plurality of processes indicated by the sense of distance control information based on the parameter. A decoding device according to any one of (13) to (15). (17) The processing is a gain control processing, a filter processing, or a reverb processing. A decoding device according to (16). (18) The distance control processing unit generates audio data of a wet component of the object through the distance control processing. A decoding device according to any one of (12) to (17). (19) The decoding device, demultiplexing the encoded data and extracting encoded audio data of the object, encoded metadata including position information of the object, and encoded distance control information for a distance control process performed on the audio data; Decoding the encoded audio data; Decoding the encoded metadata; Decoding the encoded sense of distance control information; performing the distance perception control process on the audio data of the object based on the distance perception control information; A rendering process is performed based on the audio data obtained by the distance control process and the metadata, and playback audio data for playing back the sound of the object is generated. Decryption method. (20) demultiplexing the encoded data and extracting encoded audio data of the object, encoded metadata including position information of the object, and encoded distance control information for a distance control process performed on the audio data; Decoding the encoded audio data; Decoding the encoded metadata; Decoding the encoded sense of distance control information; performing the distance perception control process on the audio data of the object based on the distance perception control information; A rendering process is performed based on the audio data obtained by the distance control process and the metadata, and playback audio data for playing back the sound of the object is generated. A program that causes a computer to execute a process including steps. [Explanation of symbols]
[0345] 11 encoding device, 21 object encoding unit, 22 metadata encoding unit, 23 distance control information determination unit, 24 distance control information encoding unit, 25 multiplexing unit, 51 decoding device, 61 demultiplexing unit, 62 object decoding unit, 63 metadata decoding unit, 64 distance control information decoding unit, 66 distance calculation unit, 67 distance control processing unit, 68 3D audio rendering processing unit, 101 gain control unit, 102 high shelf filter processing unit, 103 low shelf filter processing unit, 104 reverb processing unit
Claims
1. an object coding unit for coding audio data of an object; a metadata encoding unit that encodes metadata including position information of the object; a distance perception control information determination unit that determines distance perception control information for distance perception control processing performed on the audio data; a sense of distance control information encoding unit that encodes the sense of distance control information; a multiplexing unit that multiplexes the encoded audio data, the encoded metadata, and the encoded sense of distance control information to generate encoded data; An encoding device comprising:
2. The sense of distance control information includes control rule information for obtaining parameters used in the sense of distance control process. The encoding device according to claim 1 .
3. The parameter varies depending on the distance from the listening position to the object. The encoding device according to claim 2 .
4. The control rule information is an index indicating a function or a table for obtaining the parameter. The encoding device according to claim 2 .
5. The sense of distance control information includes configuration information indicating one or more processes to be performed in combination to realize the sense of distance control process. The encoding device according to claim 2 .
6. The configuration information is information indicating the one or more processes and the order in which the one or more processes are to be performed. The encoding device according to claim 5 .
7. The processing is a gain control processing, a filter processing, or a reverb processing. The encoding device according to claim 5 .
8. The sense of distance control information encoding unit encodes the sense of distance control information for each of the plurality of objects. The encoding device according to claim 1 .
9. The sense of distance control information encoding unit encodes the sense of distance control information for each object group consisting of one or more of the objects. The encoding device according to claim 1 .
10. The encoding device Encode the audio data of the object, encoding metadata including location information of the object; determining distance perception control information for a distance perception control process to be performed on the audio data; encoding the sense of distance control information; multiplexing the encoded audio data, the encoded metadata, and the encoded distance perception control information to generate encoded data; Encoding method.
11. Encode the audio data of the object, encoding metadata including location information of the object; determining distance perception control information for a distance perception control process to be performed on the audio data; encoding the sense of distance control information; multiplexing the encoded audio data, the encoded metadata, and the encoded distance perception control information to generate encoded data; A program that causes a computer to execute a process that includes steps.
12. a demultiplexing unit that demultiplexes the encoded data and extracts encoded audio data of objects, encoded metadata including position information of the objects, and encoded distance control information for distance control processing to be performed on the audio data; an object decoding unit that decodes the encoded audio data; a metadata decoding unit that decodes the encoded metadata; a distance control information decoding unit that decodes the encoded distance control information; a distance control processing unit that performs the distance control processing on the audio data of the object based on the distance control information; a rendering processing unit that performs rendering processing based on the audio data obtained by the distance control processing and the metadata, and generates playback audio data for playing back the sound of the object; A decoding device comprising:
13. The sense of distance control processing unit performs the sense of distance control processing based on control rule information included in the sense of distance control information and parameters obtained from the listening position. The decoding device according to claim 12.
14. The parameter varies depending on the distance from the listening position to the object. The decoding device according to claim 13.
15. The distance control processing unit adjusts the parameters in accordance with the playback environment of the playback audio data. The decoding device according to claim 13.
16. The sense of distance control processing unit performs the sense of distance control processing by combining one or more processes indicated by the sense of distance control information based on the parameters. The decoding device according to claim 13.
17. The processing is a gain control processing, a filter processing, or a reverb processing.
17. The decoding device according to claim 16.
18. The distance control processing unit generates audio data of a wet component of the object through the distance control processing. The decoding device according to claim 12.
19. The decoding device demultiplexing the encoded data to extract encoded audio data of the object, encoded metadata including position information of the object, and encoded distance control information for distance control processing performed on the audio data; Decoding the encoded audio data; Decoding the encoded metadata; Decoding the encoded sense of distance control information; performing the sense of distance control processing on the audio data of the object based on the sense of distance control information; Rendering is performed based on the audio data obtained by the distance control process and the metadata, and playback audio data for playing back the sound of the object is generated. Decryption method.
20. demultiplexing the encoded data to extract encoded audio data of the object, encoded metadata including position information of the object, and encoded distance control information for distance control processing performed on the audio data; Decoding the encoded audio data; Decoding the encoded metadata; Decoding the encoded sense of distance control information; performing the sense of distance control processing on the audio data of the object based on the sense of distance control information; Rendering is performed based on the audio data obtained by the distance control process and the metadata, and playback audio data for playing back the sound of the object is generated. A program that causes a computer to execute a process that includes steps.
Citation Information
Patent Citations
IEC23008-3
Information conversion apparatus and information conversion method, and communication apparatus and communication method
JP2006140595A
Acoustic system and acoustic characteristic control apparatus
JP2013021686A
Sound processing device and method, and program
WO2015107926A1
Sound processing device and method
WO2018047667A1
