Information processing device and method, and program
The information processing device converts spatial audio metadata by estimating sound localization based on speaker layout, addressing the issue of format incompatibility and maintaining sound localization between MPEG-H 3D Audio and AC-4 formats.
Patent Information
- Application Number
- PCT/JP2023/028857
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-01
- Filing Date
- 2023-08-08
- Publication Date
- 2024-11-07
AI Technical Summary
Existing spatial audio technologies face challenges in sharing audio data between different formats, such as MPEG-H 3D Audio and AC-4, as simple conversion of localization information does not maintain sound localization due to differences in rendering methods, leading to changes in sound image positioning.
An information processing device and method that acquires speaker layout information to convert object metadata from one spatial audio format to another, estimating sound localization and adjusting position data accordingly to maintain sound localization during playback.
Enables seamless data sharing between different spatial audio systems while preserving sound localization, allowing for conversion of Allocentric to Egocentric or vice versa, ensuring consistent audio experience across formats.
Smart Images

Figure JP2023028857_07112024_PF_FP_ABST
Abstract
Description
Information processing device, method, and program
[0001] The present technology relates to an information processing device, method, and program, and more particularly to an information processing device, method, and program that enable data sharing between different spatial audio formats.
[0002] In recent years, object-based audio technology has been attracting attention.
[0003] In object-based audio, audio data is composed of waveform data for an object and metadata that indicates the sound positioning information of the object based on that waveform data, and during playback, rendering is performed to the desired number of channels based on the metadata.
[0004] There are several methods for data formatting and rendering audio data for object-based audio.
[0005] One is a data format called MPEG (Moving Picture Experts Group)-H 3D Audio (see, for example, Non-Patent Document 1).
[0006] In MPEG-H 3D Audio, the positioning information is position information in the form of polar coordinates that indicates the direction of an object relative to the listening point (listening position), and rendering is performed based on this position information using a rendering method called VBAP (Vector Based Amplitude Panning) (see, for example, Non-Patent Document 2).
[0007] On the other hand, there is a data format called AC-4 (see, for example, Non-Patent Document 3).
[0008] In AC-4, the positioning information is in the form of absolute coordinates that indicate the position in xyz space, and rendering is performed by calculating the gain ratio of each speaker from the ratio of the distance between the object and the speaker in each of the x-axis, y-axis, and z-axis directions based on that position information.
[0009] These data formats, more specifically rendering methods, are called the egocentric method and the allocentric method, based on their respective characteristics.
[0010] ISO / IEC 23008-3 Information technology - High efficiency coding and media delivery in heterogeneous environments - Part 3: 3D audioVille Pulkki, “Virtual Sound Source Positioning Using Vector Base Amplitude Panning”, Journal of AES, vol.45, no.6, pp.456-466, 1997ETSI TS 103 448 V1.1.1(2016-09) “AC-4 Object Audio Renderer for Consumer Use”Recommendation ITU-R BS.2051-3(05 / 2022) “Advanced sound system for program production”
[0011] However, it is desirable to be able to share audio data consisting of waveform data and metadata between both formats.
[0012] Here, for example, consider converting Allocentric audio data into Egocentric audio data.
[0013] In this case, the x, y, and z coordinate values, which are the absolute coordinate-format positioning information contained in the metadata, can be converted simply based on the laws of physics into polar coordinate-format values of Azimuth (horizontal angle), Elevation (vertical angle), and Radius (radius) based on the listening position.
[0014] However, because the rendering methods used for the audio data before and after conversion are different, simply converting the localization information as described above will not result in the same sense of localization of the sound being reproduced.
[0015] A specific example will be described.
[0016] For example, in the Allocentric system, an object is located at the coordinates (x, y, z) = (-1, 1, 0), and the sound of the object is to be reproduced using a 7.1.4ch speaker layout.
[0017] In this case, if you use the AC-4 rendering method, i.e., Allocentric rendering, the sound of the object will be played from the "Left" speaker with a gain ratio of 1.0. In other words, the sound of the object will be output only from one speaker, "Left."
[0018] In the allocentric system, when the position of the "Left" speaker is "M+030" as defined in ITU-R BS (International Telecommunication Union-Radiocommunication Sector Broadcasting service) .2051 (see, for example, Non-Patent Document 4), the +30 degree position is used as the reference, and the allowable range is from +30 degrees to +45 degrees. Here, the direction directly ahead from the reference position is considered to be 0 degrees, and the counterclockwise direction is considered to be the positive direction.
[0019] Therefore, if the "Left" speaker is placed at a position of +30 degrees, the sound image of the object will be localized at a position of +30 degrees, but if the "Left" speaker is placed at a position of +45 degrees, the sound image of the object will be localized at a position of +45 degrees.
[0020] Here, the Allocentric waveform data and metadata in the above example are converted to Egocentric data and played back. Note that hereinafter, polar coordinates (Azimuth, Elevation, Radius) will also be referred to as (Az, El, Rad).
[0021] If the localization information (position information) is simply converted based on the laws of physics, the coordinates (x, y, z) = (-1, 1, 0) indicating the position of the object are converted to polar coordinates (Az, El, Rad) = (+45, 0, 1.414) using the function shown in the following equation (1).
[0022]
[0023] In the Egocentric method, when playback is performed using the same 7.1.4ch speaker layout as the Allocentric method, the sound image of the object rendered by VBAP based on the polar coordinates (+45, 0, 1.414), which are the positioning information obtained by conversion, will be localized at a position of +45 degrees, regardless of the actual speaker position.
[0024] Therefore, if the original content was produced using the Allocentric method in an environment where the "Left" speaker was positioned at +30 degrees, the positioning of the sound would change significantly.
[0025] In other words, if something is created to be localized at +30 degrees, and the localization information in absolute coordinate format is simply converted to localization information in polar coordinate format based on the laws of physics, the sound will end up being localized at +45 degrees, which will significantly change the localization position of the sound.
[0026] For the reasons described above, it has been difficult to share audio data while maintaining sound localization between data formats of different spatial audio systems, i.e., data formats with different rendering systems.
[0027] The present technology has been made in view of such circumstances, and makes it possible to share data between different spatial audio formats.
[0028] An information processing device according to one aspect of the present technology includes an acquisition unit that acquires speaker layout information when audio data in a first data format is played back, and a position data generation unit that converts, based on the speaker layout information, first position data of a sound source in the first data format or second position data of the sound source in a second data format that has a rendering method during playback different from that of the first data format, into the second position data or the first position data.
[0029] An information processing method or program according to one aspect of the present technology includes a step of acquiring speaker layout information during playback of audio data in a first data format, and converting, based on the speaker layout information, first position data of a sound source in the first data format or second position data of the sound source in a second data format that has a rendering method during playback different from that of the first data format, into the second position data or the first position data.
[0030] In one aspect of the present technology, speaker layout information during playback of audio data in a first data format is acquired, and based on the speaker layout information, first position data of a sound source in the first data format or second position data of the sound source in a second data format that has a rendering method during playback different from that of the first data format is converted into the second position data or the first position data.
[0031] FIG. 1 is a diagram illustrating an example of the configuration of a metadata conversion device. FIG. 2 is a diagram illustrating an example of the configuration of a sound image localization estimation unit. FIG. 3 is a flowchart illustrating metadata conversion processing. FIG. 4 is a diagram illustrating an example of speaker layout information. FIG. 5 is a diagram illustrating an absolute coordinate space. FIG. 6 is a diagram illustrating an example of calculated panning gain values. FIG. 7 is a diagram illustrating speaker vectors. FIG. 8 is a diagram illustrating a composite vector. FIG. 9 is a flowchart illustrating metadata conversion processing. FIG. 10 is a diagram illustrating an example of a display screen of an object metadata conversion application. FIG. 11 is a diagram illustrating an example of the configuration of a playback device. FIG. 12 is a diagram illustrating an example of the configuration of a metadata conversion device. FIG. 13 is a flowchart illustrating metadata conversion processing. FIG. 14 is a diagram illustrating an example of the configuration of a computer.
[0032] Hereinafter, embodiments to which the present technology is applied will be described with reference to the drawings.
[0033] <First embodiment> <Configuration example of metadata conversion device> The present technology uses speaker layout information to convert object metadata of a predetermined spatial audio method, i.e., a predetermined data format, into object metadata of another data format with a different rendering method, thereby enabling data sharing between different spatial audio methods.
[0034] For example, in this technology, the sound localization direction (localization position) of the sound source is estimated using a rendering method in the data format before conversion (spatial audio method). Then, based on the estimation result, position information for localizing the sound in the estimated direction using a rendering method in the data format after conversion is determined, and this position information is used as position data in the object metadata of the data format after conversion.
[0035] Furthermore, in this technology, for example, if the rendering method in the data format before conversion is the Allocentric method, the sound localization direction in the Allocentric method is estimated by combining a vector consisting of the direction of the speaker to which the gain is assigned and the magnitude of the gain value.
[0036] FIG. 1 is a diagram showing an example of the configuration of an embodiment of a metadata conversion device to which the present technology is applied.
[0037] The metadata conversion device 11 shown in Figure 1 converts object metadata in a data format that uses an allocentric rendering method, such as AC-4, into object metadata in a data format that uses an egocentric rendering method, such as MPEG-H 3D Audio.
[0038] Audio data in an Allocentric data format, i.e., a data format using an Allocentric rendering method, includes waveform data, which is an audio signal for reproducing the sound of one or more objects, and object metadata for each object. Note that the audio data may also include waveform data (audio signals) for each channel.
[0039] The object metadata includes at least the localization position of the sound of the object, that is, object position data that indicates the position of the object.
[0040] In particular, in Allocentric object metadata, object position data is data in the form of absolute coordinates that indicate the absolute position of an object in a virtual space. Specifically, the object position data is absolute coordinates (x, y, z) that indicate the position in absolute coordinate space (xyz space) where the object is placed.
[0041] The object metadata may include, as appropriate, name information indicating the name of the object (object name), type information indicating the type of object (e.g., vocals or guitar), size information indicating the size of the object, priority information indicating the priority of the object, spread information indicating the degree of spread of the object (spread information), and gain information indicating the gain value of the object. For example, each piece of object metadata may use a preset name or value, or the object metadata may be changeable by the user as appropriate. For example, the priority information may use a value from 0 to 7, with the highest priority object being set to priority 7 and the lowest priority object being set to priority 0. Furthermore, in real-time processing, the name and value of each piece of object metadata may be dynamically changed.
[0042] In contrast, audio data in an egocentric data format, i.e., a data format using egocentric rendering, includes waveform data for one or more objects and object metadata for each object. The audio data may also include waveform data for each channel.
[0043] As in the Allocentric method, object metadata in the Egocentric method includes at least object position data that indicates the position of the object.
[0044] However, in egocentric object metadata, object position data is in polar coordinate format, indicating the relative position of an object as seen from a listening position (listening point) that serves as a reference in virtual space. Specifically, the object position data is in polar coordinate format (Az, El, Rad) indicating the relative position of an object as seen from the listening position.
[0045] Furthermore, object metadata in the egocentric method also includes object name information, type information, size information, priority information, spread information, gain information, and the like, as appropriate.
[0046] As described above, the waveform data of objects in Allocentric audio data and Egocentric audio data are the same, but the rendering methods and coordinate formats of object position data are different.
[0047] The metadata conversion device 11 includes a speaker layout information input unit 21 , an object metadata input unit 22 , an object position data generation unit 23 , a meta information conversion unit 24 , and an object metadata output unit 25 .
[0048] The speaker layout information input unit 21 functions as an acquisition unit that acquires speaker layout information from outside the metadata conversion device 11 .
[0049] The speaker layout information acquired by the speaker layout information input unit 21 is information about the speaker system when playing back Allocentric audio data to be converted, that is, when playing back in the Allocentric format.
[0050] For example, the speaker layout information is information indicating the number of speakers that make up a speaker system used for playback in the Allocentric format, that is, the number of channels, the channel configuration, and the speaker placement positions.
[0051] The speaker layout information input unit 21 extracts speaker position information indicating the placement position of each speaker and speaker configuration information indicating the speaker configuration of the speaker system, such as the number of channels and channel configuration, from the acquired speaker layout information, and supplies it to the object position data generation unit 23.
[0052] The object metadata input unit 22 acquires Allocentric object metadata from outside the metadata conversion device 11 .
[0053] The object metadata input unit 22 extracts the object position data included in the acquired object metadata as object position information indicating the position of the object, and supplies it to the object position data generation unit 23 .
[0054] The object metadata input unit 22 also extracts data other than object position data (object position information) from the acquired object metadata as meta information, and supplies the extracted data to the meta information conversion unit 24 .
[0055] For example, the meta information includes name information, type information, size information, priority information, spread information, gain information, and the like.
[0056] The object position data generation unit 23 generates object position data indicating the position of an object during playback in the egocentric format based on the speaker position information, speaker configuration information, and object position information, and supplies the object position data to the object metadata output unit 25. In other words, the object position data generation unit 23 converts the object position data in the allocentric format into object position data in the egocentric format.
[0057] The object position data generated by the object position data generating unit 23 is stored in egocentric object metadata and is position data in polar coordinate format that indicates the relative position of an object with respect to the listening position.
[0058] The object position data generating unit 23 includes a sound image localization estimating unit 31 and an object position determining unit 32 .
[0059] The sound image localization estimation unit 31 generates estimated localization information based on the speaker position information and speaker configuration information supplied from the speaker layout information input unit 21 and the object position information supplied from the object metadata input unit 22, and supplies the generated estimated localization information to the object position determination unit 32 and the meta information conversion unit 24.
[0060] The estimated localization information is information that indicates the localization position of the sound of an object, i.e., the localization position of a sound image, when the sound of the object is played back based on Allocentric audio data in the speaker layout indicated by the speaker layout information. More specifically, the estimated localization information is information that indicates the estimated direction in which the sound of the object will be localized when the sound of the object is played back after rendering using the Allocentric method.
[0061] The object position determination unit 32 generates object position data in polar coordinate format based on the estimated localization information supplied from the sound image localization estimation unit 31 , and supplies the data to the object metadata output unit 25 .
[0062] The object position data in polar coordinate format is egocentric object position data when the position indicated by the estimated localization information is the object position.
[0063] The meta information conversion unit 24 appropriately converts the meta information supplied from the object metadata input unit 22 based on the estimated localization information supplied from the sound image localization estimation unit 31, and supplies the metadata obtained by the conversion to the object metadata output unit 25.
[0064] The meta-information conversion unit 24 converts the allocentric meta-information into egocentric meta-information, and outputs the converted meta-information as metadata.
[0065] The object metadata output unit 25 outputs data consisting of the object position data in polar coordinate format supplied from the object position determination unit 32 and the metadata supplied from the meta information conversion unit 24 as egocentric object metadata.
[0066] <Configuration Example of Sound Image Localization Estimation Unit> The sound image localization estimation unit 31 is configured as shown in FIG. 2, for example.
[0067] The sound image localization estimation unit 31 includes a panning gain calculation unit 61 , a speaker vector generation unit 62 , and a composite vector calculation unit 63 .
[0068] The panning gain calculation unit 61 calculates a panning gain value based on the speaker configuration information supplied from the speaker layout information input unit 21 and the object position information supplied from the object metadata input unit 22, and supplies it to the speaker vector generation unit 62.
[0069] The panning gain calculation unit 61 calculates, as a panning gain value, a gain value to be assigned to each speaker when reproducing the sound of an object based on Allocentric audio data in the speaker layout indicated by the speaker configuration information.
[0070] For example, during playback using the Allocentric method, a panning gain value is calculated for each speaker, i.e., for each channel, based on the Allocentric rendering method, and the gain is adjusted by multiplying the waveform data of the object by the panning gain value. The gain-adjusted waveform data is then supplied to the speaker corresponding to the channel, and sound based on the waveform data is output from the speaker.
[0071] The panning gain calculation unit 61 outputs the panning gain value of each speaker calculated based on the allocentric rendering method as a gain value for generating a speaker vector.
[0072] The panning gain value is a gain value that is assigned to each speaker in the Allocentric system when the sound of an object is reproduced based on audio data in the Allocentric system.
[0073] The speaker vector generation unit 62 generates a vector (hereinafter also referred to as a speaker vector) for each speaker based on the speaker position information supplied from the speaker layout information input unit 21 and the panning gain value supplied from the panning gain calculation unit 61.
[0074] For example, the speaker vector generation unit 62 generates a speaker vector whose orientation is the direction indicating the position of the speaker indicated by the speaker position information and whose magnitude is the panning gain value. That is, the speaker vector is a vector that points in the direction of the position of the speaker indicated by the speaker position information and has the magnitude of the panning gain value.
[0075] The speaker vector generating unit 62 supplies information consisting of the speaker vectors of each speaker to the composite vector calculating unit 63 as speaker vector information.
[0076] The composite vector calculation unit 63 combines the speaker vectors of each of the multiple speakers based on the speaker vector information supplied from the speaker vector generation unit 62, and supplies the resulting composite vector to the object position determination unit 32 and the meta information conversion unit 24 as estimated positioning information.
[0077] <Explanation of Metadata Conversion Processing> Next, the operation of the metadata conversion device 11 will be described.
[0078] 3, the metadata conversion process performed by the metadata conversion device 11 will be described below. This metadata conversion process is started when, for example, the user designates Allocentric audio data to be converted.
[0079] In step S11, the speaker layout information input unit 21 sets the speaker positions of the allocentric system in response to user operations or the like.
[0080] For example, when converting audio data, i.e., object metadata, the user specifies the speaker configuration, such as the channel configuration and number of channels, of the speaker system when playing back audio in the Allocentric format, as well as the placement position of each speaker, etc. In other words, information about the speaker system used for playback in the Allocentric format is specified.
[0081] Then, the speaker layout information input unit 21 sets the speaker placement positions and the like in accordance with the user's designation operation, and obtains speaker layout information indicating the setting results.
[0082] The speaker layout information input unit 21 extracts speaker position information and speaker configuration information from the acquired speaker layout information, and supplies the speaker position information to a speaker vector generation unit 62 and the speaker configuration information to a panning gain calculation unit 61 .
[0083] Ideally, the speaker configuration and placement positions indicated by the speaker layout information should be the same as the speaker configuration and placement positions when the content creator decided on the object position data included in the Allocentric object metadata.
[0084] Therefore, if the speaker configuration etc. can be specified when the object position data is determined, then that speaker configuration etc. can be set (designated). On the other hand, if the speaker configuration etc. cannot be specified when the object position data is determined, then for example, a standard speaker configuration etc. for playback in the Allocentric format can be set.
[0085] As a specific example, when the speaker layout of a speaker system used for playback in the Allocentric format is 7.1.4ch, the information shown in FIG. 4 is acquired as speaker layout information.
[0086] In Figure 4, the "3 / 4 / 4" entered in the "Loudspeaker configuration name" column indicates the speaker configuration. In particular, "3 / 4 / 4" indicates a speaker configuration with three speakers placed in the front, a total of four speakers placed in the surround (side and back), and four speakers placed on the upper layer.
[0087] "Loudspeaker name" indicates the name of each speaker, "Index" indicates an index that uniquely identifies each speaker, and "Speaker position" is speaker position information that indicates the absolute position of each speaker in xyz space.
[0088] In this example, the speaker position information is written in absolute coordinate format, i.e., the speaker position information is expressed as absolute coordinates (x, y, z) that indicate the position of the speaker in absolute coordinate space (xyz space).
[0089] The speaker layout information input unit 21 extracts "Speaker position" as speaker position information and "Loudspeaker configuration name" as speaker configuration information. Note that "Loudspeaker name" and "Index" may also be extracted as information constituting the speaker configuration information. The speaker configuration information is information required for rendering in the Allocentric method.
[0090] Returning to the explanation of the flowchart in FIG. 3, in step S12, the object metadata input unit 22 acquires Allocentric object metadata from the audio data specified by the user.
[0091] For example, in the metadata conversion device 11, for content played back using audio data, each object is selected in turn as an object to be processed. In addition, in the content, object metadata is prepared for each predetermined period, such as one or more frames, and the metadata conversion device 11 selects each period in turn as a period to be processed.
[0092] In step S12, the object metadata input unit 22 acquires object metadata for the processing period of the object to be processed.
[0093] In step S13 , the object metadata input unit 22 extracts object position information from the acquired Allocentric type object metadata and supplies it to the panning gain calculation unit 61 .
[0094] As described above, object metadata in the Allocentric system includes object position data in the absolute coordinate format, and in step S13, the object position data is extracted as object position information.
[0095] For example, assume there is an absolute coordinate space (xyz space) with the listening position as the origin O, as shown in Figure 5. In this example, in the absolute coordinate space, the right direction in the drawing is the positive direction of the x-axis, the depth direction in the drawing is the positive direction of the y-axis, and the upward direction in the drawing is the positive direction of the z-axis.
[0096] In such an absolute coordinate space, for example, if one object is placed at position P11, the absolute coordinates (x, y, z) = (0.8, 1, 0.5) indicating the absolute position of position P11 in the absolute coordinate space are taken as the object position data of the object placed at position P11.
[0097] The object metadata input unit 22 also extracts data other than object position data included in the Allocentric type object metadata as meta information, and supplies the extracted data to the meta information conversion unit 24 .
[0098] Here, for example, name information, type information, size information, priority information, spread information, gain information, etc. of the object are extracted as meta information.
[0099] Returning to the explanation of Figure 3, in step S14, the panning gain calculation unit 61 calculates a panning gain value based on the speaker configuration information supplied from the speaker layout information input unit 21 and the object position information supplied from the object metadata input unit 22.
[0100] For example, the panning gain calculation unit 61 calculates a panning gain value for each of multiple speakers indicated by the speaker configuration information for the object to be processed, based on the rendering method defined in AC-4 (Allocentric method) (see, for example, the above-mentioned Non-Patent Document 3).
[0101] In this case, the gain value for each speaker is calculated by calculating the ratio of the distance between the object and each of the multiple speakers in the x-axis direction, y-axis direction, and z-axis direction. The product of the gain values for each speaker is then used as the panning gain value for that speaker.
[0102] The panning gain calculation unit 61 supplies the calculated panning gain values for each of the plurality of speakers to the speaker vector generation unit 62 .
[0103] As a specific example, suppose that the object position information, i.e., the absolute coordinates (x, y, z) of the position of the object to be processed, is (0.8, 1, 0.5) as shown in FIG. 5, and the speaker configuration indicated by the speaker configuration information is the configuration (7.1.4ch) shown in FIG. 4.
[0104] In such a case, the panning gain values of each speaker are calculated as shown in Fig. 6. In the example of Fig. 6, the panning gain values of the speakers "Right," "Center," and "Top front right" that are near the object to be processed are non-zero, and the panning gain values of the other speakers are zero.
[0105] In step S15 , the speaker vector generating unit 62 generates speaker vector information based on the speaker position information supplied from the speaker layout information input unit 21 and the panning gain value supplied from the panning gain calculating unit 61 .
[0106] For example, when the panning gain values shown in Fig. 6 are obtained, the speaker vector generation unit 62 generates the speaker vectors shown in Fig. 7. Note that in Fig. 7, parts corresponding to those in Fig. 5 are given the same reference numerals, and their explanation will be omitted.
[0107] 7, speakers SP11, SP12, and SP13, which are arranged in the absolute coordinate space (xyz space), represent the "Center" speaker, the "Top front right" speaker, and the "Right" speaker, respectively. Absolute coordinates (x, y, z), which are speaker position information, are written near these speakers.
[0108] For example, if we focus on speaker SP11 (Centre), the speaker vector C of speaker SP11 can be calculated from the speaker position information shown in Figure 4, i.e., the absolute coordinates (x, y, z) = (0, 1, 0) of the position of speaker SP11, and the panning gain value of speaker SP11, "0.219", shown in Figure 6.
[0109] Specifically, a vector represented by an arrow V11 whose direction is from the origin O toward the speaker SP11 and whose magnitude is the panning gain value "0.219" is calculated as the speaker vector C of the speaker "Centre".
[0110] Similarly, the vector represented by the arrow V12, which points from the origin O toward the speaker SP12 (absolute coordinates (0.5, 0.5, 0.707107)) and has a magnitude of the panning gain value "0.707", is calculated as the speaker vector Tfr of the speaker "Top front right".
[0111] Furthermore, the vector represented by the arrow V13, which points from the origin O toward the speaker SP13 (absolute coordinates (0.5, 0.866025, 0)) and has a magnitude of the panning gain value "0.672", is calculated as the speaker vector R of the "Right" speaker.
[0112] These speaker vectors C, Tfr, and R can be obtained by calculating the following equation (2). That is, the speaker vector generation unit 62 obtains the speaker vectors by calculating equation (2) based on the speaker position information and the panning gain value.
[0113]
[0114] In step S15, speaker vectors are calculated for all speakers that make up the speaker system having the speaker configuration indicated by the speaker layout information. In particular, in the example shown in Fig. 7, speaker vectors for speakers other than the "Right," "Centre," and "Top front right" speakers are set to zero vectors.
[0115] The speaker vector generating unit 62 supplies information consisting of the speaker vectors of each speaker to the composite vector calculating unit 63 as speaker vector information.
[0116] Returning to the explanation of the flowchart in FIG. 3, in step S16 , the composite vector calculation unit 63 calculates estimated localization information based on the speaker vector information supplied from the speaker vector generation unit 62 .
[0117] Specifically, the composite vector calculation unit 63 generates a composite vector by combining the speaker vectors for all speakers indicated by the speaker vector information, and uses the composite vector, i.e., the coordinates of the position indicated by the composite vector, as estimated localization information.
[0118] For example, when the speaker vectors shown in Fig. 7 are obtained, the non-zero speaker vectors C, Tfr, and R are combined to generate (calculate) a combined vector CV as shown in Fig. 8. Note that parts in Fig. 8 that correspond to those in Fig. 7 are given the same reference numerals, and their explanation will be omitted where appropriate.
[0119] In the example of FIG. 8, the speaker vector C, the speaker vector Tfr, and the speaker vector R are combined to obtain a combined vector CV represented by an arrow V21.
[0120] The position indicated by this resultant vector CV is the position indicated by the absolute coordinates (x, y, z) = (0.690, 1.155, 0.500), and the resultant vector CV can be obtained by the following equation (3): In other words, the resultant vector calculation unit 63 calculates the resultant vector CV by calculating equation (3) based on the speaker vector information.
[0121]
[0122] The position indicated by the resultant vector CV is the position where the sound image of the object to be processed will be localized when the sound of the object to be processed is played back on a speaker system with the speaker layout indicated by the speaker layout information, based on the Allocentric audio data to be converted. More specifically, the sound of the object to be processed is localized in the direction of the position indicated by the resultant vector CV when viewed from the listening position (origin O).
[0123] Therefore, calculating the resultant vector CV can be said to be estimating the localization position of the sound of the object during playback in the allocentric format, more specifically, the direction in which the sound of the object is localized. In other words, the resultant vector CV can be said to be the result of estimating the localization position of the sound of the object.
[0124] The composite vector calculation unit 63 supplies the composite vector obtained by vector composition to the object position determination unit 32 and the meta information conversion unit 24 as estimated localization information.
[0125] Returning to the explanation of the flowchart in FIG. 3, in step S17 , the object position determination unit 32 converts the estimated position information supplied from the resultant vector calculation unit 63 into object position data in polar coordinate format and supplies it to the object metadata output unit 25 .
[0126] The object position determination unit 32 calculates egocentric object position data from the estimated position information by converting the estimated position information, i.e., the position (direction) indicated by the composite vector, into a position (direction) in polar coordinate format based on the listening position.
[0127] For example, if the absolute coordinates (x, y, z) = (0.690, 1.155, 0.500) shown in equation (3) are obtained as estimated localization information (synthetic vector), egocentric object position data can be obtained by calculating the following equation (4).
[0128]
[0129] In equation (4), the absolute coordinates (x, y, z) = (0.690, 1.155, 0.500) are converted into the listening position, that is, the polar coordinates (Az, El, Rad) = (-30.86, 20.39, 1.43) based on the origin O of the absolute coordinate space (xyz space) by coordinate transformation based on the laws of physics. These polar coordinates are then used as the object position data for the egocentric method.
[0130] If the absolute coordinates (x, y, z) of an object's position information are (0.8, 1, 0.5), when these absolute coordinates (x, yz) = (0.8, 1, 0.5) are converted to polar coordinates based on simple physical laws, the polar coordinates become (Az, El, Rad) = (-38.66, 21.33, 1.37), which results in a shift in the localization position.
[0131] In response to this, the metadata conversion device 11 estimates the localization position and calculates object position data in polar coordinate format based on the estimation result.
[0132] Therefore, the polar coordinates (Az, El, Rad) = (-30.86, 20.39, 1.43) obtained as the localization position are different from the above polar coordinates (Az, El, Rad) = (-38.66, 21.33, 1.37), and it can be seen that the sound localization is preserved. In other words, it is possible to convert allocentric object position data into egocentric object position data while preserving sound localization.
[0133] In step S18, the meta information conversion unit 24 converts the meta information supplied from the object metadata input unit 22 based on the estimated localization information supplied from the composite vector calculation unit 63. The estimated localization information is converted into egocentric object position data by simple coordinate conversion in the object position determination unit 32. Therefore, converting the meta information based on the estimated localization information can also be said to be converting the meta information based on egocentric object position data. Note that the egocentric object position data may be supplied from the object position determination unit 32 to the meta information conversion unit 24 and used to convert the meta information. Also, the processing of step S18 does not necessarily have to be executed.
[0134] For example, the meta information conversion unit 24 calculates the following equation (5) based on the radius "Rad" constituting the polar coordinates as egocentric object position data obtained from the estimated localization information. As a result, the gain information Gain as allocentric meta information supplied from the object metadata input unit 22 is converted into egocentric gain information Gain'.
[0135]
[0136] When estimating the localization position using the allocentric method in steps S14 to S16, a composite vector obtained by combining speaker vectors is used as estimated localization information indicating the localization position of the sound of the object to be processed.
[0137] Therefore, the larger the panning gain value of an object, the longer the distance from the listening position (origin O) to the localization position indicated by the estimated localization information. In other words, the radius Rad of the egocentric object position data becomes. In real space, the longer the distance from the listening position to the localization position, and the farther the object is, the quieter the sound of that object should be heard.
[0138] Therefore, in the meta information conversion unit 24, the gain information is corrected (modified) according to the radius Rad, i.e., the distance from the listening position to the position where the sound of the object is localized, so that the sound of the object can be heard at a more appropriate volume, and the corrected gain information is made egocentric gain information.
[0139] The meta information conversion unit 24 converts the meta information obtained by appropriately converting the Allocentric meta information into meta information to be stored in Egocentric object metadata, i.e., Egocentric metadata, and supplies the metadata to the object metadata output unit 25.
[0140] In step S19, the object metadata output unit 25 generates egocentric object metadata and outputs it to the subsequent stage.
[0141] For example, the object metadata output unit 25 outputs data consisting of object position data in polar coordinate format supplied from the object position determination unit 32 and metadata supplied from the meta information conversion unit 24 as egocentric object metadata.
[0142] Through the above process, the Allocentric object metadata for the object to be processed during the period to be processed is converted into Egocentric object metadata.
[0143] In step S20, the metadata conversion device 11 determines whether or not all object metadata in the time series of the object to be processed has been processed.
[0144] For example, in step S20, for the object to be processed, all of the multiple periods, such as all frames of the content, are set as the processing periods, and if the Allocentric object metadata for those periods is converted to Egocentric object metadata, it is determined that processing has been performed on all of the object metadata.
[0145] If it is determined in step S20 that not all object metadata have been processed, the process returns to step S12 and the above-described process is repeated, i.e., a new period is processed and the object metadata is converted.
[0146] On the other hand, if it is determined in step S20 that all object metadata has been processed, the process proceeds to step S21.
[0147] In step S21, the metadata conversion device 11 determines whether or not all objects have been processed.
[0148] For example, in step S21, if all objects are processed and the Allocentric object metadata for all periods of all those objects is converted to Egocentric object metadata, it is determined that processing has been performed for all objects.
[0149] If it is determined in step S21 that the processing has not yet been performed for all objects, the processing then returns to step S12, and the above-described processing is repeated, i.e., a new object is selected as the processing target and the object metadata is converted.
[0150] On the other hand, if it is determined in step S21 that the processing has been performed for all objects, the metadata conversion processing ends.
[0151] In this way, the metadata conversion device 11 estimates the localization positions of sounds during playback in the Allocentric format based on the speaker layout information in the Allocentric format.The metadata conversion device 11 then generates egocentric object metadata based on the localization position estimation results.
[0152] In this way, allocentric object metadata can be converted to egocentric object metadata while preserving sound localization, which makes it possible to share audio data (data sharing) between different spatial audio formats, i.e., different data formats, while preserving sound localization.
[0153] In the above description, the audio data contains waveform data of objects as waveform data for each of a plurality of sound sources, and the object metadata of the objects is converted. That is, the example in which the sound sources are objects has been described. However, the audio data may also contain waveform data for each channel (channel-based waveform data) as waveform data for the sound sources.
[0154] In such a case, the sound based on the waveform data of a channel is output (played) from the speaker corresponding to that channel, and the speaker corresponding to the channel becomes the sound source. In other words, the speaker corresponding to the channel can be considered as an object.
[0155] Therefore, if speaker position information indicating the position of the speaker corresponding to the channel is used as object position information indicating the position of the object, it is possible to obtain egocentric object metadata for the waveform data of the channel.
[0156] In other words, the waveform data of a channel can be treated as waveform data of one object (sound source), and object position data of the waveform data of that channel can be generated.
[0157] For example, when channel waveform data is included in Allocentric audio data, the speaker layout information input unit 21 supplies the panning gain calculation unit 61 with speaker position information of the speaker corresponding to the channel extracted from the speaker layout information.
[0158] The panning gain calculation unit 61 uses the speaker position information supplied from the speaker layout information input unit 21 as object position information, and calculates a panning gain value for the speaker corresponding to the channel based on the object position information and the speaker configuration information.
[0159] Furthermore, in the speaker vector generation unit 62, composite vector calculation unit 63, and object position determination unit 32 subsequent to the panning gain calculation unit 61, object position data is generated in the same manner as in the case of the object position data of the object.
[0160] As a result, the object position data generating unit 23 converts the speaker position information of the speaker corresponding to the channel into object position data in the allocentric system, and then converts the object position data into object position data in the egocentric system.
[0161] The metadata conversion process described with reference to FIG. 3 is an example of a case where all object metadata in time series is available and object metadata can be converted offline.
[0162] On the other hand, there may be cases where object metadata for each period, such as frames, is supplied in order in time series, and the object metadata must be converted in order in real time.
[0163] In such a case, the process of converting the object metadata of all objects in one period such as a frame is repeated for each period. That is, in the metadata conversion process shown in Fig. 3, the process of step S19 is followed by the process of step S21, and the process of step S21 is followed by the process of step S20.
[0164] <First Modification of the First Embodiment> <Explanation of Metadata Conversion Process> In the above, an example has been described in which the localization positions of the sounds (sound images) of all objects are estimated and object position data is converted.
[0165] However, the present invention is not limited to this, and for some objects, the object position data may be converted by coordinate transformation based on simple physical laws without estimating the localized position.
[0166] In such a case, the metadata conversion device 11 performs, for example, the metadata conversion process shown in Fig. 9. The metadata conversion process performed by the metadata conversion device 11 will be described below with reference to the flowchart of Fig. 9.
[0167] The processes in steps S51 to S53 are the same as those in steps S11 to S13 in FIG. 3, and therefore will not be described further.
[0168] In step S54, the object position data generating unit 23 determines whether or not the object to be processed is a target for estimating the sound image localization position (whether or not to set it as a target for estimation).
[0169] For example, the object position data generating unit 23 makes the determination based on a selection operation by the user, meta information extracted from object metadata in the Allocentric system, flag information separately supplied together with the object metadata, and the like.
[0170] As an example, when a signal corresponding to a user's selection operation is supplied to the object position data generation unit 23, the object position data generation unit 23 determines, based on the supplied signal, whether or not the object to be processed has been selected by the user as a target for estimating a sound image localization position. Note that, instead of the user's selection operation, flag information indicating whether or not the object is to be a target for estimating a sound image localization position, which is set in advance by a content creator or a user (listener) and attached to the waveform data or object metadata of the object, may be used.
[0171] Furthermore, for example, when the meta information includes name information of an object to be processed, if the name of the object indicated by the name information indicates a predetermined specific type of object, such as an object of a reverb component, the object may be determined not to be a target for estimation of the sound image localization position. Here, the object of a reverb component is, for example, an object of an early reflection sound or a late reverberation sound.
[0172] For example, for objects where the sense of positioning of the sound image is not important, such as reverb components, it is possible to reduce processing time and processing load by generating estimated positioning information using coordinate transformation based on simple physical laws.
[0173] Similarly, for example, an object with a low priority indicated by priority information, i.e., an object with a priority below a predetermined threshold, can also be considered an object for which the sense of localization of a sound image is not important. Therefore, when priority information is included as meta information, whether or not an object is a target for estimating a sound image localization position may be determined based on the priority information. Here, the priority of each object may be constant throughout the entire content or may be dynamically changed for each processing period. When the priority is constant throughout the content, for objects with a priority below a predetermined threshold, i.e., objects for which the sense of localization of a sound image is not important, the generation of estimated localization information can be performed using coordinate transformation based on simple physical laws, thereby reducing processing time and processing load. Furthermore, when the priority is dynamically changed for each processing period, the sound image localization position is estimated for objects with a high priority for each processing period. This is particularly effective for content such as free viewpoint audio, in which the distance between the user and objects changes as the user moves freely within a space. In other words, it is possible to realize content and a user experience that better reflects the intentions of the creator (producer) in content such as free viewpoint audio. Furthermore, object metadata other than priority may be dynamically changed for each period being processed.
[0174] If it is determined in step S54 that the object is not a target for estimating the sound image localization position, in step S55 the object position data generation unit 23 converts the object position information into object position data in polar coordinate format by coordinate transformation based on simple physical laws.
[0175] Specifically, the object position data generation unit 23 calculates (generates) object position data in polar coordinate format by calculating the following equation (6) based on the object position information supplied from the object metadata input unit 22, i.e., the object position data in the Allocentric format.
[0176]
[0177] Equation (6) converts the allocentric method, i.e., absolute coordinates (x, y, z), which are object position information in absolute coordinate format, into egocentric method, i.e., polar coordinates (Az, El, Rad), which are object position data in polar coordinate format based on the origin O of the absolute coordinate space (xyz space).
[0178] The object position data generating unit 23 supplies the object position data in polar coordinate format obtained by the calculation of equation (6) to the object metadata output unit 25, and then the process proceeds to step S61.
[0179] On the other hand, if it is determined in step S54 that the object to be processed is the object for which the sound image localization position is to be estimated, then steps S56 to S60 are performed, and the process proceeds to step S61.
[0180] The processing of steps S56 to S60 is the same as the processing of steps S14 to S18 in Fig. 3, and therefore a description thereof will be omitted. Also, for objects that were not subject to estimation of the sound image localization position, meta information may be converted based on the object position data calculated in step S55, for example, in the same manner as the processing of step S18 in Fig. 3.
[0181] After the processing of step S55 or step S60 is performed, the processing of steps S61 to S63 is performed, and the metadata conversion processing ends. The processing of steps S61 to S63 is the same as the processing of steps S19 to S21 in Figure 3, so a description thereof will be omitted.
[0182] In this way, the metadata conversion device 11 determines whether or not an object is a target for estimating the sound image localization position, and generates object position data in polar coordinate format using a method according to the determination result, thereby reducing the processing time and processing load.
[0183] <Application Example> FIG. 10 shows an example of a display screen of an application having the function of converting object metadata, ie, converting audio data, as described above.
[0184] For example, an information processing device such as a computer that realizes the metadata conversion device 11 displays the display screen shown in FIG. 10 by executing a program for realizing an application.
[0185] In this example, the display screen is provided with an input specification field W11, which is an interface for specifying the file name of an input file, and an output specification field W12, which is an interface for specifying the file name of an output file.
[0186] For example, the user operates the input specification field W11 to specify the input file, that is, the audio data to be converted.
[0187] As an example, the input file here is in ADM (Audio Definition Model) BWF (Broadcast Wave Format) file format, but it may also be in a format that specifies a file containing PCM (Pulse Code Modulation) data and object metadata, or a group of files such as a folder.
[0188] The ADM BWF file referred to here is a BWF file that consists of chunks that store metadata in accordance with the ADM standard and PCM data.
[0189] For example, the ADM standard is described in detail in "Recommendation ITU-R BS.2076-2 (10 / 2019) 'Audio definition model'." For example, the BWF standard is described in detail in "EBU Tech 3285 - Specification of the Broadcast Wave Format (BWF) version 2.0."
[0190] Furthermore, the user operates the output specification field W12 to specify the file name of the output file, that is, the audio data obtained by the conversion.
[0191] As an example, the output file here is in ADM BWF file format, but as with the input file, it may also be in a format that specifies a file containing PCM data and object metadata, or a group of files such as a folder.
[0192] The display screen also has an execute button W13 that is operated when converting input audio data (input file) into an output file, and a setting area W14 for setting the speaker layout to be used in Allocentric rendering playback.
[0193] The setting area W14 is provided with a pull-down menu W15 for selecting the number of channels and channel configuration of the speaker layout, and the user specifies the above-mentioned speaker configuration information by operating the pull-down menu W15.
[0194] For example, the user operates the pull-down menu W15 to switch the channel configuration of the speaker system between 5.1.2ch, 7.1.4ch, 9.1.6ch, and so on.
[0195] The setting area W14 also includes text boxes such as text box W16 for setting the placement position of each speaker in the specified speaker layout, more specifically, the direction in which each speaker is installed as seen from the listening position.
[0196] Specifically, for example, the horizontal angle (Azimuth) of the speaker is entered in the column (text box) on the left side of text box W16, and the vertical angle (Elevation) of the speaker is entered in the column (text box) on the right side of text box W16.
[0197] In this case, the speaker layout information input unit 21 converts the speaker positions represented by the input horizontal and vertical angles into speaker position information in absolute coordinate format as appropriate.
[0198] Note that, although an example is shown here in which the speaker placement position is input as an angle (horizontal angle and vertical angle) as seen from the listening position, the speaker placement position may also be input as coordinates in absolute coordinate format (absolute coordinates).
[0199] After operations are performed in the input specification field W11 and the output specification field W12 to specify the input file and the output file, and operations are performed in the setting area W14 to set the speaker layout, when the execute button W13 is operated (pressed), the audio data conversion is performed.
[0200] That is, the Allocentric audio data specified as the input file is converted into Egocentric audio data as the output file.
[0201] At this time, the metadata conversion device 11 performs the metadata conversion process described with reference to FIGS. 3 and 9, and generates egocentric object metadata.
[0202] Then, a file containing object metadata of each object in the egocentric format and PCM data as waveform data of each object in the egocentric format is generated as an output file.
[0203] Although an independent application is shown as an example here, this conversion function may also be implemented within an audio editing application called a DAW (Digital Audio Workstation). That is, for example, an allocentric edit on a DAW may be converted to an egocentric edit by providing speaker layout information and converting object position information and gain information using this technology. This conversion function may also be implemented in a DAW or the like by installing a plug-in incorporating this technology.
[0204] <Configuration Example of Playback Device> Furthermore, by incorporating the metadata conversion device 11 to which the present technology is applied into a playback device, it is also possible to play back content on an egocentric device using allocentric audio data as input.
[0205] Such a playback device is configured, for example, as shown in Figure 11. In Figure 11, parts corresponding to those in Figure 1 are given the same reference numerals, and their explanation will be omitted where appropriate.
[0206] The playback device 101 shown in FIG. 11 includes a metadata separation unit 111 , a metadata conversion device 11 , and an egocentric audio playback device 112 .
[0207] Allocentric audio data to be played back is input (supplied) to the metadata separation unit 111, which separates the supplied audio data into object metadata and waveform data. That is, the object metadata and waveform data of each object are extracted from the audio data.
[0208] For example, audio data is extracted from a bitstream and input to the playback device 101 in a decoded state, that is, in an uncompressed state.
[0209] The metadata separation unit 111 supplies the extracted object metadata of each object to the metadata conversion device 11 , and supplies the extracted waveform data of each object to the egocentric audio playback device 112 .
[0210] The metadata conversion device 11 is supplied with the object metadata from the metadata separation unit 111 as well as speaker layout information for playback in the Allocentric format, as in the case of FIG.
[0211] This speaker layout information may be stored in the bitstream together with the Allocentric audio data.
[0212] The metadata conversion device 11 performs the same metadata conversion process as in FIG. 3 or 9 based on the supplied speaker layout information and object metadata, and supplies the resulting egocentric object metadata to the egocentric audio playback device 112.
[0213] Allocentric audio data may also contain channel-based waveform data, i.e., waveform data for channels.
[0214] In such a case, the speaker position information of the speaker corresponding to the channel may be converted into object position data in the allocentric system as described above, and the object position data may be converted into object position data in the egocentric system. In other words, the waveform data of the channel may be regarded as waveform data of the object, and object position data may be generated.
[0215] Note that object position data of the waveform data of a channel may not be generated, and the waveform data of a channel may be directly supplied to the egocentric audio playback device 112 as waveform data for each channel (channel basis).
[0216] Moreover, hereinafter, the egocentric object metadata generated by the metadata conversion device 11 is also referred to as converted object metadata.
[0217] The egocentric audio playback device 112 performs egocentric rendering processing based on the converted object metadata of each object supplied from the metadata conversion device 11 and the waveform data of each object supplied from the metadata separation unit 111. That is, the egocentric audio playback device 112 functions as a rendering processing unit. Here, the egocentric rendering processing may be, for example, rendering processing using VBAP, or may be a rendering processing other than VBAP.
[0218] By the rendering process, output data for each channel for reproducing the sound of each object is generated from the waveform data of each object.
[0219] For example, if the output data is output to headphones, two-channel output data is generated, and if the output data is output to a 7.1.4-channel speaker system, 7.1.4-channel output data is generated. Furthermore, binaural two-channel output data that virtually reproduces a 7.1.4-channel speaker system, for example, may be output to headphones using a convolution operation of BRIR (Binaural Room Impulse Response) or HRTF (Head Related Transfer Function). While headphones are used as an example here, similar processing may also be performed on earphones, hearing aids, HMDs (Head Mounted Displays), and other devices.
[0220] The egocentric audio playback device 112 supplies the output data generated by the rendering process to headphones or a speaker system as the output destination, and plays back the sound of the content, that is, the sound of each object.
[0221] Second Embodiment Example of Configuration of Metadata Conversion Apparatus In the above, an example has been described in which object metadata (audio data) in the Allocentric format is converted into object metadata (audio data) in the Egocentric format.
[0222] However, the present invention is not limited to this, and object metadata (audio data) in the egocentric format may be converted into object metadata (audio data) in the allocentric format.
[0223] In such a case, the metadata conversion device may be configured as shown in FIG.
[0224] The metadata conversion device 201 shown in FIG. 12 includes a speaker layout information input unit 211 , an object metadata input unit 212 , an object position data generation unit 213 , a meta information conversion unit 214 , and an object metadata output unit 215 .
[0225] The speaker layout information input unit 211 acquires speaker layout information input from an external source, extracts speaker position information and speaker configuration information from the speaker layout information, and supplies the information to the object position data generation unit 213. This speaker configuration information is necessary for rendering processing in the Allocentric method.
[0226] Here, the speaker layout information acquired by the speaker layout information input unit 211 is information about the speaker system when actually playing back content, based on the converted Allocentric audio data. That is, the acquired speaker layout information indicates the channel configuration and speaker placement positions of the speaker system actually used for playback in the Allocentric format.
[0227] For example, when the speaker layout of the speaker system used for actual playback in the Allocentric format is 7.1.4ch, the speaker layout information shown in FIG. 4 is acquired.
[0228] The object metadata input unit 212 acquires egocentric object metadata from the outside, and extracts object position information and meta information from the acquired object metadata.
[0229] For example, object position data in polar coordinate format contained in egocentric object metadata is extracted as object position information. Also, name information, type information, size information, priority information, spread information, gain information, and other information contained in egocentric object metadata are extracted as meta information.
[0230] The object metadata input unit 212 supplies the extracted object position information to the object position data generation unit 213 , and also supplies the extracted meta information to the meta information conversion unit 214 .
[0231] The object position data generator 213 generates object position data indicating the position of an object during playback in the Allocentric format, based on the speaker position information, speaker configuration information, and object position information. In other words, the object position data generator 213 converts the object position data in the Egocentric format into object position data in the Allocentric format.
[0232] The object position data generation unit 213 includes a panning gain calculation unit 221 and an object position determination unit 222 .
[0233] The panning gain calculation unit 221 calculates a panning gain value based on the speaker position information and speaker configuration information supplied from the speaker layout information input unit 211 and the object position information supplied from the object metadata input unit 212, and supplies the calculated value to the object position determination unit 222.
[0234] For example, the panning gain calculation unit 221 calculates, for each object, a panning gain value to be assigned to each speaker indicated by the speaker configuration information based on an egocentric rendering method such as VBAP. Then, the panning gain value is directly supplied to the object position determination unit 222 as the panning gain value for each object to be assigned to each speaker during playback in the allocentric method. Note that the panning gain calculation unit 221 may use a rendering method other than VBAP.
[0235] The object position determination unit 222 determines the position of each object based on the panning gain value of each speaker for each object supplied from the panning gain calculation unit 221, and sets the determination result as object position data.
[0236] For example, the object position determination unit 222 calculates object position data in absolute coordinate format from the panning gain values of each speaker of the object by using an allocentric rendering method, i.e., by performing reverse calculations to determine panning gain values from object position data in absolute coordinate format.
[0237] The object position determination unit 222 supplies the calculated object position data in absolute coordinate format to the meta information conversion unit 214 and the object metadata output unit 215 as object position data in the Allocentric system.
[0238] In the egocentric method, the position of the object indicated by the object position data is directly used as the localized position of the sound of the object.
[0239] Therefore, the object position data generation unit 213 that converts egocentric object position data into allocentric object position data does not include the sound image localization estimation unit 31 that estimates the localization position of the sound of the object. In other words, the object position data generation unit 213 calculates a panning gain value and determines the object position.
[0240] The meta information conversion unit 214 appropriately converts the meta information supplied from the object metadata input unit 212 based on the object position data supplied from the object position determination unit 222, and supplies the metadata obtained by the conversion to the object metadata output unit 215.
[0241] The meta information conversion unit 214 converts the egocentric meta information into allocentric meta information, and outputs the converted meta information as metadata. Note that conversion of the meta information does not necessarily have to be performed.
[0242] The object metadata output unit 215 outputs data consisting of the object position data in absolute coordinate format supplied from the object position determination unit 222 and the metadata supplied from the meta information conversion unit 214 as object metadata in the Allocentric format.
[0243] 13, the metadata conversion process performed by the metadata conversion device 201 will be described. This metadata conversion process is started when, for example, the user designates egocentric audio data to be converted.
[0244] In step S101, the speaker layout information input unit 211 sets the speaker positions in the allocentric system in response to a user operation or the like.
[0245] For example, in step S101, similar to step S11 in FIG. 3, information about the speaker system when actually performing audio playback in the Allocentric format is set in response to a user's designated operation, and speaker layout information indicating the setting result is obtained.
[0246] The speaker layout information input unit 211 extracts speaker position information and speaker configuration information from the speaker layout information and supplies them to the panning gain calculation unit 221 .
[0247] In step S102, the object metadata input unit 212 acquires egocentric object metadata from the audio data specified by the user.
[0248] For example, in step S102, similar to step S12 in FIG. 3, object metadata for the processing period of the object to be processed is acquired.
[0249] In step S103, the object metadata input unit 212 extracts object position information from the acquired egocentric object metadata and supplies the information to the panning gain calculation unit 221. Here, the object position information extracted is in polar coordinate format. The object metadata input unit 212 also extracts meta information from the egocentric object metadata and supplies the information to the meta information conversion unit 214.
[0250] In step S104, the panning gain calculation unit 221 calculates a panning gain value based on the speaker position information and speaker configuration information supplied from the speaker layout information input unit 211 and the object position information supplied from the object metadata input unit 212, and supplies the calculated panning gain value to the object position determination unit 222.
[0251] For example, in step S104, a panning gain value for playing back the sound of the object to be processed egocentrically is calculated for each speaker located at the position indicated by the speaker position information using an egocentric rendering method such as VBAP. The panning gain value of each speaker is then used as the allocentric panning gain value for each speaker.
[0252] In step S105 , the object position determination unit 222 calculates object position data in the absolute coordinate format of the object to be processed based on the panning gain values of the speakers supplied from the panning gain calculation unit 221 .
[0253] Specifically, for example, the object position data is calculated from the panning gain values of each speaker by performing the calculations in reverse order (back calculation) of finding the panning gain values from object position data in absolute coordinate format using an allocentric rendering method.
[0254] In the Egocentric method, the sound of an object is localized in the same position regardless of the placement of the speakers. Therefore, by back-calculating the object position data in the Allocentric method using the panning gain values assigned to the speakers in the Egocentric method, it is possible to convert the object position data in the Egocentric method into object position data in the Allocentric method while maintaining the sound localization.
[0255] The object position determination unit 222 supplies the calculated object position data in absolute coordinate format to the meta information conversion unit 214 and the object metadata output unit 215 .
[0256] In step S106, the meta information conversion unit 214 converts the meta information supplied from the object metadata input unit 212 based on the object position data supplied from the object position determination unit 222. Note that the processing of step S106 does not necessarily have to be executed.
[0257] For example, the meta information conversion unit 214 converts the gain information according to the distance from the listening position to the object, etc., based on the object position data, as in step S18 of Figure 3, i.e., corrects (modifies) the gain information.
[0258] The meta information conversion unit 214 supplies the Allocentric meta information obtained by appropriately converting the Egocentric meta information to the object metadata output unit 215 as Allocentric metadata to be stored in the object metadata.
[0259] In step S107, the object metadata output unit 215 generates object metadata in the Allocentric format and outputs it to the subsequent stage.
[0260] For example, the object metadata output unit 215 outputs data consisting of object position data in absolute coordinate format supplied from the object position determination unit 222 and metadata supplied from the meta information conversion unit 214 as object metadata in the Allocentric format.
[0261] Through the above process, the egocentric object metadata for the object to be processed during the period to be processed is converted into allocentric object metadata.
[0262] After the processing of step S107 is performed, the processing of steps S108 and S109 is performed, and the metadata conversion processing is completed. However, since these processing steps are similar to the processing steps S20 and S21 of FIG. 3, their description will be omitted.
[0263] In this way, the metadata conversion device 201 calculates the panning gain value for playback in the egocentric format based on the speaker layout information in the allocentric format, and calculates object position data in the allocentric format.
[0264] In this way, egocentric object metadata can be converted to allocentric object metadata while preserving sound localization, which makes it possible to share audio data (data sharing) between different spatial audio formats, i.e., different data formats, while preserving sound localization.
[0265] As described above, according to the present technology, it is possible to convert object metadata between different data formats (spatial audio formats) in object-based audio, specifically between the allocentric format and the egocentric format, while preserving the sense of sound localization. This makes it possible to realize data sharing between different data formats while preserving the sense of sound localization.
[0266] For example, in the first embodiment described above, the sound localization direction, i.e., the localization position, of an object is estimated using the rendering method (Allocentric method) in the data format before conversion. Then, when audio is played back using the rendering method (Egocentric method) in the converted data format, object position data that localizes the sound in that localization direction is determined, and that object position data is used as the object position data in the converted data format. This makes it possible to convert audio data while maintaining the sound localization.
[0267] In particular, in the first embodiment, a composite vector indicating the localization position of the sound of an object is generated by combining speaker vectors consisting of the position (direction) of each speaker during playback in the allocentric system and the gain values (panning gain values) assigned to the speakers. This method makes it possible to estimate the localization direction, i.e., the localization position, of the sound of an object produced by allocentric rendering processing.
[0268] Furthermore, in the first modification of the first embodiment, it is determined for each object whether to convert the object position data based on an estimation of the sound image localization position or to convert the object position data by coordinate conversion based on simple physical laws. In this way, it is possible to reduce the processing time and processing load (amount of processing) during conversion.
[0269] Furthermore, in the second embodiment, for example, the panning gain values for each speaker during playback using the rendering method (Allocentric method) for the converted data format are calculated using the rendering method (Egocentric method) for the data format before conversion. Then, object position data for the converted data format is calculated by reverse calculation based on the panning gain values for each speaker. This makes it possible to convert audio data while maintaining the sound localization.
[0270] <Example of Computer Configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed on a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, for example, that can execute various functions by installing various programs.
[0271] FIG. 14 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.
[0272] In the computer, a CPU (Central Processing Unit) 501 , a ROM (Read Only Memory) 502 , and a RAM (Random Access Memory) 503 are interconnected by a bus 504 .
[0273] An input / output interface 505 is further connected to the bus 504. An input unit 506, an output unit 507, a recording unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.
[0274] The input unit 506 includes a keyboard, a mouse, a microphone, an image sensor, etc. The output unit 507 includes a display, a speaker, etc. The recording unit 508 includes a hard disk, a non-volatile memory, etc. The communication unit 509 includes a network interface, etc. The drive 510 drives a removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.
[0275] In a computer configured as described above, the CPU 501 loads a program recorded in the recording unit 508, for example, into the RAM 503 via the input / output interface 505 and the bus 504, and executes the program, thereby performing the above-described series of processes.
[0276] For example, in a computer, when a user operates the input unit 506, the CPU 501 displays the display screen shown in Fig. 10 on a display serving as the output unit 507 in response to a signal from the input unit 506. The CPU 501 then executes a program in response to the user's operation on the display screen shown in Fig. 10, thereby realizing the metadata conversion device 11 and executing the metadata conversion process described with reference to Figs. 3 and 9.
[0277] The program executed by the computer (CPU 501) can be provided by being recorded on a removable recording medium 511 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.
[0278] In a computer, a program can be installed in the recording unit 508 via the input / output interface 505 by inserting a removable recording medium 511 into the drive 510. The program can also be received by the communication unit 509 via a wired or wireless transmission medium and installed in the recording unit 508. Alternatively, the program can be installed in the ROM 502 or the recording unit 508 in advance.
[0279] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.
[0280] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.
[0281] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.
[0282] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.
[0283] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.
[0284] Furthermore, the present technology can also be configured as follows.
[0285] (1) An information processing device comprising: an acquisition unit that acquires speaker layout information during playback of audio data in a first data format; and a position data generation unit that converts, based on the speaker layout information, first position data of a sound source in the first data format or second position data of the sound source in a second data format whose rendering method during playback is different from that of the first data format, into the second position data or the first position data. (2) The information processing device described in (1), in which the rendering method during playback of the audio data in the first data format is an allocentric method. (3) The information processing device described in (1) or (2), in which the rendering method during playback of the audio data in the second data format is an egocentric method. (4) The information processing device described in (3), in which the rendering method is a rendering method using at least one of VBAP, HRTF, and BRIR. (5) The information processing device described in any one of (1) to (4), in which the coordinate format of the first position data is different from the coordinate format of the second position data. (6) The information processing device according to (5), wherein the first position data is position data in an absolute coordinate format, and the second position data is position data in a polar coordinate format based on a listening position. (7) The information processing device according to any one of (1) to (6), wherein the sound source is an object and has object metadata. (8) The information processing device according to (7), wherein the object metadata is at least any of name information indicating a name of the object, type information indicating a type of the object, size information indicating a size of the object, priority information indicating a priority of the object, spread information indicating a spread of the object, and gain information indicating a gain value of the object.(9) The information processing device according to any one of (1) to (8), wherein the position data generation unit estimates a localization position of the sound of the sound source when the audio data in the first data format is played back based on the speaker layout information and the first position data, and generates the second position data based on the estimated localization position. (10) The information processing device according to (9), wherein the position data generation unit calculates gain values of one or more speakers for the sound source based on the speaker layout information and the first position data, generates, for each speaker, a vector of the magnitude of the gain value of the speaker that faces in the direction of the speaker position, and estimates the localization position by combining the vectors generated for each of the one or more speakers. (11) The information processing device according to (9) or (10), wherein the position data generation unit determines whether the sound source is to be a target for estimation of the localization position, and if it is determined that the sound source is not to be a target for estimation, generates the second position data by performing coordinate transformation on the first position data of the sound source. (12) The information processing device according to (11), wherein the position data generation unit makes the determination based on object metadata of the audio data in the first data format, a user selection, or preset flag information. (13) The information processing device according to any one of (1) to (8), wherein the position data generation unit calculates gain values of one or more speakers for the sound source when the audio data in the first data format or the second data format is played back based on the speaker layout information and the first position data or the second position data, and generates the second position data or the first position data based on the gain values of the one or more speakers.(14) The information processing device according to any one of (1) to (13), further comprising: a meta information conversion unit that converts other information different from the first position data or the second position data, which is included in object metadata of the audio data in the first data format or the second data format, into the other information in the object metadata of the audio data in the second data format or the first data format, based on the second position data or the first position data obtained by conversion by the position data generation unit. (15) The information processing device according to (2), wherein, when the audio data includes waveform data of a channel, the position data generation unit converts the first position data into the second position data, using information indicating a position of a speaker corresponding to the channel as the first position data. (16) The information processing device according to (7), further comprising: a rendering processing unit that performs rendering processing based on object metadata of the object, which includes the second position data or the first position data obtained by conversion by the position data generation unit, and waveform data for reproducing sound of the object. (17) An information processing method in which an information processing device acquires speaker layout information when audio data in a first data format is played back, and converts, based on the speaker layout information, first position data of a sound source in the first data format or second position data of the sound source in a second data format whose rendering method during playback is different from that of the first data format, into the second position data or the first position data. (18) A program that causes a computer to execute a process in which:
[0286] REFERENCE SIGNS LIST 11 Metadata conversion device, 21 Speaker layout information input unit, 22 Object metadata input unit, 23 Object position data generation unit, 24 Metainformation conversion unit, 25 Object metadata output unit, 31 Sound image localization estimation unit, 32 Object position determination unit, 201 Metadata conversion device, 211 Speaker layout information input unit, 212 Object metadata input unit, 213 Object position data generation unit, 214 Metainformation conversion unit
Claims
1. An information processing device comprising: an acquisition unit that acquires speaker layout information when audio data in a first data format is played back; and a position data generation unit that converts, based on the speaker layout information, first position data of a sound source in the first data format, or second position data of the sound source in a second data format that has a rendering method during playback different from that of the first data format, into the second position data or the first position data.
2. The information processing apparatus according to claim 1, wherein a rendering method during playback of said audio data in said first data format is Allocentric.
3. The information processing apparatus according to claim 1, wherein a rendering method during playback of said audio data in said second data format is an egocentric method.
4. The information processing device according to claim 3, wherein the rendering method uses at least one of VBAP, HRTF, and BRIR.
5. The information processing device according to claim 1, wherein the first position data and the second position data have different coordinate formats.
6. The information processing device according to claim 5, wherein the first position data is position data in absolute coordinate format, and the second position data is position data in polar coordinate format with the listening position as the reference.
7. The information processing device according to claim 1, wherein the sound source is an object and has object metadata.
8. The information processing device of claim 7, wherein the object metadata is at least any one of name information indicating a name of the object, type information indicating a type of the object, size information indicating a size of the object, priority information indicating a priority of the object, spread information indicating a spread of the object, and gain information indicating a gain value of the object.
9. The information processing device according to claim 1, wherein the position data generation unit estimates a sound localization position of the sound source when the audio data in the first data format is played back based on the speaker layout information and the first position data, and generates the second position data based on the estimated localization position.
10. The information processing device of claim 9, wherein the position data generation unit calculates gain values for one or more speakers for the sound source based on the speaker layout information and the first position data, generates, for each speaker, a vector of the magnitude of the gain value of the speaker that faces in the direction of the speaker's position, and estimates the localization position by combining the vectors generated for each of the one or more speakers.
11. The information processing device according to claim 9, wherein the position data generation unit determines whether or not the sound source is to be a target for estimating the localization position, and if it determines that the sound source is not to be an estimation target, generates the second position data by performing coordinate transformation on the first position data of the sound source.
12. The information processing device according to claim 11, wherein the position data generating section makes the determination based on object metadata of the audio data in the first data format, a user selection, or preset flag information.
13. The information processing device of claim 1, wherein the position data generation unit calculates gain values for one or more speakers for the sound source when the audio data in the first data format or the second data format is played back based on the speaker layout information and the first position data or the second position data, and generates the second position data or the first position data based on the gain values for the one or more speakers.
14. The information processing device of claim 1, further comprising a meta information conversion unit that converts other information, different from the first position data or the second position data, contained in object metadata of the audio data in the first data format or the second data format, into the other information of object metadata of the audio data in the second data format or the first data format, based on the second position data or the first position data obtained by conversion by the position data generation unit.
15. The information processing device according to claim 2, wherein when the audio data includes waveform data for a channel, the position data generation unit converts the first position data into the second position data by using information indicating the position of a speaker corresponding to the channel as the first position data.
16. The information processing device according to claim 7, further comprising a rendering processing unit that performs rendering processing based on object metadata of the object, including the second position data or the first position data obtained by conversion by the position data generation unit, and waveform data for playing the sound of the object.
17. An information processing method in which an information processing device acquires speaker layout information when playing audio data in a first data format, and converts, based on the speaker layout information, first position data of a sound source in the first data format, or second position data of the sound source in a second data format having a rendering method during playback different from that of the first data format, into the second position data or the first position data.
18. A program causing a computer to execute a process of acquiring speaker layout information during playback of audio data in a first data format, and converting, based on the speaker layout information, first position data of a sound source in the first data format, or second position data of the sound source in a second data format having a rendering method during playback different from that of the first data format, into the second position data or the first position data.
Citation Information
Patent Citations
Acoustic coordinate conversion device and program
JP2022092487A
Information processing device and method, and program
WO2019198486A1
Cited By
Information processing method, information processing system, and program
WO2026005047A1
Conversion of rendering metadata of audio objects
WO2026226751A1
