Audio metadata processing device and program

Abstract

To convert the description of acoustic metadata in an object-based audio and avoid generation of different interpretations in the position of regeneration of a voice object according to time.SOLUTION: An acoustic metadata processor 1 includes: a division determination unit 11 for conducting a division determination as to whether the acoustic metadata of a specific time is to be divided by comparing the position of regeneration of a voice object between the acoustic metadata of the specific time and the acoustic metadata before the specific time; and an acoustic metadata conversion unit 12 for dividing the acoustic metadata of the specific time in a time direction on the division determination, or adding the acoustic metadata of the specific time to the acoustic metadata before the specific time instead of dividing the acoustic metadata of the specific time.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an audio metadata processing apparatus and a program for converting audio metadata in object-based audio. [Background technology]

[0002] Object-based audio is an audio method that records and transmits audio objects and audio metadata, and plays (renders) content in a format appropriate for the playback environment. It has become increasingly popular in the film industry as a method for playing back 3D audio, but in recent years, there have been moves to use object-based audio in next-generation broadcasting in various countries around the world. For example, the United States and South Korea, which have adopted the ATSC3.0 broadcasting format, have begun introducing it into their broadcasting systems using AC-4 or MPEG-H 3DA as the audio encoding format.

[0003] ITU-R has defined the Audio Definition Model (ADM) as an international standard for audio metadata for program production (see, for example, Non-Patent Document 1). The ADM is written in XML (Extensible Markup Language) and consists of a Content section in which the combination of audio objects that make up the program audio is described, and a Format section in which the speaker arrangement, playback position and gain of the audio objects are described. When producing object-based audio content, it is not realistic to manually write the metadata in the Format section, especially the playback position and gain of audio objects whose values ​​change over time, because of the effort involved. In reality, it is expected that production equipment such as an audio console and audio editing software will read parameters such as playback position and gain that have been adjusted by an audio engineer by operating the equipment, and automatically generate metadata descriptions.

[0004] When object-based audio is used in broadcasting or streaming services, audio metadata must be repeatedly transmitted so that playback can be started from any time when reception begins. For this reason, the time series representation format of ADM (Serial ADM: S-ADM) (see, for example, Non-Patent Document 2) and the transmission method (SMPTE ST 2116:2019) that synchronizes S-ADM with audio signals and outputs them using existing audio digital interfaces have been internationally standardized. S-ADM contains metadata in the Content section, which does not change throughout the content, as well as metadata in the Format section, which is necessary to generate the playback signal at that time. It is expected that audio metadata automatically generated by production equipment used in live production, such as audio consoles, will be output in this S-ADM format. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] ITU-R, Rec. ITU-R BS.2076-2, “Audio definition model”, 2019 [Non-Patent Document 2] ITU-R, Rec. ITU-R BS.2125-0, “A serial representation of the Audio Definition Model”, 2019 Summary of the Invention [Problem to be solved by the invention]

[0006] In ADM, each audio channel, such as an audio object whose playback position changes, is described using a descriptor called audioChannelFormat. Specific parameter changes such as playback position and gain are expressed by further dividing audioChannelFormat into multiple audio blocks called audioBlockFormat (hereinafter simply referred to as "blocks") in the time direction. Information about the start time and duration of the block, as well as the playback position at that time (azimuth, elevation, distance, etc.) and parameters such as gain are described in audioBlockFormat, and a new audioBlockFormat is added every time a parameter changes. However, with regard to the playback position, the parameters specified in each block are the parameters at the end of each block, and if there is a change in the playback position compared to the previous block, it is basically interpreted as a linear change from the parameters of the previous block to the parameters of the block over the time length of the block.

[0007] In ADM, the azimuth angle is described as 0 degrees in front of the listener as seen from above, with counterclockwise as positive and clockwise as negative. Here, the value changes discontinuously from ±180 degrees directly behind the listener. As shown in Figure 10(a), assume that an audio engineer operates production equipment such as an audio console to move the playback position of a certain audio object past directly behind the listener. In this case, in the audioBlockFormat of the automatically generated metadata, the playback position of the audio object may be described as either before or after the azimuth angle of ±180 degrees, depending on the method of obtaining the parameters, for example, +170 degrees in the previous block and -170 degrees in the current block, as shown in Figure 10(b). However, in this case, the object-based audio playback device (renderer) that reads the generated metadata may mechanically calculate the difference in the playback position and interpret it as moving in the reverse direction from +170 degrees to -170 degrees, passing through 0 degrees. In this way, depending on the description of the metadata relating to the vicinity of the azimuth angle discontinuity, multiple interpretations may arise regarding the movement direction of the playback position of the audio object, and the renderer may generate a playback signal that differs from the creative intent.

[0008] In consideration of the above circumstances, the object of the present invention is to provide an acoustic metadata processing device and program that converts descriptions of acoustic metadata that change the playback position of an audio object over time in object-based audio, and generates acoustic metadata that does not result in multiple interpretations of the change in playback position. [Means for solving the problem]

[0009] In order to solve the above problems, the acoustic metadata processing device of the present invention is an acoustic metadata processing device that converts acoustic metadata in which the playback position of an audio object is described in object-based audio, and is equipped with a split determination unit that compares the playback positions of the audio objects for the acoustic metadata at a given time with the acoustic metadata up to the time before that time, and makes a split determination as to whether or not to split the acoustic metadata at that time, and an acoustic metadata conversion unit that, based on the split determination, splits the acoustic metadata at that time in the time direction, or adds the acoustic metadata at that time to the acoustic metadata up to the time before that time without splitting it.

[0010] Furthermore, in the acoustic metadata processing device of the present invention, the split determination unit may determine the movement direction of the audio object from the difference between the playback position of the acoustic metadata at a given time and the playback position of the acoustic metadata up to the time prior to that time.

[0011] Furthermore, in the acoustic metadata processing device of the present invention, the split determination unit may determine the movement direction of the audio object from the difference between the rate of change of the playback position of the acoustic metadata at the time and the rate of change of the playback position of the acoustic metadata up to the time prior to the time.

[0012] Furthermore, in the acoustic metadata processing device of the present invention, the split determination unit may determine the movement direction of the audio object from the difference between the playback position of the acoustic metadata at a given time and a predicted playback position at that time calculated from the playback position of the acoustic metadata up to the time prior to that time.

[0013] Furthermore, in the audio metadata processing device according to the present invention, the division determination section may obtain the predicted playback position at the time by using Lagrange's interpolation formula.

[0014] Furthermore, in the acoustic metadata processing device of the present invention, when the playback position of an audio object changes discontinuously, the acoustic metadata conversion unit may divide the acoustic metadata into a chronological order and insert acoustic metadata with a duration of 0 seconds or acoustic metadata whose initial value is the discontinuous point.

[0015] In order to achieve the above object, a program according to the present invention causes a computer to function as the above audio metadata processing device. Effect of the Invention

[0016] According to the present invention, in object-based audio, audio metadata can be generated that does not lead to multiple interpretations regarding changes in playback position, and it is possible to prevent the renderer from generating a playback signal that differs from the production intention. [Brief description of the drawings]

[0017] [Figure 1] 1 is a block diagram showing an example of the configuration of an acoustic metadata processing device according to an embodiment; [Diagram 2] 1 is a flowchart illustrating an example of a processing procedure in an acoustic metadata processing device according to an embodiment. [Diagram 3] 11 is a table showing an example of setting main parameters during block transformation. [Figure 4] FIG. 13 is a diagram showing an example of a voice object moving counterclockwise while gradually accelerating as a first embodiment. [Diagram 5] 11 is a diagram showing the audioBlockFormat of an ADM before and after conversion in the first embodiment. FIG. [Figure 6] FIG. 11 is a diagram showing an example of a second embodiment in which a voice object moves counterclockwise while gradually accelerating at an acceleration faster than that of the first embodiment. [Figure 7] 13 is a diagram showing the audioBlockFormat of an ADM before and after conversion in the second embodiment. FIG. [Figure 8] As a third embodiment, an example will be shown in which a voice object moves repeatedly from side to side while gradually accelerating. [Figure 9] FIG. 13 is a diagram showing the audioBlockFormat of an ADM before and after conversion in the third embodiment. [Figure 10] FIG. 13 is a diagram showing an example in which the playback position of an audio object passes through 180 degrees. [Figure 11] FIG. 11 is a diagram showing an example of dividing the acoustic metadata in FIG. 10. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0019] Fig. 1 is a block diagram showing an example of the configuration of an acoustic metadata processing device according to an embodiment. The acoustic metadata processing device 1 shown in Fig. 1 includes a division determination unit 11, an acoustic metadata conversion unit 12, and an acoustic metadata storage unit 13. The acoustic metadata processing device 1 may include an acoustic metadata output unit that outputs metadata stored in the acoustic metadata storage unit 13 as a processed ADM at any point in time. In this embodiment, the acoustic metadata processing device 1 uses ADM as acoustic metadata, but is not limited to this. For example, the same processing can be performed on S-ADM.

[0020] The audio metadata processing device 1 is a device that generates audio metadata that does not lead to multiple interpretations of changes in playback position by converting descriptions of audio metadata that change the playback position of an audio object according to time in object-based audio. In this specification, "conversion" is a concept that includes both "division" and "addition," which will be described in detail below.

[0021] The acoustic metadata processing device 1 compares the unconverted audioBlockFormat included in the ADM, block by block, with the processed metadata (after conversion by the acoustic metadata conversion unit 12) stored in the acoustic metadata storage unit 13, and uses the result to convert the audioBlockFormat being processed to generate new metadata. The acoustic metadata processing device 1 stores the generated metadata in the acoustic metadata storage unit 13 one by one, and can output the processed ADM at any time.

[0022] The division determination unit 11 compares the playback positions of the audio objects for the acoustic metadata at a certain time (the newly input ADM) and the acoustic metadata up to that time (the converted ADM stored in the acoustic metadata storage unit 13), and performs a division determination as to whether or not to divide the acoustic metadata at that time. The division determination unit 11 outputs determination information indicating the determination result to the acoustic metadata conversion unit 12.

[0023] The acoustic metadata conversion unit 12 divides the acoustic metadata at the time in the time direction and adds it to the acoustic metadata up to the time, or adds the acoustic metadata at the time without dividing it to the acoustic metadata up to the time, based on the determination information input from the division determination unit 11. Then, the acoustic metadata conversion unit 12 outputs the converted acoustic metadata to the acoustic metadata storage unit 13.

[0024] FIG. 2 is a flowchart showing an example of a processing procedure of the acoustic metadata processing device 1. As shown in FIG.

[0025] In step S11, the division determination unit 11 starts a comparison process from the block with the smallest number (earliest time) among the blocks that have not yet been compared.

[0026] In step S12, the division determination unit 11 determines whether the block is the first block of the audioChannelFormat. If the block is the first block of the audioChannelFormat (step S12-Yes), the playback position described in the block is the initial value of the playback position of the audio object, so it is determined that division is unnecessary and the process proceeds to step S19. If the block is other than the first block of the audioChannelFormat (step S12-No), the process proceeds to step S13.

[0027] In step S13, the division determination unit 11 calculates whether the difference in azimuth angle between the playback positions of the block and the previous block is 180 degrees or more. If the difference in the described azimuth angle is less than 180 degrees (step S13-No), it is unlikely that there are multiple interpretations in the direction in which the playback position changes as described above, so it is determined that division is not necessary and the process proceeds to step S19. If the difference in azimuth angle with the previous block is 180 degrees or more (step S13-Yes), the process proceeds to step S14.

[0028] However, in special circumstances, such as when the frequency at which the production equipment, such as an audio console, acquires parameters of the playback position is low compared to the moving speed of the playback position of the audio object, even if the difference in the described azimuth angle is less than 180 degrees, it may be interpreted as moving in a direction different from that during production. In preparation for such conditions, the comparison process of whether the difference exceeds 180 degrees may not be performed, and all blocks may proceed to the calculation of the evaluation function described below. In that case, the process of step S13 surrounded by the dashed line is omitted. Note that when the evaluation function is calculated for all blocks, the evaluation function is calculated for all blocks except the first block, and therefore the amount of processing by the audio metadata processing device 1 is considered to be large.

[0029] In step S14, the division determination unit 11 determines the blocks to be used for calculating the evaluation index in step S15.

[0030] In step S15, the division determination unit 11 compares the playback position of the previous block with the playback position of the current block to calculate an evaluation index. A specific calculation method will be described later.

[0031] In step S16, the division determination unit 11 determines the moving direction of the audio object from the value of the evaluation index, and determines whether or not it will pass an azimuth angle of 0 degrees or an azimuth angle of 180 degrees. The azimuth angle of 180 degrees may be either positive or negative. If it is determined in step S16 that it will pass an azimuth angle of 0 degrees, the process proceeds to step S17, and if it is determined that it will pass an azimuth angle of 180 degrees, the process proceeds to step S18. In addition, if the process of step S13 is omitted, and if it is determined in step S16 that it will not pass an azimuth angle of 0 degrees or an azimuth angle of 180 degrees, the process proceeds to step S19.

[0032] In step S17, the audio metadata conversion unit 12 divides the block at the time when the block passes through an azimuth angle of 0 degrees. When passing through an azimuth angle of 0 degrees, it is expected that a renderer that reads metadata described without dividing the block as is will generate an audio signal that moves through 0 degrees, but when the difference in playback position with the previous block exceeds 180 degrees, the block may be divided at the time when the block passes through an azimuth angle of 0 degrees in order to clearly indicate the moving direction of the audio object. When division at an azimuth angle of 0 degrees is not performed, the process of step S17 surrounded by a dashed line is omitted.

[0033] When dividing at an azimuth angle of 0 degrees, the time when the azimuth angle of 0 degrees is passed is calculated based on the inclination of the block, and the block is divided into a block from the block start time to the time when the azimuth angle of 0 degrees is passed, and a block from the time when the azimuth angle of 0 degrees is passed to the block end time. The division times are calculated using the following formula. Here, t divided is the time when the block is divided, and t n is the start time of the block, and d n is the duration of the current block and the previous block.

[0034] [Number]

[0035] FIG. 3 is a table showing an example of settings of main parameters (audioBlockFormatID, rtime, duration, position, jumpPosition, and interpolationLength) during block conversion.

[0036] In step S17, the block is divided, specifically, according to the setting shown in Fig. 3 *2. That is, the audioBlockFormatID of the divided first half block (hereinafter referred to as the "first half block") is set to a value obtained by incrementing the ID of the converted metadata already stored in audio metadata storage unit 13. The audioBlockFormatID of the divided second half block (hereinafter referred to as the "second half block") is set to a value obtained by further incrementing the audioBlockFormatID of the first half block.

[0037] In step S18, the audio metadata conversion unit 12 divides the block at the time when the point of 180 degrees azimuth angle is passed. For example, as shown in Fig. 10(a), in a movement exceeding 180 degrees azimuth angle, the difference in azimuth angle is 180 degrees or more (Yes in step S13). Then, since it is determined in step S16 that the azimuth angle of 180 degrees is passed, in step S18, the block is divided as shown in Fig. 11.

[0038] The audio metadata conversion unit 12 finds the time when the block passes an azimuth angle of 180 degrees based on the inclination of the block, divides the block into a first half block from the block start time to the time when the block passes an azimuth angle of 180 degrees, and a second half block from the time when the block passes an azimuth angle of 180 degrees to the block end time, and further inserts a block (hereinafter referred to as an "inserted block") with a duration of 0 seconds that indicates instantaneous movement to the described playback position between the first half block and the second half block. The time to divide is calculated using the following formula, where t divided is the time when the block is divided, and t n is the start time of the block, and d n is the duration of the current block and the previous block.

[0039]

number

[0040] In step S18, the block is divided by the setting shown in FIG. 3. That is, the playback position of the first half block (parameter at the end of the first half block) is set to the playback position θ n-1 If ≥ 0 degrees, set it to +180 degrees, and set the playback position θ of the previous block n-1 If the offset is <0 degrees, it is set to -180 degrees. Also, the playback position of the insert block is set to -180 degrees if the playback position of the first half block is +180 degrees, and +180 degrees if the playback position of the first half block is -180 degrees, and jumpPosition=1 is written to indicate an instantaneous move to the described playback position. The audioBlockFormatID of the first half block is set to an incremented value of the ID of the converted metadata already stored in the audio metadata storage unit 13. The audioBlockFormatID of the insert block is set to an incremented value from the audioBlockFormatID of the first half block. The audioBlockFormatID of the second half block is set to an incremented value from the audioBlockFormatID of the insert block.

[0041] Here, in cases where the output of the audio metadata processing device 1 is an S-ADM and the division point coincides with the start point of a frame of the S-ADM, an initialization block may be inserted instead of an insertion block. An initialization block is a special block that can be used only in S-ADM and describes parameters at the start point of the block, and does not have a duration. Since the last eight digits of the audioBlockFormatID of an initialization block are fixed to 00000000, when an initialization block is used, the audioBlockFormatID of the latter block is set to a value incremented from the audioBlockFormatID of the first block. In this way, when the playback position of an audio object changes discontinuously, the audio metadata conversion unit 12 divides the audio metadata into chronological order and inserts audio metadata (insertion block) with a duration of 0 seconds, or audio metadata (initialization block) with the discontinuous point as its initial value.

[0042] In step S19, if the block is the first block of the audioChannelFormat, the audio metadata conversion unit 12 records the block as it is as the first block of the converted blocks. If the block is other than the first block of the audioChannelFormat, the block is added as it is after the converted metadata already stored in the audio metadata storage unit 13. However, since the block before the block may have been divided, only the audioBlockFormatID is rewritten to a value obtained by incrementing the ID of the converted metadata already stored in the audio metadata storage unit 13. That is, the block is added with the setting shown in FIG. 3 *1.

[0043] In step S20, it is determined whether the block division determination process and conversion process have been performed on all received blocks, and the processes of steps S11 to S19 are repeated until the processing of all blocks is completed. The acoustic metadata conversion unit 12 may output the converted ADM to the acoustic metadata storage unit 13 each time one block is processed. Furthermore, when the acoustic metadata processing device 1 receives an S-ADM, the converted ADM may be output for each frame of the S-ADM, or may be output when reception of the S-ADM is completed.

[0044] Next, several specific examples of the calculation of the evaluation index in step S15 and the determination of the movement direction in step S16 will be described.

[0045] <First evaluation index> In a first example, the division determination unit 11 calculates the amount of change from the playback position of the previous block to the playback position of the block (i.e., the difference between the playback position of the audio metadata at the time and the playback position of the audio metadata up to the time before the time) as a first evaluation index for each of the cases where it is interpreted as passing through an azimuth angle of 180 degrees and the case where it is interpreted as passing through an azimuth angle of 0 degrees. For example, the division determination unit 11 calculates the first evaluation index by the following formula.

[0046]

number

[0047] Here, a 180 is an evaluation index when it is interpreted as passing through an azimuth angle of 180 degrees, and a0 is an evaluation index when it is interpreted as passing through an azimuth angle of 0 degrees. θ n and θ n-1 is the azimuth angle at the playback position of the current block and the previous block. 180 The smaller of a0 and a1 is adopted as the moving direction of the voice object. 180 If a0 is smaller, it is determined that the beam will pass through an azimuth angle of 180 degrees, and if a1 is smaller, it is determined that the beam will pass through an azimuth angle of 0 degrees.

[0048] <Second evaluation index> In a second example, the division determination unit 11 calculates the difference between the rate of change of the playback position of the block (amount of change in azimuth angle tilt) and the rate of change of the playback position of the previous block (i.e., the difference between the rate of change of the playback position of the audio metadata at the time and the rate of change of the playback position of the audio metadata up to the time before the time) as a second evaluation index for each of the cases where it is interpreted as passing through an azimuth angle of 180 degrees and where it is interpreted as passing through an azimuth angle of 0 degrees. For example, the division determination unit 11 calculates the second evaluation index by the following formula.

[0049]

number

[0050] where b 180 is an evaluation index when it is interpreted as passing through an azimuth angle of 180 degrees, and b0 is an evaluation index when it is interpreted as passing through an azimuth angle of 0 degrees. θ n-2 is the azimuth angle at the playback position of the block before the previous one. n and d n-1 is the duration of the current block and the previous block. 180 The smaller of b0 and b1 is adopted as the moving direction of the voice object. 180 If a0 is smaller, it is determined that the signal passes through an azimuth angle of 180 degrees, and if b1 is smaller, it is determined that the signal passes through an azimuth angle of 0 degrees.

[0051] <Third evaluation index> In a third example, the split judgment unit 11 expresses the change in the playback position of the block immediately preceding the block in question as a curve consisting of a polynomial, and calculates the difference between the predicted value of the playback position of the block in question extrapolated based on the curve and the actually written playback position of the block in question (i.e., the difference between the playback position of the acoustic metadata at that time and the predicted playback position at that time obtained from the playback position of the acoustic metadata up to that time) as a third evaluation index, for both the case where it is interpreted as passing through an azimuth angle of 180 degrees and the case where it is interpreted as passing through an azimuth angle of 0 degrees.

[0052] There are several methods for calculating a curve that passes through several measured values, but many of them assume that the values ​​are obtained at equal intervals. However, the standardized ADM does not express the change in playback position as a set of many points, but rather uses a "drawing method" that expresses it as a line segment connecting one playback position to the next, and the time length of the audioBlockFormat, that is, the time interval at which the playback position is described, is not necessarily constant. In addition, audio editing software has long been equipped with so-called automation functions that record changes in the playback position of audio signals operated by audio engineers in some way, but the points at which this playback position is recorded are often not evenly spaced, and even if it becomes possible in the future to output metadata recorded in the automation of such audio editing software as ADM, it is expected that the time length of the audioBlockFormat will differ depending on the block.

[0053] Therefore, it is desirable to calculate the curve using a method that can be used for non-uniform grids, such as the Lagrange interpolation formula or the least squares method. For example, by using the third-order Lagrange interpolation formula, the reproduction position (prediction value) of the block in question can be calculated using the following formula.

[0054]

number

[0055] Here, θ p is the predicted position of the block. n …d n-3 are the durations of the block, the previous block, the block before the previous block, and the block before the previous block. n-1 …θ n-3 are the playback positions of the previous block, the block before previous, and the block before previous. For example, the division determination unit 11 calculates the third evaluation index by the following formula.

[0056]

number

[0057] where c 180 is an evaluation index when it is interpreted that the beam passes through an azimuth angle of 180 degrees, and c0 is an evaluation index when it is interpreted that the beam passes through an azimuth angle of 0 degrees. 180 The smaller of c0 and c1 is adopted as the moving direction of the voice object. 180 If c0 is smaller, it is determined that the signal passes through an azimuth angle of 180 degrees, and if c1 is smaller, it is determined that the signal passes through an azimuth angle of 0 degrees.

[0058] It is known that any polynomial-based interpolation method, such as Lagrange's interpolation formula or the least squares method, can cause the Runge phenomenon, in which fine oscillations occur in the values, when the degree is increased too much, resulting in poor accuracy. Even if the audio metadata regarding the playback position of an audio object is automatically generated by the production equipment, considering that the movement of the original audio object is expressed by the operation of the production equipment by an audio engineer, it is difficult to imagine an operation that finely oscillates the playback position in a very short time in terms of content production. Therefore, it is considered sufficient for the division determination unit 11 to perform quadratic or cubic interpolation, which is generally used for curved line interpolation.

[0059] <Fourth evaluation index> Of the above-mentioned first to third evaluation indexes, the first evaluation index is the easiest to calculate. In addition, when the audioBlockFormat is recorded periodically with a very short time length (for example, with the same period as the video signal), it is considered that in many cases, it is sufficient to use the first evaluation index for practical purposes. On the other hand, the second and third evaluation indexes also use information on several blocks immediately preceding the block in question, so that when the production equipment does not frequently obtain parameters related to the playback position or when the time length of the audioBlockFormat is at irregular intervals, it is possible to reduce the rate of erroneously adopting a direction different from the producer's intention as the movement direction.

[0060] Therefore, in the fourth example, in order to standardize the processing under various conditions, the division judgment unit 11 calculates an evaluation value obtained by normalizing the first to third evaluation indexes and then weighting and adding them as a fourth evaluation index for each of the cases where it is interpreted that the light passes through an azimuth angle of 180 degrees and the case where it is interpreted that the light passes through an azimuth angle of 0 degrees. For example, the division judgment unit 11 calculates the fourth evaluation index by the following formula.

[0061]

number

[0062] Here, w 180 is an evaluation index when it is interpreted that the direction passes through an azimuth angle of 180 degrees, and w0 is an evaluation index when it is interpreted that the direction passes through an azimuth angle of 0 degrees. α, β, and γ are weighting coefficients for the first to third evaluation indexes, respectively. The direction with the higher evaluation value is adopted as the moving direction. The division determination unit 11 uses w 180 The larger of w and w0 is adopted as the moving direction of the voice object. 180 If w0 is larger, it is determined that the azimuth angle will be 180 degrees, and if w0 is larger, it is determined that the azimuth angle will be 0 degrees.

[0063] To calculate the first evaluation index, the playback position of the previous block is required in addition to the block in question. To calculate the second evaluation index, the playback positions of the previous block and the block before that are required in addition to the block in question. To calculate the third evaluation index, the playback positions of multiple previous blocks are required in addition to the block in question. If there is no block before the block in question, or if a block exists but the playback position has not changed, the evaluation index to be used may be limited. For example, if only the previous block exists before the block in question, the weighting coefficients in formula (7) are set to α=1, β=0, γ=0, etc. so that the judgment can be made using only the previous block. Here, for each block whose movement direction needs to be judged, the weighting may be changed each time according to the number of blocks available for calculation (the number of blocks that exist before the block in question and have a change in playback position). For example, if the previous block can be used, α=1, β=0, γ=0, and if the previous two blocks can be used in the processing of the next block, α=1, β=1, γ=0.

[0064] Similarly, if the block contains a jumpPosition description that instantaneously moves to the playback position described in the block, the direction of movement cannot be determined from the transition of the playback position prior to the previous block. Therefore, it is possible to use weighting coefficients α=1, β=0, γ=0 and adopt the direction that physically brings the playback position closer as the direction of movement.

[0065] Also, if the previous block includes a movement of an azimuth angle exceeding ±180 degrees, it is possible to use the block before the azimuth angle exceeds ±180 degrees by appropriately adding ±360 degrees to the block before or after the azimuth angle exceeds ±180 degrees, but since this makes the calculations complicated, weighting may be set to use only the block after the azimuth angle exceeds ±180 degrees. As mentioned above, in cases where the audioBlockFormat is recorded at a period of sufficiently short time length with respect to the movement of the audio object, it is considered sufficient in many cases to use only the first evaluation index, i.e., the previous block.

[0066] Next, specific examples of ADMs converted by the audio metadata processing device 1 will be described below as Examples 1 to 4. Since the audioBlockFormatID that uniquely identifies the audioBlockFormat of the input ADM and the recorded ADM differ, the audioBlockFormatID of the input ADM will be described as source AB_xxxxyyyy_zzzzzzzz, and that of the recorded ADM will be described as destination AB_xxxxyyyy_zzzzzzzz.

[0067] Example 1 As a first embodiment, Fig. 4(a) shows an example in which an audio object moves counterclockwise while gradually accelerating. Here, the first evaluation index is used as an index for determining the moving direction of the audio object. Fig. 4(b) shows the behavior that may occur when the audio metadata is not divided, and Fig. 4(c) shows an example in which the audio metadata is divided to achieve the intended behavior.

[0068] Fig. 5 shows the audioBlockFormat of the ADM before and after conversion. When an ADM including audioBlockFormat from AB_00030001_00000001 to AB_00030001_00000005 as shown in Fig. 5 is input to the audio metadata processing device 1, processing begins with the block with the lowest number, AB_00030001_00000001.

[0069] First, the division determination unit 11 determines that the block (source AB_00030001_00000001) is the first block and therefore does not need to be divided. Therefore, the audio metadata conversion unit 12 records the block (source AB_00030001_00000001) as it is as the first block of the converted blocks (destination AB_00030001_00000001).

[0070] For the next block (source AB_00030001_00000002), the division determination unit 11 calculates whether the difference in azimuth angle of the playback position between the block (source AB_00030001_00000002) and the previous block (destination AB_00030001_00000001) is 180 degrees or more. Since the difference in the described azimuth angle is 5 degrees, which is less than 180 degrees, it is determined that conversion is not necessary. The audio metadata conversion unit 12 adds the block (source AB_00030001_00000002) after the already stored converted metadata (destination AB_00030001_00000001 in this case) after incrementing the ID (destination AB_00030001_00000002).

[0071] The next block (source AB_00030001_00000003) and the block after that (source AB_00030001_00000004) are also added without splitting (destination AB_00030001_00000003, destination AB_00030001_00000004), just like source AB_00030001_00000002.

[0072] For the original block AB_00030001_00000005, the comparison process of the split judgment unit 11 shows that the change in playback position in azimuth angle is |(-160 degrees)-100 degrees|=260 degrees, which is greater than 180 degrees, so an evaluation index for determining the movement direction of the audio object is calculated.

[0073] The first evaluation index determines the movement direction from the magnitude of the change in playback position. Because the playback position of the previous block (conversion destination AB_00030001_00000004) is greater than 0 degrees, the first evaluation index is calculated by the following formula. Therefore, the division determination unit 11 determines that the movement passes through an azimuth angle of 180 degrees, which is a small amount of change.

[0074]

number

[0075] The audio metadata conversion unit 12 performs division at a point passing through an azimuth angle of 180 degrees in accordance with the determination result of the division determination unit 11. Since the playback position of the previous block (converted block AB_00030001_00000004) is greater than 0 degrees, the division time is found by the following formula.

[0076]

number

[0077] Therefore, as shown in FIG. 5, the audio metadata conversion unit 12 converts the block (the original block AB_00030001_00000005) into divided The first half block has a length up to t, the second half block has a duration of 0 seconds, and the third half block has a duration of t divided The audioBlockFormatID is incremented from the ID of the previous block (converted AB_00030001_00000004) to become converted AB_00030001_00000005, converted AB_00030001_00000006, and converted AB_00030001_00000007, respectively. When the processing of the current block is completed, all input blocks have been processed, and the audio metadata processing device 1 ends its processing.

[0078] Example 2 As the second embodiment, Fig. 6(a) shows an example in which an audio object moves counterclockwise while gradually accelerating at an acceleration faster than that of the first embodiment. Here, the second evaluation index is used as an index for determining the moving direction of the audio object. Fig. 6(b) shows the behavior that may occur when the audio metadata is not divided, and Fig. 6(c) shows an example in which the audio metadata is divided to achieve the intended behavior.

[0079] 7 shows the audioBlockFormat of the ADM before and after conversion. The audioBlockFormat from AB_00030001_00000001 to AB_00030001_00000004 is processed without division as in the first embodiment (converted AB_00030001_00000001 to AB_00030001_00000004).

[0080] For the original block AB_00030001_00000005, the comparison process of the division judgment unit 11 shows that the change in playback position in azimuth angle is |(-90 degrees)-90 degrees|=180 degrees, so an evaluation index for determining the movement direction of the audio object is calculated.

[0081] The second evaluation index determines the movement direction from the magnitude of the gradient of the playback position change. Because the playback position of the previous block (conversion destination AB_00030001_00000004) is greater than 0 degrees, the following formula is used. Therefore, division determination unit 11 determines that the movement passes through an azimuth angle of 180 degrees, which is a small change in gradient.

[0082]

number

[0083] The audio metadata conversion unit 12 performs division at a point passing through an azimuth angle of 180 degrees in accordance with the determination result of the division determination unit 11. Since the playback position of the previous block (converted block AB_00030001_00000004) is greater than 0 degrees, the division time is found by the following formula.

[0084]

number

[0085] Therefore, as shown in FIG. 7, the audio metadata conversion unit 12 converts the block (the original block AB_00030001_00000005) into divided The first half block has a length up to t, the inserted block has a duration of 0 seconds, anddivided The audioBlockFormatID is incremented from the ID of the previous block (converted AB_00030001_00000004) to become converted AB_00030001_00000005, converted AB_00030001_00000006, and converted AB_00030001_00000007, respectively. When processing of the current block is completed, all input blocks have been processed, and therefore the audio metadata processing device 1 terminates processing.

[0086] In the second embodiment, the magnitude of the change in the playback position is equal to 180 degrees in either direction of movement, and the first evaluation index cannot determine the direction of movement. The second evaluation index includes the playback position of not only the previous block but also the block before the previous block in the calculation, so that the second evaluation index takes into account the tendency of the playback position to move, and may be able to make a more appropriate determination than the first evaluation index. However, in the case of a system that generates an audioBlockFormat in which the playback position is recorded at a constant period, as in the first and second embodiments, in order to reduce the error with the operation of the production equipment by the engineer, it is desirable to generate the audioBlockFormat at a period that is sufficiently short with respect to the expected moving speed of the audio object. In the second embodiment, the direction of movement is determined by using the second evaluation index, but the audioBlockFormat should be recorded at a period shorter than 2 seconds, and in that case, it is considered that the first evaluation index can be used to make a sufficient determination.

[0087] Example 3 As a third embodiment, Fig. 8(a) shows an example in which an audio object moves repeatedly left and right while gradually accelerating. Here, the third evaluation index is used as the index for determining the moving direction of the audio object, and the third-order Lagrange interpolation formula is used as the method for calculating the curve. Fig. 8(b) shows the behavior that may occur when the audio metadata is not divided, and Fig. 8(c) shows an example in which the audio metadata is divided to achieve the intended behavior.

[0088] 9 shows the audioBlockFormat of the ADM before and after conversion. The audioBlockFormat from AB_00030001_00000001 to AB_00030001_00000004 is processed without division as in the first embodiment (converted AB_00030001_00000001 to AB_00030001_00000004).

[0089] For the original block AB_00030001_00000005, the comparison process of the division determination unit 11 shows that the change in playback position with respect to azimuth is |(-90 degrees)-100 degrees|=190 degrees, which is greater than 180 degrees, so the evaluation index for determining the movement direction of the audio object is calculated by the following formula. As a result, the division determination unit 11 determines that the movement passes through an azimuth angle of 0 degrees, which is a small deviation from the predicted value.

[0090]

number

[0091] Since the block in question passes through an azimuth angle of 0 degrees and the difference in playback position with the previous block is large, exceeding 180 degrees, the audio metadata conversion unit 12 performs division at the point where the block passes through an azimuth angle of 0 degrees in accordance with the determination result of the division determination unit 11. The time of division is calculated by the following formula.

[0092]

number

[0093] Therefore, as shown in FIG. 9, the audio metadata conversion unit 12 converts the block (the original block AB_00030001_00000005) into divided The first half of the block, with a length of t dividedThe audioBlockFormatID is incremented from the ID of the previous block (converted AB_00030001_00000004) to become converted AB_00030001_00000005 and converted AB_00030001_00000006, respectively. When processing of the current block is completed, all input blocks have been processed, and therefore the audio metadata processing device 1 terminates processing.

[0094] Example 4 As the fourth embodiment, as in the third embodiment, an example is shown in which an audio object moves repeatedly left and right while gradually accelerating as shown in Fig. 8(a). Here, the fourth evaluation index is used as an index for determining the moving direction of the audio object, and the third evaluation index is calculated using a third-order Lagrange interpolation formula. Also, an example is shown in which the division determination unit 11 does not perform a comparison process to determine whether the difference in azimuth angles exceeds 180 degrees, but calculates the evaluation function for all blocks.

[0095] The audioBlockFormat of the ADM before and after conversion is as shown in Fig. 9, the same as in the third embodiment. First, the division determination unit 11 determines that the source AB_00030001_00000001 is the first block and therefore does not need to be divided. Therefore, the audio metadata conversion unit 12 records this block (source AB_00030001_00000001) as it is as the first block of the converted blocks (destination AB_00030001_00000001).

[0096] For the next block (source AB_00030001_00000002), the division determination unit 11 does not calculate whether the difference in azimuth angle between the playback positions of the block (source AB_00030001_00000002) and the previous block (destination AB_00030001_00000001) is 180 degrees or more, but calculates the evaluation index. At this point, the only block that can be used for index calculation is the previous block (destination AB_00030001_00000001), so the weighting of the fourth evaluation index is set to α=1, β=0, γ=0. In this case, the calculation is performed as follows:

[0097]

number

[0098] Therefore, the division determination unit 11 determines that the voice object does not pass through 180 degrees, but moves from 0 degrees to -5 degrees clockwise. Since neither 0 degrees nor 180 degrees pass between the block (source AB_00030001_00000002) in question, it determines that there is no need to divide the block (source AB_00030001_00000002) in question. It increments the ID and adds it (destination AB_00030001_00000002) after the previous block (destination AB_00030001_00000001).

[0099] When the division determination unit 11 calculates the evaluation index for the next block (source AB_00030001_00000003), there are two blocks (destination AB_00030001_00000001, destination AB_00030001_00000002) that can be used to calculate the index, so the weighting of the fourth evaluation index is set to α = 1, β = 1, γ = 0. Since the playback position of the previous block (destination AB_00030001_00000002) is < 0 degrees, the calculation is performed as follows:

[0100]

number

[0101] Therefore, the division determination unit 11 determines that the voice object does not pass through 180 degrees, but moves from -5 degrees counterclockwise to 105 degrees. The block (source AB_00030001_00000003) passes through 0 degrees, but the difference is below 180 degrees, so there is no need to divide it at the point where it becomes 0 degrees. Therefore, the ID of the block (source AB_00030001_00000003) is incremented from the ID of the previous block (destination AB_00030001_00000002) and added as destination AB_00030001_00000003.

[0102] For the next block (source AB_00030001_00000004), when the division determination unit 11 calculates the evaluation index, there are three blocks (destination AB_00030001_00000001, destination AB_00030001_00000002, destination AB_00030001_00000003) that can be used for the index calculation. In the third evaluation index, when the order of the Lagrange interpolation formula, which is the curve calculation method, is set to third order, four blocks are required for the calculation, so the third evaluation index cannot be used for the block (source AB_00030001_00000004). In order to make the calculation of the evaluation index more flexible, the order may be calculated to second order, but in the fourth embodiment, an example in which the order is calculated only to third order will be described. Since the third evaluation index cannot be used, the weighting of the fourth evaluation index is set to α = 1, β = 1, γ = 0. Since the playback position of the previous block (transformed block AB_00030001_00000004) is ≧0 degrees, the calculation is performed as follows.

[0103]

number

[0104] Therefore, the division determination unit 11 determines that the audio object does not pass through the point with an azimuth angle of 180 degrees, but moves clockwise from 105 degrees to 100 degrees. Since the block (source AB_00030001_00000004) does not pass through either 0 degrees or 180 degrees, it is determined that there is no need to divide it, and the ID is incremented and the block (destination AB_00030001_00000003) is added after the previous block (destination AB_00030001_00000004).

[0105] When the division determination unit 11 calculates the evaluation index for the next block (source AB_00030001_00000005), there are four blocks (destination AB_00030001_00000001 to AB_00030001_00000004) that can be used to calculate the index. Since all evaluation indexes can be used, the weighting of the fourth evaluation index is set to α=1, β=1, γ=1. Since the playback position of the previous block (destination AB_00030001_00000004) is ≧0 degrees, the calculation is performed as follows:

[0106]

number

[0107] Therefore, the division determination unit 11 determines that the voice object does not pass through 180 degrees, but moves clockwise from 100 degrees to -90 degrees.

[0108] Since the block in question passes through an azimuth angle of 0 degrees and the difference in playback position with the previous block is large, exceeding 180 degrees, the audio metadata conversion unit 12 divides the block at the point where the block passes through an azimuth angle of 0 degrees, in accordance with the determination result of the division determination unit 11. The division process is performed in the same manner as in the third embodiment. When the processing of the block in question is finished, all input blocks have been processed, and so the audio metadata processing device 1 ends the process.

[0109] (program) It is also possible to use a computer capable of executing program instructions to function as the above-mentioned audio metadata processing device 1. Here, the computer may be a general-purpose computer, a special-purpose computer, a workstation, a PC (Personal Computer), an electronic notepad, etc. The program instructions may be program code, code segments, etc. for performing the necessary tasks.

[0110] The computer includes a processor, a storage unit, an input unit, an output unit, and a communication interface. The processor may be a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), a SoC (System on a Chip), or the like, and may be composed of multiple processors of the same type or different types. The processor reads a program from the storage unit and executes it to control each of the above components and perform various arithmetic processing. Note that at least a part of the processing contents may be realized by hardware. The input unit is an input interface that receives a user's input operation and acquires information based on the user's operation, and is a pointing device, a keyboard, a mouse, or the like. The output unit is an output interface that outputs information, and is a display, a speaker, or the like. The communication interface is an interface for communicating with an external device.

[0111] The program may be recorded on a computer-readable recording medium. By using such a recording medium, the program can be installed on the computer. Here, the recording medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, and may be, for example, a CD-ROM, a DVD-ROM, or a USB (Universal Serial Bus) memory. In addition, the program may be in a form that is downloaded from an external device via a network.

[0112] In this way, according to the present invention, in object-based audio, the description of audio metadata that changes the playback position of an audio object according to time, which is automatically generated by a production facility, can be converted and the description of the playback position can be appropriately divided. Therefore, audio metadata that does not lead to multiple interpretations regarding the change in playback position can be generated, and it is possible to prevent the renderer from generating a playback signal that differs from the production intention.

[0113] Moreover, since the audio metadata processing device 1 performs conversion based on the playback position described in the audioBlockFormat of the relevant time and the audioBlockFormat of the immediately preceding few blocks, successive conversion of new metadata is also possible by applying the present invention to the S-ADM. In this way, it is clear to those skilled in the art that by applying the present invention, both online conversion, such as converting an S-ADM input from an audio console at a front stage in a live production system and outputting it as an S-ADM between a renderer at a rear stage, and offline conversion, such as re-recording an audio file including an ADM in which metadata from the start to the end of a program is described, as an audio file including the converted ADM, are possible.

[0114] Although the above-mentioned embodiment has been described as a representative example, it is obvious to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention should not be interpreted as being limited by the above-mentioned embodiment, and various modifications and changes are possible without departing from the scope of the claims. For example, it is possible to combine multiple building blocks shown in the configuration diagram of the embodiment into one, or to divide one building block. [Explanation of symbols]

[0115] 1. Acoustic Metadata Processing Device 11 Division determination section 12 Audio Metadata Conversion Unit 13. Audio Metadata Storage

Claims

1. An audio metadata processing device that converts audio metadata describing a playback position of an audio object in object-based audio, comprising: a division determination unit that compares playback positions of audio objects for the audio metadata at the time and the audio metadata up to the time, and performs a division determination as to whether or not to divide the audio metadata at the time; an acoustic metadata conversion unit that divides the acoustic metadata at the time in a time direction based on the division decision, or adds the acoustic metadata at the time to the acoustic metadata up to the time before the time without dividing the acoustic metadata; An audio metadata processing device comprising:

2. The audio metadata processing device according to claim 1 , wherein the division determination section determines the moving direction of the audio object from a difference between a playback position of the audio metadata at the time and a playback position of the audio metadata up to the time before the time.

3. 3. The acoustic metadata processing device according to claim 1, wherein the division determination unit determines the movement direction of the audio object from a difference between a rate of change in the playback position of the acoustic metadata at the time and a rate of change in the playback position of the acoustic metadata up to the time prior to the time.

4. 4. The acoustic metadata processing device according to claim 1, wherein the division determination unit determines the movement direction of the audio object from the difference between the playback position of the acoustic metadata at the time and a predicted playback position at the time calculated from the playback position of the acoustic metadata up to the time before the time.

5. The audio metadata processing device according to claim 4 , wherein the division determination section determines the predicted playback position at the time by using a Lagrange's interpolation formula.

6. 6. The acoustic metadata processing device according to claim 1, wherein the acoustic metadata conversion unit divides the acoustic metadata into a time series when the playback position of an audio object changes discontinuously, and inserts acoustic metadata with a duration of 0 seconds or acoustic metadata whose initial value is the discontinuous point.

7. A program for causing a computer to function as the acoustic metadata processing device according to any one of claims 1 to 6.

Images