Audio metadata processing device and program

Abstract

To provide technique capable of adaptively changing the position of a voice object in a content space according to the positional relationship between a user and the voice object.SOLUTION: An acoustic metadata processing device according to one embodiment of the present invention includes a spatial masking determination unit that determines whether multiple audio objects exist within a range in which spatial masking is deemed to be occurring, starting from a specified position, on the basis of the coordinates of audio objects described in acoustic metadata, and an audio object coordinate calculation unit that, when the multiple audio objects exist within the range, changes the coordinate of at least one of the multiple audio objects in the acoustic metadata such that it falls outside the range, starting from the specified position, when the multiple audio objects exist within the range.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an audio metadata processing apparatus and program for changing the coordinates of an object described in audio metadata in object-based audio. [Background technology]

[0002] In object-based audio, information such as the position and gain of an audio object is described as audio metadata, and a renderer generates a playback signal based on the audio metadata.

[0003] Here, object-based audio is an audio method that records and transmits audio objects and audio metadata, and renders content in a format appropriate for the playback environment. Audio objects refer to the sound materials that make up the sound of the content, such as commentary, commentary, dialogue, music, background sounds, and sound effects. In this specification, when referring simply to an "object," the term "object" may be interpreted as meaning a video object in the content space.

[0004] The application of object-based audio to movies and broadcasting contents is being promoted. Furthermore, the expansion and development of object-based audio is also being promoted for 6DoF (Degrees of Freedom) contents such as Augmented Reality (AR) / Virtual Reality (VR) contents, which allow users to view from various positions and angles (see, for example, Non-Patent Document 1).

[0005] An example of applying object-based audio to AR / VR content is described in Non-Patent Document 2. Non-Patent Document 2 describes a system in which the playback level of an audio object varies according to the distance between the user and the object when the user moves freely within a specific range. It is expected that the user will feel immersed in the content by rendering audio signals according to the laws of physics.

[0006] However, depending on the positional relationship between the multiple objects and the user, the directions from which the audio signals of the multiple objects arrive may be close to each other.

[0007] 1A is a diagram showing an example of the positional relationship between two objects and a user viewed from directly above. As shown in FIG. 1A, two objects 1A and 2A exist in the content space, and the user moves from user position A to user position B as shown by the dotted line.

[0008] 1B is a diagram showing an example of the directions from which the sounds of two objects come from the user's viewpoint when the user is located at user position A in FIG. 1A. In this example, it is assumed that the two sound objects are the sounds of two people. When the user watches in this positional relationship, the directions from which the sounds of the two objects 1A and 2A come are separated.

[0009] Fig. 1C is a diagram showing an example of the directions from which the sounds of two objects come from the user's viewpoint when the user is located at user position B in Fig. 1A. When the user watches from this positional relationship, the sounds of the two objects 1A and 2A come to the user from approximately the same direction.

[0010] In this way, when the positions (arrival directions) of multiple audio objects are spatially close, a phenomenon called spatial masking occurs, in which some audio signals are masked by other audio signals and become difficult to hear (see, for example, non-patent documents 3 and 4).

[0011] As a method for eliminating this spatial masking, it has been confirmed that when spatial masking occurs between the audio signals of background sound and dialogue, a method of localizing the background sound and dialogue at different positions is effective (for example, see non-patent document 5). [Prior art documents] [Non-patent literature]

[0012] [Non-Patent Document 1] ISO / IEC 23090-4, “MPEG-I Immersive Audio” (under standardization) [Non-Patent Document 2] Yo Sasaki, “AR / VR Content Creation Using Object-Based Audio”, ITE Technical Report vol.45, no.19, 3DIT2021-18, 2021, p.15-20 [Non-Patent Document 3] K. Saberi, L. Dostal, T. Sadralodabai, V. Bull and DR Perrott: “Freefield release from masking”, J. Acoust. Soc. Am., 90, 3, pp.1355-1370 (1991) [Non-Patent Document 4] F. Santon, “Detection of a pure sound in the presence of masking noise and its dependence on the angle of incidence of the noise”, Acustica, 63, pp.222-228 (1987) [Non-Patent Document 5] Tomoyasu Komori, Toru Miyagi, Yasuhiro Oikawa, “A study on sound reproduction method using spatial masking release to make it easier for elderly people to hear”, Journal of the Institute of Image Information and Television Engineers, Vol. 71, No. 5, pp. J172 - J178, 2017 [Non-Patent Document 6] Masahiro Kimura, Hiroshi Kajii, Makoto Takahashi, Katsuyuki Yamamoto, “Ventriloquism effect in peripheral vision”, Transactions of the Virtual Reality Society of Japan, Vol. 4, No. 2, pp. 253 - 260, 1999 Summary of the Invention [Problem to be solved by the invention]

[0013] The technology described in non-patent document 5 assumes that audio objects are presented at fixed positions where spatial masking is unlikely to occur when the user is viewing in the center of a surround circle in channel-based audio, and targets audio objects that do not have a corresponding video object on the screen, such as program narration.

[0014] However, in VR / AR content, the user can freely change the position to watch, so the position where spatial masking occurs changes dynamically depending on the user's position. Therefore, it is not expected to apply the technology described in Non-Patent Document 5 to VR / AR content, or the technology described in Non-Patent Document 5 cannot be applied.

[0015] In view of the above, the present invention aims to provide a technology that can adaptively change the position of an audio object (i.e., change the coordinates of an audio object described in acoustic metadata) in a content space (e.g., a 6DoF content space) depending on the positional relationship between the user and the audio object. [Means for solving the problem]

[0016] An acoustic metadata processing device according to one embodiment of the present invention comprises a spatial masking determination unit that determines whether multiple audio objects exist within a range in which spatial masking is deemed to be occurring, starting from a specified position, based on the coordinates of audio objects described in acoustic metadata, and an audio object coordinate calculation unit that, if the multiple audio objects exist within the range, changes the coordinates of at least one of the multiple audio objects in the acoustic metadata so that it falls outside the range, starting from the specified position, if the multiple audio objects exist within the range. Effect of the Invention

[0017] According to one aspect of the present invention, the position of a voice object can be adaptively changed in a content space according to the positional relationship between the user and the voice object. [Brief description of the drawings]

[0018] [Figure 1A] A diagram showing an example of the positional relationship between two objects and the user as seen from directly above [Figure 1B] FIG. 1B is a diagram showing an example of the directions from which sounds of two objects come from the user's viewpoint when the user is located at user position A in FIG. 1A ; [Figure 1C] FIG. 1B is a diagram showing an example of the directions from which sounds of two objects come from the user's viewpoint when the user is located at user position B in FIG. 1A; [Figure 2A] FIG. 2 is a diagram showing an overview of processing performed by an acoustic metadata processing device according to an embodiment of the present invention; [Figure 2B] FIG. 13 is a diagram showing an example of a coordinate operation of an audio object by the audio metadata processing device according to the embodiment; [Figure 2C] FIG. 13 is a diagram showing another example of the coordinate operation of an audio object by the audio metadata processing device according to the embodiment; [Diagram 3] FIG. 2 is a block diagram showing an example of a hardware configuration of an audio metadata processing device according to the embodiment; [Figure 4] FIG. 2 is a block diagram showing an example of a functional configuration of an audio metadata processing device according to the embodiment; [Diagram 5] A flowchart showing an example of a process executed by the audio metadata processing device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] <Processing Overview> First, an overview of the processing of an acoustic metadata processing device according to one embodiment of the present invention will be described with reference to FIGS. 2A to 2C.

[0020] As shown in FIG. 2A, an audio metadata processing device 10, which is an example of an audio metadata processing device according to the present disclosure, receives as input audio metadata including coordinates of an audio object, an essentiality level, and user coordinates (user position).

[0021] 2A, the audio metadata processing device 10 determines an audio object to be subjected to coordinate manipulation based on an input, manipulates (changes or corrects) the coordinates (which may also be called coordinate values) of the determined audio object, and outputs audio metadata including the coordinates of the audio object updated after manipulation. This will be described in more detail below.

[0022] When there are multiple audio objects whose audio arrival directions are close to each other with the user (user coordinates or user position; specified position) as the origin (point of origin), that is, when it is deemed (determined) that spatial masking is occurring, the audio metadata processing device 10 changes the coordinates of one or more of these audio objects so that the arrival directions of these audio objects are different. Note that determining whether spatial masking is occurring in multiple audio objects is referred to as spatial masking determination.

[0023] Furthermore, the audio metadata processing device 10 may change the amount of manipulation of the coordinates according to the "degree of necessity" set for an audio object by, for example, a content provider (e.g., a content creator). Specifically, the audio metadata processing device 10 may reduce the amount of manipulation of the coordinates of an audio object as the degree of necessity of the audio object increases, and may increase the amount of manipulation of the coordinates of the audio object as the degree of necessity of the audio object decreases. Here, the "degree of necessity" is a parameter indicating how much the audio of the object is desired to be conveyed to the user in the content (in other words, a parameter indicating the importance of the audio object), and may take a value of, for example, 0 or more and 1 or less. For example, it is assumed that the degree of necessity of the lines of the main characters of the content is set high (e.g., the degree of necessity is 1), and the degree of necessity of the voices of passersby involved in creating the atmosphere of the content is set low (e.g., the degree of necessity is 0). The degree of necessity may be set in advance by the content provider, or may be dynamically set according to the viewing situation of the user. In this application, when the term "degree of necessity" is simply used, it refers to the absolute degree of necessity, and the (absolute) degree of necessity is represented by α.

[0024] Furthermore, the audio metadata processing device 10 may control so that the coordinates of the audio object do not change beyond a set maximum value, so that the coordinates of the audio object are manipulated within a range that does not cause discomfort in the perception of the direction of the video and audio images. For example, according to Non-Patent Document 6, if the video position and the audio image position of an object differ by 15 degrees or more, the user will perceive that the video and audio directions do not match (and therefore, there is a high possibility that discomfort will occur).

[0025] 2B is a diagram showing an example of the coordinate operation of an audio object by the audio metadata processing device according to the embodiment. In the example shown in FIG. 2B, a triangular region is a range in which spatial masking is considered to occur (referred to as a spatial masking range).

[0026] In the example shown in Fig. 2B, the essentiality α of audio object 1A is 1, and the essentiality α of audio object 2A is 0. For example, it is assumed that the content creator wants to make audio object 1A heard, and audio object 1A is an essential audio object for the content (e.g., the audio of the lead actor). On the other hand, it is assumed that the content creator does not want audio object 2A to be heard, and audio object 2A is an audio object of low importance for the content (e.g., the audio of an extra actor).

[0027] "β" in Fig. 2B represents the relative necessity within the spatial masking range. In the example shown in Fig. 2B, the relative necessity β of the audio object 1A is 1 / (0+1)=1, and the relative necessity β of the audio object 2A is 0 / (0+1)=0.

[0028] As in the example shown in FIG. 2B, when there is an audio object with a high degree of imperativeness (e.g., audio object 1A) and an audio object with a low degree of imperativeness (e.g., audio object 2A), the audio metadata processing device 10 largely moves only the audio object with a low degree of imperativeness. That is, the audio metadata processing device 10 largely manipulates only the coordinates of the audio object with a low degree of imperativeness, and outputs audio metadata having the coordinates of the updated audio object. In the example shown in FIG. 2B, the audio object 2A is moved to the top of the paper so that the arrival directions of the audio objects 1A and 2A are different, and the audio object 1A remains the same (unmoved). In this way, the audio object 1A with a high degree of imperativeness is not moved, i.e., its image position does not change, so there is little possibility that the impression of the audio object 1A will be damaged.

[0029] 2C is a diagram showing an example of the coordinate operation of an audio object by the audio metadata processing device according to the embodiment, in which the triangular region is also a spatial masking range.

[0030] In the example shown in FIG. 2C, the essentiality α of the voice object 1A is 1, and the essentiality α of the voice object 2A is 1.

[0031] "β" in Fig. 2C represents the relative degree of necessity within the range where spatial masking is occurring (determined). In the example shown in Fig. 2C, the relative degree of necessity β of the audio object 1A is 1 / (1+1)=0.5, and the relative degree of necessity β of the audio object 2A is 1 / (1+1)=0.5.

[0032] As in the example shown in Figure 2C, when there are multiple audio objects with high essentiality (e.g., audio objects 1A and 2A), the audio metadata processing device 10 moves each audio object with high essentiality by a small amount. That is, the audio metadata processing device 10 manipulates the coordinates of all audio objects with high essentiality by a small amount and outputs audio metadata having the coordinates of the updated audio objects. In the example shown in Figure 2C, audio object 1A is moved to the bottom of the paper and audio object 2A is moved to the top of the paper so that the directions from which the audio of audio objects 1A and 2A arrive are different.

[0033] Thus, in the example shown in Fig. 2B, only one audio object (i.e., audio object 2A) is moved, whereas in the example shown in Fig. 2C, both audio objects (i.e., audio objects 1A and 2A) are moved, i.e., at least one audio object present within the spatial masking range is moved out of the spatial masking range.

[0034] According to the above, it is possible to reduce spatial masking caused by the positional relationship between the user and the audio object, without causing discomfort in the perception of the direction of the video and audio images. Also, it is more likely that audio that is in line with the creative intent of the content provider (producer) is delivered to the user.

[0035] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. However, the scope of the invention is not limited to the illustrated examples. In the following description, the same reference numerals are used to designate components having the same functions and configurations, and the description thereof may be omitted.

[0036] <Configuration of Acoustic Metadata Processing Device> 3 is a block diagram showing an example of the hardware configuration of an audio metadata processing device according to this embodiment. The audio metadata processing device 10 may be realized by any one of a computer, a server, a PC, a tablet, a smartphone, or an information processing device. The audio metadata processing device 10 may be incorporated in a renderer, or may be separate from the renderer. The updated audio metadata output by the audio metadata processing device 10 may be provided from the audio metadata processing device 10 to the renderer.

[0037] 3, an acoustic metadata processing device 10 may include a storage device 11, a processing device 12, a user interface (UI) device 13, a communication device 14, and a bus 15. The storage device 11, the processing device 12, the UI device 13, and the communication device 14 may be connected to each other via the bus 15.

[0038] The programs or instructions for implementing the functions and processes described above and below in the audio metadata processing device 10 may be downloaded from some external device (e.g., a server) via a network, etc. Also, such programs or instructions may be provided from a removable storage medium such as a CD-ROM (Compact Disc Read Only Memory) or a flash memory.

[0039] Storage device 11 is realized by a random access memory (RAM), a flash memory, a hard disk drive, etc., and stores installed programs or instructions as well as files, data, etc. used in executing the programs or instructions. Storage device 11 may include a non-transitory storage medium.

[0040] The processing device 12 may be realized, for example, by a general-purpose processor or controller (circuit), or may be realized by a dedicated processor or controller (circuit). When realized by a general-purpose processor, the processing device 12 may be realized by one or more central processing units (CPUs), graphics processing units (GPUs), processing circuitry, etc., which may be composed of one or more processor cores. In this case, the processing device 12 executes the functions and processes of the acoustic metadata processing device 10 described above and below according to a program or instruction stored in the storage device 11, data such as parameters used to execute the program or instruction, etc. Such a program may be a program that causes the computer or processing device 12 to function as the acoustic metadata processing device 10 according to an embodiment of the present invention when executed by the computer or processing device 12. Furthermore, such a program may be a program that causes the computer or processing device 12 to execute an acoustic metadata processing method according to an embodiment of the present invention when executed by the computer or processing device 12.

[0041] The UI device 13 may be composed of input devices such as a keyboard, a mouse, a camera, a microphone, etc., output devices such as a display, a speaker, a headset, a printer, etc., and input / output devices such as a haptic device, a touch panel, etc., and realizes an interface between a user and the acoustic metadata processing device 10. For example, a user operates the acoustic metadata processing device 10 by operating a graphical user interface (GUI) displayed on a display or a touch panel using a keyboard, a mouse, etc.

[0042] The communication device 14 may be realized by various communication circuits that execute communication processes with external devices and communication networks such as the Internet, an intranet, a LAN, and a VPN.

[0043] The above-described hardware configuration of the acoustic metadata processing device 10 is merely an example, and the acoustic metadata processing device 10 according to the embodiment of the present invention may be realized by any other appropriate hardware configuration.

[0044] FIG. 4 is a block diagram showing an example of the functional configuration of an audio metadata processing device according to this embodiment.

[0045] 4, the audio metadata processing device 10 includes an acquisition unit 101, a spatial masking determination unit 102, and an audio object coordinate calculation unit 103. The acquisition unit 101, the spatial masking determination unit 102, and the audio object coordinate calculation unit 103 may be realized by the processing device 12 as hardware, software, or a combination of hardware and software.

[0046] The acquisition unit 101 acquires audio metadata, the degree of necessity of an audio object, and user coordinates together with video information (video signal) and audio information (audio signal), or without video information (video signal) and audio information (audio signal). The coordinates of the audio object are described in the audio metadata. The degree of necessity of the audio object may also be described in the audio metadata. For example, the audio metadata, the degree of necessity of an audio object, and user coordinates may be appropriately acquired from a content providing device such as a broadcast transmission device, a content recording medium, a camera, a head mounted display, or the like. The acquisition unit 101 outputs the acquired audio metadata, the degree of necessity of an audio object, and user coordinates to the spatial masking determination unit 102 and the audio object coordinate calculation unit 103.

[0047] The spatial masking determination unit 102 determines whether spatial masking is occurring (or is considered to be occurring) in an audio object in the content. In other words, the spatial masking determination unit 102 determines whether multiple audio objects exist within a range where spatial masking is considered to be occurring (spatial masking range) starting from the user position based on the coordinates of the audio object described in the audio metadata.

[0048] For example, the spatial masking determination unit 102 may determine, identify, or specify an object whose coordinates are to be changed by the audio object coordinate calculation unit 103 (referred to as a coordinate operation target) based on a preset angle (spatial masking angle) from the origin, which is the user position. The spatial masking angle refers to an angle in the azimuth direction and / or elevation angle direction of the position of the audio object where spatial masking occurs with respect to the audio object. Specifically, the spatial masking determination unit 102 may determine that multiple audio objects exist within the spatial masking range when multiple audio objects exist within a preset angle in at least one of the azimuth direction and the elevation direction from the origin. That is, the spatial masking determination unit 102 may determine that multiple audio objects exist within the spatial masking range when multiple audio objects exist within a preset angle in the azimuth direction from the origin, when multiple audio objects exist within a preset angle in the elevation direction from the origin, or when multiple audio objects exist within a preset angle in the azimuth direction and within a preset angle in the elevation direction from the origin.

[0049] The spatial masking angle may be set by, for example, a content provider. According to Non-Patent Document 5, the discrimination angle (discrimination threshold) of spatial masking also varies depending on the age of the user, so the spatial masking angle may additionally or alternatively be set by the renderer based on the age or hearing ability of the user (for example, according to an instruction from the user of the renderer), or may be dynamically set by the spatial masking determination unit 102 according to the positional relationship between the user position output by the acquisition unit 101 and the audio object.

[0050] The spatial masking determination unit 102 outputs to the audio object coordinate calculation unit 103 identification information (for example, an index) regarding the audio object (the audio object to be subjected to coordinate operation) for which it is determined that spatial masking occurs.

[0051] The spatial masking determination will be described in detail later.

[0052] In this application, "A outputs B to C" does not only mean "A directly outputs B to C" but may also mean the following. "A sends B to C." "A stores B in a memory area accessible by C."

[0053] The audio object coordinate calculation unit 103 receives as input the identification information of the audio object determined to be subjected to spatial masking output by the spatial masking determination unit 102. The audio object coordinate calculation unit 103 changes the coordinates of at least one audio object among the audio objects in the audio metadata determined to be subjected to spatial masking, which corresponds to the received identification information, so as to be outside the spatial masking range, with the user position as the origin.

[0054] The voice object coordinate calculation unit 103 may change the coordinates of at least one voice object without changing the distance from the user position to each voice object of the at least one voice object, thereby maintaining the volume of each voice object at the listening point.

[0055] More specifically, for example, the audio object coordinate calculation unit 103 calculates an operation vector (localization operation vector) of the coordinates of the audio object based on the relative position of the audio object corresponding to the identification information regarding the audio object output by the audio object coordinate calculation unit 103, with the user coordinates (user position) output by the acquisition unit 101 as the origin, the essentiality of the audio object output by the acquisition unit 101, and the maximum value of the coordinate operation amount (also called the maximum operation amount, for example, 15 degrees).

[0056] The maximum amount of operation may be set, for example, by a content provider, may be set by a renderer, or may be fixed. By setting the maximum amount of operation to 15 degrees or less, it is highly likely that no discomfort will be caused in the perception of the direction of the video and audio images.

[0057] The audio object coordinate calculation unit 103 changes the coordinates of the audio object by moving the coordinates of the audio object output by the acquisition unit 101 by the calculated operation vector. In this way, the audio object coordinate calculation unit 103 updates the coordinates of the audio object, i.e., the audio metadata.

[0058] By calculating the operation vector and the position of the audio object as described above, it is possible to provide a directional difference between the audio objects that is larger than the directional difference between the images of the object (e.g., a person). As a result, it is possible to reduce spatial masking caused by the positional relationship between the user and the audio object, within a range that does not cause discomfort in the perception of the direction of the image and the sound image.

[0059] The calculation of the operation vector of the coordinates and the calculation of the coordinates of the voice object will be described in detail later.

[0060] <Processing details> The processes performed by the spatial masking determination unit 102 and the audio object coordinate calculation unit 103 will now be described in detail.

[0061] First, let V represent the 6DoF content space (V∈R 3 ), and let O denote the set of indices of audio objects present in the 6DoF content space V, where O = {1, 2, , i, , N}.

[0062] Let t be the representative time of the time window for a certain rendering, and from now on, we will explain the coordinate operation algorithm in the time window including time t. Let the position of the voice object i∈Ο be a i If (t), then ai (t) is expressed in polar coordinate format by the following equation (1), where φ i (t), θ i (t) and d i (t) indicates the azimuth angle, elevation angle, and distance, respectively.

number

[0063] Any voice object i has a parameter, which is the degree of necessity. i Let (t)∈[0,1]. As mentioned above, α i When (t)=1, the audio object i indicates that it is a sound that the user must hear, whereas α i When (t)=0, it indicates that the voice object i is a "sound that does not necessarily need to be heard by the user." However, the degree of necessity does not necessarily need to be given. In that case, all voice objects i are treated as the same, and α i =0.

[0064] [Spatial masking assessment] An exemplary spatial masking determination performed by the spatial masking determination unit 102 will now be described in detail.

[0065] For the sound object i, the azimuth and elevation angles of the sound image position where spatial masking occurs (spatial masking angles) are defined as Δφ i (t) and Δθ i (t). This spatial masking angle may be, for example, a constant (i.e., Δφ i (t) = Δφ i (Constant), Δθ i (t) = Δθ i (constant)).

[0066] In addition, since it is known that the discrimination threshold (discrimination angle) of spatial masking varies depending on the age of the user as described above (see Non-Patent Document 5), the value of this constant may be changed depending on the age and hearing ability of the user. Furthermore, as described above, the spatial masking angle may be dynamically given depending on the positional relationship between the user and the audio object. Here, the spatial masking range Φ i (t) (azimuthal direction) and Θ i (t) (elevation angle direction) is given by the following equations (2) and (3).

number

[0067] Φ i (t) and Θ i The direct product of (t) and Ω i (t)=Φ i (t)×Θ i Let us denote it as (t) and Ω i (t) is called the neighborhood of audio object i at time t. i When an object other than the audio object i exists in (t), it is determined that spatial masking is occurring. i When an audio object other than audio object i exists in (t), it is determined that spatial masking has occurred to audio object i and audio objects other than audio object i, i.e., audio object i and audio objects other than audio object i exist within the spatial masking range.

[0068] Here, given an audio object i and its neighborhood Ω i Let W be the set of indices of the voice objects belonging to (t). i Let (t).

[0069] The spatial masking determination unit 102 performs the above process for an arbitrary audio object i∈Ο to perform spatial masking determination. Note that, in order to reduce the amount of calculation, the spatial masking determination unit 102 performs the spatial masking determination process for, for example, α iThe search range may be limited to voice objects that satisfy (t)>0.5. Note that the value "0.5" is merely an example and can be changed as appropriate.

[0070] [Calculation of coordinate operation vector] The following describes in detail an exemplary coordinate operation vector calculation performed by the audio object coordinate calculation unit 103. The coordinate operation may be called a localization operation.

[0071] The voice object coordinate calculation unit 103 assumes that voice object i is a voice object to be subjected to coordinate operation, and voice object j∈W i Calculate the coordinate operation vector at (t), where voice object j∈W i All the degrees of necessity of (t) are α j When (t)=0, all of the voice objects that are the subject of coordinate manipulation "do not need to be heard," so no coordinate manipulation is performed.

[0072] First, the voice object coordinate calculation unit 103 calculates W i Calculate the relative necessity of any voice object i belonging to (t). Let the relative necessity of voice object i be β i (t), then β i (t) is calculated using equation (4).

number

[0073] At this time, the distance of the audio object may be set to be unchanged so that the volume of the sound of each audio object at the listening point is maintained. At this time, the audio object coordinate calculation unit 103 calculates the coordinate operation vector represented by the formula (5)

number

[0074] Here, the audio object coordinate calculation unit 103 calculates, by equations (6) and (7),

number

number

[0075] [Calculating the position of the audio object] The following describes in detail an exemplary calculation of an audio object position by the audio object coordinate calculation unit 103. Note that the audio object position may be called a sound image vector or a position vector.

[0076] The audio object coordinate calculation unit 103 calculates the position of the audio object i whose localization has been manipulated by using the coordinate manipulation vector (5) calculated above, according to equation (8).

number

number

[0077] Alternatively, the voice object coordinate calculation unit 103 calculates the position of the voice object i by using the formula (9) while taking into consideration the continuity from the coordinate position at the previous time t-1.

number

number

[0078] [Example of processing application] Next, an example of applying the above-mentioned process will be described. In the following examples 1 and 2, the coordinates of the user origin of two voice objects (voice objects 1A and 2A) at a certain time t are a1(t)=(30.0,0.0,1.0) and a2(t)=(32.0,0.0,3.0), and the above-mentioned process is applied as follows. The parameter values ​​are Δφ for i={1,2}. i (t) = Δθ i (t) = Δθ i = 3.0, η i In this case, a1(t)-a2(t)=(-2.0, 0.0, -2.0), and the elevation angle difference |-2.0| is smaller than the preset spatial masking angle Δθ1(t)=Δθ1=3.0, so audio object 1A (and object 2A) are subject to coordinate operation.

[0079] Example 1 (when necessity level α1(t)=1, α2(t)=1) Example 1 corresponds to the example shown in FIG. 2C. First, the relative degree of necessity β i (t) is calculated as follows: β1(t)=1.0 / (1.0+1.0)=0.5, β2(t)=1.0 / (1.0+1.0)=0.5

[0080] In this case, the center of gravity vector with respect to the angle is given by equation (7):

number

number

[0081] Therefore, the positions (coordinates) of the voice objects 1A and 2A are calculated (operated or updated) as follows.

number

[0082] Looking at these results, the amount of manipulation for all audio objects was 15 degrees or less, and all audio objects were manipulated within the range in which the video and audio directions were perceived to match, as shown in Non-Patent Document 6. In addition, the angular difference between audio object 1A and audio object 2A after the coordinate manipulation was 17 degrees, which is larger than the angle of 15 degrees at which it was confirmed that spatial masking is reduced, as shown in Non-Patent Document 5.

[0083] Example 2 (when necessity level α1(t)=1, α2(t)=0) Example 2 corresponds to the example shown in FIG. 2B. First, the relative degree of necessity β i (t) is calculated as follows: β1(t) = 1.0 / (1.0+0.0) = 1.0, β2(t) = 0.0 / (1.0+0.0) = 0.0

[0084] In this case, the center of gravity vector with respect to the angle is given by equation (7):

number

number

[0085] Therefore, the positions (coordinates) of the voice objects 1A and 2A are calculated (operated or updated) as follows.

number

[0086] Looking at these results, no coordinate manipulation was performed on the highly essential audio object 1A, and only the less essential audio object 2A was manipulated. The amount of manipulation for all audio objects was 15 degrees or less, and all audio objects were manipulated within the range in which the video and audio directions shown in Non-Patent Document 6 are perceived to match. In addition, the angle difference between audio object 1A and audio object 2A after the coordinate manipulation was 17 degrees, which is larger than the angle of 15 degrees at which it was confirmed that spatial masking is reduced as shown in Non-Patent Document 5.

[0087] <Operation of the Acoustic Metadata Processing Device> Next, an example of the operation of the audio metadata processing device according to this embodiment will be described with reference to Fig. 5. In the flowchart shown in Fig. 5, each block corresponds to a process for each time window related to rendering.

[0088] In step S11, the audio metadata processing device 10 acquires the coordinates of the audio object, the essentiality of the audio object, and the user coordinates described in the audio metadata. Step S11 is executed by the acquisition unit 101.

[0089] In step S12, the audio metadata processing device 10 determines whether or not another audio object exists within a range of a preset spatial masking angle from a user position that is an origin (spatial masking range) for a certain audio object. If another audio object exists within the spatial masking range, the audio metadata processing device 10 determines that spatial masking has occurred for the certain audio object and the other audio object, and determines the certain audio object (and the other audio object) as a coordinate operation target. Step S12 may be executed by the spatial masking determination unit 102.

[0090] If it is determined that there is no other audio object within the spatial masking range (step S12; NO), the flow proceeds to step S14. Then, in step S14, the audio metadata processing device 10 calculates the position (coordinates) of the audio object, overwrites the audio metadata with the calculated coordinates of the audio object, and the flow ends. Step S14 may be executed by the audio object coordinate calculation unit 103. Note that in this case, since the position of the audio object is not changed, step S14 does not need to be executed.

[0091] If it is determined that another audio object exists within the spatial masking range (step S12; YES), in step S13, the audio metadata processing device 10 calculates a coordinate operation vector for the certain audio object and the other audio object based on the relative position with the user position as the origin, the relative necessity within the spatial masking range, the maximum amount of operation, etc., as described above. Step S13 may be executed by the audio object coordinate calculation unit 103.

[0092] Then, in step S14, the audio metadata processing device 10 calculates the positions (coordinates) of the certain audio object and the other audio object using the coordinate operation vector calculated in step S13, overwrites the audio metadata with the calculated coordinates of the audio objects, and the flow ends. Step S14 may be executed by the audio object coordinate calculation unit 103.

[0093] The flow shown in FIG. 5 may be executed repeatedly and continuously until a scene change occurs in the content.

[0094] <Effects of the embodiment> As described above, the audio metadata processing device 10 according to the present embodiment determines whether multiple audio objects exist within a spatial masking range with the user position as the origin, and if multiple audio objects exist within the spatial masking range, changes the coordinates of at least one of the multiple audio objects existing within the spatial masking range so as to move the audio object out of the spatial masking range with the user position as the origin. This makes it possible to adaptively change the position of the audio object in the content space according to the positional relationship between the user and the audio object, and to reduce spatial masking caused by the positional relationship between the user and the audio object.

[0095] <Modifications of the embodiment> In the above embodiment, an example in which processing is applied to two audio objects has been mainly described, but as described in <Processing Details>, the above-described processing may also be similarly applied to three or more audio objects.

[0096] The functional units described and illustrated in the above embodiments may be divided into multiple functional units, or two or more functional units may be integrated into one functional unit. Furthermore, the names of the functional units are merely examples and may be changed as appropriate.

[0097] The steps in the flowcharts described and illustrated in the above embodiments may be executed in a different order or in parallel as appropriate.

[0098] <Summary of the embodiment> An acoustic metadata processing device according to one embodiment of the present invention comprises a spatial masking determination unit that determines whether multiple audio objects exist within a range in which spatial masking is deemed to be occurring, starting from a specified position, based on the coordinates of audio objects described in acoustic metadata, and an audio object coordinate calculation unit that, if the multiple audio objects exist within the range, changes the coordinates of at least one of the multiple audio objects in the acoustic metadata so that it falls outside the range, starting from the specified position, if the multiple audio objects exist within the range.

[0099] In one example, the spatial masking determination unit determines that the plurality of sound objects are present within the range when the plurality of sound objects are present within a preset angle from the origin.

[0100] In one example, the audio object coordinate calculation unit modifies coordinates of the at least one audio object in the acoustic metadata without changing a distance from the origin to each audio object of the at least one audio object.

[0101] In one example, the audio object coordinate calculation unit modifies the coordinates of the at least one audio object in the acoustic metadata so that the angle between a vector from the origin to the coordinates of each audio object of the at least one audio object that has been modified and a vector from the origin to the coordinates of the audio object before the modification described in the acoustic metadata is less than a predetermined angle.

[0102] In one example, the audio object coordinate calculation unit calculates relative parameter values ​​corresponding to each of the multiple audio objects from parameter values ​​corresponding to each of the multiple audio objects, calculates coordinate operation vectors for each of the multiple audio objects based on the relative positions of each of the multiple audio objects with the specified position as the origin and the relative parameter values, and changes the coordinates of the at least one audio object in the acoustic metadata based on the coordinate operation vector of the at least one audio object.

[0103] In one example, the voice object coordinate calculation unit calculates the coordinate operation vector of each of the plurality of voice objects such that the magnitude of the coordinate operation vector decreases as the relative parameter value increases.

[0104] A program according to one embodiment of the present invention is a program for causing a computer to execute a process including: determining whether multiple audio objects exist within a range in which spatial masking is deemed to be occurring, starting from a specified position, based on the coordinates of audio objects described in acoustic metadata; and if the multiple audio objects exist within the range, changing the coordinates of at least one of the multiple audio objects in the acoustic metadata so that it falls outside the range, starting from the specified position.

[0105] Although the embodiment of the present invention has been described above with reference to the drawings, the present invention is not limited to such examples. It is clear that a person skilled in the art can come up with various modified or altered examples within the scope of the claims. It is understood that such modified or altered examples also belong to the technical scope of the present invention. Furthermore, each component or each item in the embodiment (including modified examples) may be arbitrarily combined within the scope of the present invention.

[0106] The present invention can be realized in software, hardware, or software in conjunction with hardware. [Industrial Applicability]

[0107] One aspect of the present invention is useful as an audio metadata processing device that can adaptively change the coordinates of an audio object described in audio metadata according to the positional relationship between a user and the audio object. [Explanation of symbols]

[0108] 10. Acoustic Metadata Processing Device 11 Storage device 12 Processing equipment 13 User Interface (UI) Devices 14 Communication equipment 101 Acquisition Department 102 Spatial masking judgment unit 103 Audio object coordinate calculation unit

Claims

1. a spatial masking determination unit that determines whether a plurality of audio objects are present within a range in which spatial masking is considered to occur, starting from a specified position based on the coordinates of the audio objects described in the audio metadata; an audio object coordinate calculation unit that changes the coordinates of at least one of the audio objects in the acoustic metadata so that the audio object is outside the range, when the audio objects are within the range, with the specified position as an origin; An audio metadata processing device comprising:

2. the spatial masking determination unit determines that the plurality of sound objects are present within the range when the plurality of sound objects are present within a preset angle from the origin; 2. An apparatus for processing acoustic metadata according to claim 1.

3. the audio object coordinate calculation unit changes the coordinates of the at least one audio object in the acoustic metadata without changing a distance from the origin to each audio object of the at least one audio object.

2. An apparatus for processing acoustic metadata according to claim 1.

4. the audio object coordinate calculation unit changes the coordinates of the at least one audio object in the audio metadata so that an angle between a vector from the origin to the coordinates of each audio object of the at least one audio object that has been changed and a vector from the origin to the coordinates of the audio object before the change described in the audio metadata is equal to or smaller than a predetermined angle.

2. An apparatus for processing acoustic metadata according to claim 1.

5. the voice object coordinate calculation unit calculates relative parameter values ​​corresponding to each of the multiple voice objects from parameter values ​​indicating the importance of each of the multiple voice objects, calculates coordinate operation vectors for each of the multiple voice objects based on the relative parameter values ​​and the relative positions of each of the multiple voice objects with the specified position as the origin, and changes the coordinates of the at least one voice object in the acoustic metadata based on the coordinate operation vector of the at least one voice object.

2. An apparatus for processing acoustic metadata according to claim 1.

6. the voice object coordinate calculation unit calculates the coordinate operation vectors of the plurality of voice objects such that the magnitude of the coordinate operation vector becomes smaller as the relative parameter value becomes larger; 6. An acoustic metadata processing apparatus according to claim 5.

7. determining whether a plurality of audio objects are present within a range in which spatial masking is considered to occur, starting from a specified position based on the coordinates of the audio objects described in the acoustic metadata; If the plurality of audio objects are present within the range, changing the coordinates of at least one of the plurality of audio objects in the acoustic metadata so as to move out of the range with the specified position as an origin; A program for causing a computer to execute a process including the steps of:

Images