Reverberation control apparatus and program

Abstract

To provide a reverberation control device and a program capable of reducing difficulty in hearing the sound of a desired object due to the influence of reverberation sound.SOLUTION: A reverberation control device includes a metadata analysis unit that analyzes metadata and specifies an identifier of an object to be controlled, a reverberation parameter calculation unit that calculates a gain of late reverberation of the object on the basis of the identifier of the object and a longest distance of an area in which a clarity of the object is secured, and a rendering unit that calculates an amplitude gain coefficient for adjusting an amplitude of the late reverberation in accordance with a distance between a user and the object and the gain of the late reverberation and multiplies the late reverberation of the object by the amplitude gain coefficient to add reverberation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a reverberation control device and a program, and more particularly to a reverberation control device and a program for controlling the reverberation of a specific object. [Background technology]

[0002] In recent years, various contents have been researched in the fields of AR (Augmented Reality) and VR (Virtual Reality), and efforts are being made to put 6DoF (Degree of Freedom) contents into practical use.

[0003] In a 6DoF content space, after putting on the VR goggles, the user can not only "shake their head left and right, front and back, or turn their head to look around," but also "move forward and backward, up and down, left and right" by walking, etc. In other words, unlike 3DoF VR, where users could only move their head, in 6DoF VR the content is linked to the user's physical movements. With 6DoF content, which users can view from various positions in the content space, it is expected that sound will be reproduced according to the positional relationship between the sound source and the user.

[0004] In conventional 3DoF content production, a method for designing the spatial impression of content is to convolve the audio signal with the room's impulse response. Convolution of impulse responses can achieve a spatial impression similar to that of a real acoustic space, but it has the disadvantage of requiring a large computational load. In contrast, 6DoF content production for games and other applications uses a computationally inexpensive feedback delay network (FDN) to characterize the spatial impression for real-time acoustic processing. In 6DoF content production, rather than simulating all acoustic events, attempts have been made to geometrically calculate the early reflections of reverberant sounds and use the FDN-generated reverberation for the later portion of the reverberant sounds. For example, because early reflections vary depending on the relative positions of the sound source and the user, they are calculated geometrically based on the positional information of both. On the other hand, assuming a diffuse sound field, late reverberation is uniform regardless of the user's position. Therefore, generating reverberant sounds using this method requires updating only the early reflections according to the user's position, which can reduce computational costs compared to convolution of impulse responses.

[0005] In 6DoF content, it is expected that sound will be reproduced according to the relative positions of the sound source and the user, so a system using object-based audio is suitable, in which the position of the sound source is described as metadata and the renderer generates the reproduced sound using the sound material and metadata. As metadata related to spatial impressions, room size, direct-to-area ratio, reverberation time, etc. have been considered (Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Takehiro Sugimoto, "Requirements for non-sound source metadata used in sound space design of 6DoF audio content", Proceedings of the 2024 Spring Meeting of the Acoustical Society of Japan, 2-9-5, pp.1187-1188, (2024) [Non-patent document 2] Recommendation ITU-R BS.2076-2 “Audio Definition Model”,(2019) [Non-patent document 3] Kuttruff H., “Room Acoustics”, 4th ed. (Elsevier Science Publishers Ltd., Lindin), (1991) [Non-patent document 4] Hiroshi Onaga, "Proposal of a physical index corresponding to speech intelligibility in reverberant sound fields", Journal of the Acoustical Society of Japan, Vol. 66, No. 3, pp. 97-104, (2010) Summary of the Invention [Problem to be solved by the invention]

[0007] As mentioned above, conventional reverberation control devices assume a diffuse sound field when creating 6DoF content. Therefore, as shown in Figure 3A, as the user moves away from the sound source, the direct sound becomes quieter due to distance attenuation, but the reverberant sound (late reverberation) is uniform regardless of position, so it is expected that the direct sound will be buried in the reverberant sound, reducing its clarity and making it difficult to hear.

[0008] On the other hand, from the viewpoint of content creation, producers may want to go beyond conventional acoustic rules and make important objects (such as the words of the main character) more audible than other objects.

[0009] Therefore, in consideration of the above-mentioned problems, an object of the present invention is to provide a reverberation control device and program that can reduce the difficulty in hearing the sound of a desired object due to the influence of reverberation sounds. [Means for solving the problem]

[0010] In order to solve the above problems, the reverberation control device according to the present invention comprises: (1) A reverberation control device comprising: a metadata analysis unit that analyzes metadata and identifies an identifier of an object to be controlled; a reverberation parameter calculation unit that calculates a gain of a rear reverberation of the object based on the identifier of the object and the longest distance of an area where clarity of the object is ensured; and a rendering unit that calculates an amplitude gain coefficient that adjusts the amplitude of the rear reverberation according to the distance between a user and the object and the gain of the rear reverberation, and multiplies the amplitude gain coefficient by the rear reverberation of the object to add reverberation.

[0011] (2) In the reverberation control device of (1) above, it is preferable that the rendering unit calculates the amplitude gain coefficient by interpolation based on the distance d between the user and the object, and multiplies the amplitude gain coefficient by the rear reverberation to add reverberation, so that the energy ratio of direct sound to rear reverberation specified in the metadata is satisfied when the user is in the same location as the object, and the gain of rear reverberation is satisfied at the furthest distance in the area where clarity is ensured.

[0012] (3) In the reverberation control device of (1) or (2) above, it is preferable that the reverberation parameter calculation unit further calculates the clarity of the object at the longest distance in the area, and when the clarity is less than a predetermined threshold, calculates the gain for adjusting the rear reverberation so that the clarity is equal to or greater than the threshold, and when the clarity is equal to or greater than the predetermined threshold, sets the gain of the rear reverberation to 1 and outputs it to the rendering unit.

[0013] (4) In any of the reverberation control devices (1) to (3) above, it is preferable that the rendering unit adjusts the decay curve of the late reverberation by adding a first FDN having a specified reverberation time and a second FDN having a reverberation time shorter than the specified reverberation time at a predetermined ratio, and adds reverberation based on the adjusted decay curve of the late reverberation.

[0014] (5) It is preferable that any of the reverberation control devices described above in (1) to (4) further calculates gains for adjusting the rear reverberation of each of the first FDN and the second FDN at the longest distance of the area where the clarity of the object is ensured, based on the gain of the rear reverberation and the predetermined ratio.

[0015] In order to solve the above problems, the program according to the present invention comprises: (6) A program that causes a computer to function as any one of the reverberation control devices (1) to (5) above. [Effects of the Invention]

[0016] The reverberation control device and program of the present invention can reduce the difficulty in hearing the sound of a particular object that a producer wants viewers to hear due to the influence of reverberation. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a block diagram of a reverberation control device according to an embodiment of the present invention. [Figure 2] 1 is a flowchart illustrating an example of processing performed by a reverberation control device according to the present invention. [Figure 3A] FIG. 1 is a conceptual diagram of the relationship between direct sound and reverberation sound (late reverberation) in a conventional reverberation control device. [Figure 3B] 1 is a conceptual diagram showing the relationship between direct sound and reverberation sound (late reverberation) in the reverberation control device of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In each drawing, the same or corresponding parts are designated by the same reference numerals. In the description of each embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate.

[0019] 1 is a block diagram of a reverberation control device 1 according to an embodiment of the present invention. The reverberation control device 1 includes a metadata analysis unit 10, a reverberation parameter calculation unit 20, and a rendering unit 30.

[0020] 2 is an example of a flowchart of the processing performed by the reverberation control device 1 of the present invention. First, an overview of the processing performed by each block of the reverberation control device 1 will be described based on the flowchart of FIG.

[0021] Step S1: The metadata analysis unit 10 determines whether the object is to be controlled. Specifically, the metadata analysis unit 10 analyzes the metadata of the content, and identifies the object that the creator wants to hear (the object to be controlled) and the area that the creator wants to hear. The area that the creator wants to hear is an area where clarity of the object is ensured. If the object is to be controlled (Yes), proceed to step S2. If the object is not to be controlled (No), proceed to step S5.

[0022] Step S2: The reverberation parameter calculation unit 20 calculates the clarity of the object to be controlled at the longest distance (the farthest position) in the area where the object is to be heard.

[0023] Step S3: The reverberation parameter calculation unit 20 determines whether the clarity of the object is less than a predetermined threshold. If the clarity of the object at the longest distance is less than the predetermined threshold (Yes), the process proceeds to step S4. If the clarity is equal to or greater than the predetermined threshold (No), the process proceeds to step S5.

[0024] Step S4: The reverberation parameter calculation unit 20 calculates a gain for adjusting the energy of the late reverberation in order to maintain clarity, and then outputs the calculated gain to the rendering unit 30.

[0025] Step S5: The rendering unit 30 performs rendering of the object. The rendering unit 30 adjusts and adds reverberation to the object that the creator wants to hear (the object to be controlled) using the calculated late reverberation gain. For other objects, reverberation is added by normal rendering.

[0026] (First embodiment) The reverberation control by the reverberation control device 1 of the first embodiment will be described in detail, focusing on the processing of each block.

[0027] <Metadata Analysis Division> The metadata analysis unit 10 analyzes the metadata described in the content and identifies the object to be controlled. Here, the object to be controlled, whose clarity is to be maintained, is, for example, a human voice or an object of high importance. According to Non-Patent Document 2, for example, the metadata includes an element that indicates whether the object is a dialogue or not by using 1 or 0. There is also an element that indicates importance by using a numerical value from 0 to 10. By analyzing these elements, a dialogue object or an object of high importance can be identified and targeted for reverberation control (step S1). In this embodiment, the target object may be identified by setting a new flag or the like instead of using the existing element. The metadata analysis unit 10 outputs the object identifier (ID) described in the metadata for the object to be controlled to the reverberation parameter calculation unit 20.

[0028] Furthermore, the metadata analysis unit 10 analyzes the metadata described in the content and specifies the area where the object is to be played. Existing metadata (Non-Patent Document 2) does not have a mechanism for indicating the area where the object is to be played, but in metadata compatible with 6DoF, it is assumed that the area where the object is to be played for each object is described in the ranges of X, Y, and Z, with the object at the center. The metadata analysis unit 10 determines the longest distance (the farthest distance from the object) d maxand outputs it to the reverberation parameter calculation unit 20. If the above range is not set in the metadata, a predetermined distance (for example, 3 m) may be set in advance on the renderer side as the default value for the maximum distance.

[0029] Furthermore, in the metadata compatible with 6DoF, it is desirable to describe the energy ratio DR between direct sound and late reverberation and the reverberation time for each object as metadata related to spatial impression. It is desirable that the metadata analysis unit 10 outputs this energy ratio DR between direct sound and late reverberation and the reverberation time to the rendering unit 30.

[0030] <Reverberation parameter calculation section> The reverberation parameter calculation unit 20 calculates the longest distance d of the area where the object is to be heard, which is output from the metadata analysis unit 10 (or set as a default value). max In step S2, the intelligibility is calculated from the impulse response of the reverberation system based on metadata such as the reverberation time and the energy ratio DR. The metadata for the reverberation time and the energy ratio DR may be set for each content and for each space (position) within the content. While there are several indices of intelligibility, here we use D50 (the ratio of the energy up to 50 ms after the direct sound to the total energy) (Non-Patent Document 3). Since it is considered that if the energy up to 50 ms is 40% or more of the total energy, word intelligibility is high and intelligibility is sufficient (Non-Patent Document 4), in this embodiment, "D50 of 40%" is used as the intelligibility threshold. Note that this intelligibility standard is merely an example, and other indices such as C80 (the ratio of the energy arriving up to 80 ms after the direct sound arrives to the energy thereafter: a parameter related to intelligibility for music) or other indices may also be used. The threshold may be set to 50% or even another value instead of 40%.

[0031] The reverberation parameter calculation unit 20 calculates the longest distance d between the areas where the object is to be heard. maxIn step S3, it is determined whether the clarity is below a threshold. If the energy up to 50 ms of the clarity index is less than 40% of the total energy, a gain is calculated to lower the level of late reverberation. This gain is later used in the rendering unit 30 to control late reverberation to improve the clarity of the object. If D50 is 40% or more, the clarity is considered sufficient as is, so rendering is performed in the rendering unit 30 without any control (gain, described below, is set to 1).

[0032] Here, the total energy of the sound (impulse response) is E all If the energy up to 50ms is E1 and the energy of the late reverberation after 50ms is E2, then E1+E2=E all The maximum distance of the area is d max In step S4, the gain value that satisfies equation (1) is calculated as the gain for adjusting the late reverberation energy (hereinafter simply referred to as "late reverberation gain"). Note that this gain is a coefficient for reducing the late reverberation energy when the intelligibility is below a threshold, so gain≦1. When the intelligibility is above the threshold, adjustment of the late reverberation is not necessary, so gain=1 is used.

[0033]

number

[0034] The reverberation parameter calculation unit 20 outputs to the rendering unit 30 the identifier (ID) of the object to be controlled and the gain of the late reverberation of the object.

[0035] <Rendering section> The rendering unit 30 adjusts and adds reverberation to the object corresponding to the identifier (ID) of the object to be controlled using the gain of late reverberation obtained from the reverberation parameter calculation unit 20. For other objects, reverberation is added by normal rendering.

[0036] When the user is in the same location as the object, reverberation is added so that the energy ratio DR between the direct sound and the late reverberation defined in the metadata is satisfied. When the user moves away from the object, the square root of the value calculated according to equation (2) is used as a coefficient (hereinafter referred to as the "amplitude gain coefficient") for adjusting the late reverberation (amplitude) at the distance d between the user and the object. The rendering unit 30 adds reverberation by multiplying this amplitude gain coefficient by the late reverberation (amplitude).

[0037]

number

[0038] FIG. 3B is a conceptual diagram of the relationship between direct sound and reverberation sound (late reverberation) produced by the reverberation control device 1 of this embodiment. max In the audio playback mode (up to 1000 kHz), the level of late reverberation is reduced according to the user's distance from the object. This improves clarity and reduces the effect of reverberation on the sounds of specific objects that the producer wants the viewer to hear.

[0039] In this embodiment, the coefficient to be multiplied by the late reverberation is calculated by linear interpolation according to the distance d, but equation (2) is merely an example. At a certain distance from the object, for example, 1 m, no reverberation control is performed. max Alternatively, interpolation may be performed using other functions.

[0040] (Second embodiment) The reverberation control by the reverberation control device 1 of the second embodiment will be described, focusing on the differences from the first embodiment.

[0041] The reverberation control device of the second embodiment of the present invention improves the intelligibility of an object by adjusting the decay curve of the late reverberation. In the second embodiment, an FDN with a shorter reverberation time than the value specified in the metadata is used, and the initial decay amount of the late reverberation is increased by combining multiple FDNs. This processing can be performed, for example, by the rendering unit 30.

[0042] FDN adds a signal obtained by multiplying the input signal m samples earlier by an attenuation coefficient a to the resulting signal and outputs the result. For simplicity, m is set to 1 here, but in implementation, m may be set to another value, such as a prime number.

[0043] Reverberation time refers to the time it takes for the sound energy density to decay to one millionth of its steady state after the sound is stopped, and is expressed as rt60 (Reverberation Time 60). In FDN, when a certain reverberation time rt60 (seconds) is given, the decay coefficient a can be calculated to satisfy equation (3). Here, f s is the sampling frequency.

[0044]

number

[0045] Here, by making the reverberation decay curve steeper and increasing the amount of initial decay, it is possible to increase the value of clarity D50 and improve clarity.

[0046] We will explain an example of improving intelligibility by adjusting the decay curve of late reverberation using two FDNs.

[0047] For example, the first FDN (first FDN) uses the FDN with the reverberation time rt60 (attenuation coefficient a) originally specified in the metadata. The second FDN (second FDN) calculates the attenuation coefficient a2 using equation (4) so ​​that the sound energy density attenuates to one millionth of the steady state at half the reverberation time (rt60 / 2).

[0048]

number

[0049] These two FDNs are weighted and added together. For example, the weighting ratio between the first FDN and the second FDN is set to 3:7. The energy of the late reverberation is controlled so that the specified intelligibility is maintained as a whole for the two added FDNs.

[0050] To maintain a certain level of intelligibility (e.g., D50 of 40% or more), the late reverberation energy after 50 ms must be distributed over the longest distance d from the desired listening area, as calculated by equation (1). max In this embodiment, this relationship is expressed by equation (5).

[0051]

number

[0052] In equation (5), E2 is the late reverberation energy after 50 ms of the first FDN with the same attenuation coefficient a as in the first embodiment. 2-2 is the energy of the late reverberation after 50 ms by the second FDN with attenuation coefficient a2. The weighting coefficient k can be calculated from equation (5).

[0053] Based on the weighting coefficient k, the longest distance d of the area max In the above equation, the value of k is used as a gain for adjusting the energy of the late reverberation of the first FDN, and the value of 7k / 3 is used as a gain for adjusting the energy of the late reverberation of the second FDN.

[0054] Thereafter, as in the first embodiment, an amplitude gain coefficient for adjusting the late reverberation (amplitude) at the distance d between the user and the object is calculated for each FDN based on equation (2), and the late reverberation (amplitude) of each FDN is multiplied by this amplitude gain coefficient to add reverberation.

[0055] In this embodiment, the intelligibility D50 is improved by combining multiple FDNs to increase the initial attenuation of the late reverberation. max In this case, the late reverberation energy (E2, E 2-2 ) by the gain, the need to adjust the late reverberation becomes less, or the adjustment amount becomes smaller. Therefore, making the attenuation curve steeper can contribute to clarity.

[0056] In this embodiment, the attenuation coefficient of the second FDN is calculated assuming a reverberation time of half, but it may be calculated based on other settings, such as a reverberation time of 1 / 4. The weighting coefficient ratio (3:7) of the two FDNs can be set appropriately depending on the desired reverberation sound attenuation curve. Furthermore, by using a third FDN with a larger attenuation coefficient and combining three or more FDNs, intelligibility can be further improved.

[0057] A computer can be suitably used to function as the reverberation control device 1 described above. Such a computer can be realized by storing a program describing the processing contents for realizing each function of the reverberation control device 1 in a memory unit of the computer, and having the central processing unit (CPU) of the computer read and execute this program.

[0058] The program can also be stored in a non-transitory computer-readable medium. Examples of the non-transitory computer-readable medium include flash memory, magnetic recording devices, optical disks, magneto-optical recording media, and ROM (read-only memory). The program can be distributed by selling, transferring, or lending portable media such as SD (Secure Digital) cards, DVDs (digital versatile discs), and CD-ROMs (compact disc read-only memory) that store the program.

[0059] The program causes a computer to execute the operations of the reverberation control device 1, thereby causing the computer to function as the reverberation control device 1. In other words, the computer functions as the reverberation control device 1 by executing the operations of the reverberation control device 1 in accordance with the program.

[0060] In the above embodiment, the configuration and operation of the reverberation control device 1 have been described, but the present invention is not limited to this and may be configured as a method for controlling reverberation for a desired object. That is, the present invention may be configured as a method for controlling reverberation by performing each step according to the flow of the flowchart in Fig. 2.

[0061] Although the above-described embodiments have been described as typical examples, it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention should not be construed as being limited by the above-described embodiments, and various modifications or alterations are possible without departing from the scope of the claims. For example, the functions included in each block, step, etc. described in the embodiments can be rearranged so as not to be logically inconsistent, and multiple constituent blocks, steps, etc. can be combined or divided into one. [Explanation of symbols]

[0062] 1 Reverberation control device 10 Metadata Analysis Section 20 Reverberation parameter calculation section 30 Rendering Section

Claims

1. a metadata analysis unit that analyzes the metadata and identifies an identifier of an object to be controlled; a reverberation parameter calculation unit that calculates a gain of a late reverberation of the object based on an identifier of the object and the longest distance of an area where clarity of the object is ensured; a rendering unit that calculates an amplitude gain coefficient for adjusting the amplitude of the late reverberation according to the distance between the user and the object and the gain of the late reverberation, and multiplies the amplitude gain coefficient by the late reverberation of the object to add reverberation; A reverberation control device comprising:

2. 2. The reverberation control device according to claim 1, the rendering unit calculates the amplitude gain coefficient by interpolation based on the distance d between the user and the object, and multiplies the amplitude gain coefficient by the rear reverberation to add reverberation, so that the energy ratio of direct sound to rear reverberation specified in metadata is satisfied when the user is in the same location as the object, and the gain of rear reverberation is satisfied at the furthest distance in the area where clarity is ensured.

3. 3. The reverberation control device according to claim 2, the reverberation parameter calculation unit calculates the clarity of the object at the longest distance in the area, and when the clarity is less than a predetermined threshold, calculates the gain for adjusting the rear reverberation so that the clarity is equal to or greater than the threshold, and when the clarity is equal to or greater than the predetermined threshold, sets the gain of the rear reverberation to 1 and outputs it to a rendering unit.

4. 3. The reverberation control device according to claim 1, the rendering unit adjusts a decay curve of late reverberation by adding a first FDN having a specified reverberation time and a second FDN having a reverberation time shorter than the specified reverberation time at a predetermined ratio, and adds reverberation based on the adjusted decay curve of late reverberation.

5. 5. The reverberation control device according to claim 4, A reverberation control device that calculates gains for adjusting the rear reverberation of each of the first FDN and the second FDN at the longest distance of an area where clarity of the object is ensured, based on the gain of the rear reverberation and the predetermined ratio.

6. A program that causes a computer to function as the reverberation control device according to claim 1.

Images