Audio processing device and method, and computer-readable storage medium
By computing the expansion vector and gain processing, the problem of asymmetry in the MPEG-H 3D audio standard is solved, and higher quality sound reproduction and more accurate sound image positioning are achieved, adapting to renderers of different hardware scales.
Patent Information
- Application Number
- CN202011538529.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2015-07-28
- Filing Date
- 2016-06-09
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2036-06-09
AI Technical Summary
The existing MPEG-H 3D audio standards have asymmetric constraints on the spread of sound and image in extended sound image processing, which makes it impossible to effectively consider the directionality of the sound and cannot obtain sufficient high-quality sound.
By obtaining the position information of the audio object and the diffusion metadata of the audio object, the expansion vector is calculated to determine the diffusion area of the audio image, and the gain of each sound output unit is calculated based on these vectors, more precise positioning and diffusion processing of the audio image is achieved.
It realizes higher quality sound reproduction, can obtain more accurate sound image positioning and sound diffusion effects within the allowed processing amount, reduces the amount of rendering processing, and adapts to renderers of different hardware scales.
Smart Images

Figure CN112562697B_ABST
Abstract
Description
[0001] This invention application is a divisional application of the invention patent application with application date of June 9, 2016, application number “201680034827.1” and invention name “Device, method and program for processing sound”. Technical Field
[0002] The present technology relates to an audio processing device, method, and program, and more particularly to an audio processing device, method, and program by which higher quality sound can be obtained. Background Art
[0003] Conventionally, as a technique for controlling the localization of a sound image using a plurality of speakers, VBAP (Vector-Based Amplitude Phase Shift) is known (for example, refer to NPL 1).
[0004] In VBAP, by outputting sounds from three speakers, a sound image can be localized at an arbitrary point inside a triangle defined by the three speakers.
[0005] However, in the real world, sound images are not localized at a single point, but rather are localized in a partial space with a certain degree of diffusion. For example, when a human voice is produced by the vocal cords, the vibrations of the sound are thought to spread to the face, body, etc., and thus the sound is thought to emanate from a partial space that is the entire human body.
[0006] As a technique for localizing sound in such a partial space, that is, for extending the sound image, MDAP (Multi-Directional Amplitude and Phase Shifting) is generally known (see, for example, NPL 2). MDAP is also used in the rendering processing unit of the MPEG-H 3D (Moving Picture Experts Group - High Quality 3D) audio standard (see, for example, NPL 3).
[0007] [Citation List]
[0008] [Non-Patent Literature (NPL)]
[0009] [NPL 1]
[0010] Ville Pulkki, “Virtual Sound Source Positioning Using Vector Base Amplitude Panning”, Journal of the Electrochemical Society, Vol. 45, No. 6, pp. 456-466, 1997
[0011] [NPL 2]
[0012] Ville-Pulkki, “Uniform Spreading of Amplitude Panned Virtual Sources,” Proceedings of the 1999 IEEE Symposium on Applications of Signal Processing to Audio and Acoustics, New Paltz, NY, October 17–20, 1999
[0013] [NPL 3]
[0014] ISO / IEC JTC1 / SC29 / WG11 N14747, August 2014, Sapporo, Japan, “Text of ISO / IEC 23008-3 / DIS, 3D Audio” Summary of the Invention
[0015] [Technical Issues]
[0016] However, the above-mentioned techniques cannot obtain sufficiently high-quality sound.
[0017] For example, in the MPEG-H 3D Audio standard, information indicating the degree of diffusion of the sound image (referred to as "spread") is included in the metadata of audio objects, and processing for extending the sound image is performed based on this spread. However, this processing is constrained by the requirement that the diffusion of the sound image must be symmetrical in the vertical and horizontal directions relative to the center of the audio object. Consequently, processing that takes into account the directionality (radial direction) of the sound from the audio object cannot be performed, and thus sufficiently high-quality sound cannot be obtained.
[0018] The present technology has been made in view of the above circumstances, and makes it possible to obtain higher quality sound.
[0019] [Solution to the problem]
[0020] An audio processing device according to one aspect of the present technology includes: an acquisition unit configured to acquire metadata including position information indicating the position of an audio object and sound image information composed of vectors of at least two dimensions or more and representing the diffusion of the sound image from the position; a vector calculation unit configured to calculate an expansion vector indicating the position in the area based on the horizontal direction angle and the vertical direction angle of the area representing the diffusion of the sound image determined by the sound image information; and a gain calculation unit configured to calculate the gain of each of the audio signals provided to two or more sound output units located near the position indicated by the position information based on the expansion vector.
[0021] The vector calculation unit may calculate the extension vector based on a ratio between the horizontal direction angle and the vertical direction angle.
[0022] The vector calculation unit may calculate a predetermined number of extended vectors.
[0023] The vector calculation unit can calculate an arbitrarily variable number of extended vectors.
[0024] The sound image information may be a vector indicating the center position of the area.
[0025] The sound image information may be a two-dimensional or more dimensional vector indicating the extent to which the sound image is diffused from the center of the area.
[0026] The sound image information may be a vector indicating the relative position of the center position of the area viewed from the position indicated by the position information.
[0027] The gain calculation unit can calculate the gain of each extension vector of each sound output unit in the sound output unit, calculate the added value of the gains calculated for the extension vector of each sound output unit in the sound output unit, quantize the added value into a gain of two or more values for each sound output unit in the sound output unit, and calculate the final gain for each sound output unit in the sound output unit based on the quantized added value.
[0028] The gain calculation unit can select the number of grids to be used for calculating the gain, each grid in the grid is an area surrounded by three sound output units in the sound output unit, and the gain calculation unit calculates the gain of each extended vector in the extended vector based on the selection result of the number of grids and the extended vector.
[0029] The gain calculation unit may select the number of grids to be used for calculating the gain, whether to perform quantization, and the quantization number of the added value at the time of quantization, and calculate the final gain in response to the selection result.
[0030] The gain calculation unit may select the number of grids to be used for calculating the gain, whether to perform quantization, and the quantization amount based on the number of audio objects.
[0031] The gain calculation unit may select the number of meshes to be used for calculating the gain, whether to perform quantization, and the amount of quantization based on the degree of importance of the audio object.
[0032] The gain calculation unit may select the number of meshes to be used for calculating the gain such that the number of meshes to be used for calculating the gain increases as the position of the audio object is located closer to an audio object having a high degree of importance.
[0033] The gain calculation unit may select the number of grids to be used for calculating the gain, whether to perform quantization, and the quantization amount based on the sound pressure of the audio signal of the audio object.
[0034] The gain calculation unit may select three or more sound output units among a plurality of sound output units including sound output units located at different heights from each other in response to a selection result of the number of grids, and calculate gains based on one or more grids formed by the selected sound output units.
[0035] An audio processing method or program according to one aspect of the present technology includes the following steps: acquiring metadata including position information indicating the position of an audio object and sound image information composed of vectors of at least two dimensions or more and representing the diffusion of the sound image from the position; calculating an expansion vector indicating the position in the area based on the horizontal direction angle and the vertical direction angle of the area representing the diffusion of the sound image determined by the sound image information; and calculating the gain of each of the audio signals provided to two or more sound output units located near the position indicated by the position information based on the expansion vector.
[0036] In one aspect of the present technology, metadata is acquired, including positional information indicating the location of an audio object and sound image information, which is composed of at least two or more dimensional vectors and represents the spread of the sound image from the location. A spread vector is then calculated, indicating the location within the region based on the horizontal and vertical angles of the region representing the spread of the sound image determined by the sound image information. Furthermore, based on the spread vector, a gain is calculated for each of the audio signals provided to two or more sound output units located near the location indicated by the positional information.
[0037] An audio processing device according to one aspect of the present technology includes: an acquisition unit configured to acquire metadata including position information indicating the position of an audio object and sound image information composed of vectors of at least two dimensions or more and representing the diffusion of the sound image from the position; a vector calculation unit configured to calculate an expansion vector indicating the position in the area based on the horizontal direction angle and the vertical direction angle of the area representing the diffusion of the sound image determined by the sound image information, wherein the number of expansion vectors is predetermined and does not depend on the diffusion of the sound image; and a gain calculation unit configured to calculate the gain of each of the audio signals provided to two or more sound output units located near the position indicated by the position information based on the expansion vector.
[0038] An audio processing method or program according to one aspect of the present technology includes the following steps: acquiring metadata including position information indicating the position of an audio object and sound image information composed of vectors of at least two dimensions or more and representing the diffusion of the sound image from the position; calculating an expansion vector indicating the position in the area based on the horizontal direction angle and the vertical direction angle of the area representing the diffusion of the sound image determined by the sound image information, wherein the number of expansion vectors is predetermined and does not depend on the diffusion of the sound image; and calculating the gain of each of the audio signals provided to two or more sound output units located near the position indicated by the position information based on the expansion vector.
[0039] [Beneficial Effects of the Invention]
[0040] Utilizing one aspect of the present technology, higher quality sound can be achieved.
[0041] It should be noted that the effects described here are not necessarily limitative, but any of the effects described in the present disclosure may be exhibited. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 is a diagram showing VBAP.
[0043] Figure 2 is a diagram showing the position of a sound image.
[0044] Figure 3 is the graph of the extended vector.
[0045] Figure 4 is a diagram illustrating the extended center vector method.
[0046] Figure 5 is a diagram illustrating the diffuse radiation vector method.
[0047] Figure 6 is a diagram showing an example of the configuration of an audio processing device.
[0048] Figure 7 is a flowchart showing the reproduction process.
[0049] Figure 8 is a flowchart showing the extended vector calculation process.
[0050] Figure 9 is a flowchart showing an extended vector calculation process based on an extended three-dimensional vector.
[0051] Figure 10 : is a flowchart showing the expansion vector calculation process based on the expansion center vector.
[0052] Figure 11It is a flowchart showing the extended vector calculation process based on the extended end vector.
[0053] Figure 12 It is a flowchart showing the extended vector calculation process based on the extended radiation vector.
[0054] Figure 13 It is a flowchart showing the extended vector calculation process based on the extended vector position information.
[0055] Figure 14 It is a diagram showing the switching of the number of grids.
[0056] Figure 15 It is a diagram showing the switching of the number of grids.
[0057] Figure 16 It is a diagram showing the formation of grids.
[0058] Figure 17 It is a diagram showing an example of the configuration of an audio processing device.
[0059] Figure 18 It is a flowchart showing the reproduction process.
[0060] Figure 19 It is a diagram showing an example of the configuration of an audio processing device.
[0061] Figure 20 It is a flowchart showing the reproduction process.
[0062] Figure 21 It is a flowchart showing the VBAP gain calculation process.
[0063] Figure 22 It is a diagram showing an example of the configuration of a computer. Detailed implementation manners
[0064] Hereinafter, embodiments applying the present technology will be described with reference to the drawings.
[0065] <The first embodiment>
[0066] <Processing of VBAP and extended sound image>
[0067] When obtaining an audio signal of an audio object and metadata such as the position information of the audio object for rendering, the present technology enables obtaining higher-quality sound. It should be noted that in the following description, the audio object is simply referred to as an object.
[0068] First, the processing of VBAP and the extended sound image in the MPEG-H 3D audio standard will be described below.
[0069] For example, as Figure 1As shown, it is assumed that a user U11 who enjoys content of moving images, musical compositions, or the like with sound is listening to three-channel sound output from three speakers SP1 to SP3 as the sound of the content.
[0070] In the case just described, localization of the sound image at the position p is verified using information on the positions of the three speakers SP1 to SP3 that output sounds of different channels.
[0071] For example, the position p is represented by a three-dimensional vector (hereinafter also referred to as vector p) whose origin is the origin O in the three-dimensional coordinate system, and the origin O of the three-dimensional coordinate system is given by the position of the head of the user U11. In addition, if the three-dimensional vectors whose origins are given by the origin O and are oriented in the direction toward the positions of the speakers SP1 to SP3 are represented as vectors I1 to I3, respectively, then the vector p can be represented by the linear sum of the vectors I1 to I3.
[0072] In other words, the vector p can be expressed as p=g1I1+g2I2+g3I3.
[0073] Here, if coefficients g1 to g3 multiplied by the vectors I1 to I3 are calculated and determined as gains of sounds output from the speakers SP1 to SP3 , respectively, a sound image may be localized at the position p.
[0074] The technique of determining coefficients g1 to g3 using position information of the three speakers SP1 to SP3 and controlling the localization position of the sound image in the above manner is called three-dimensional VBAP. In particular, in the following description, the gain determined for each speaker, such as coefficients g1 to g3, is referred to as VBAP gain.
[0075] exist Figure 1 In the example shown in FIG, the sound image can be positioned at any position in a triangular region TR11 on a sphere including the positions of the speakers SP1, SP2, and SP3. Here, the region TR11 is a region on the surface of the sphere centered at the origin O and passing through the positions of the speakers SP1 to SP3, and is a triangular region surrounded by the speakers SP1 to SP3.
[0076] Using this three-dimensional VBAP allows for positioning of the sound image at any position in space. It should be noted that VBAP is described in detail in, for example, Ville Pulkki, “Virtual Sound Source Positioning Using Vector Base Amplitude Panning,” Journal of the Electrochemical Society of America, Vol. 45, No. 6, pp. 456-466, 1997.
[0077] Now, a process of extending the sound image according to the MPEG-H 3D audio standard is described.
[0078] In the MPEG-H 3D Audio standard, a bit stream obtained by multiplexing encoded audio data obtained by encoding an audio signal of each object and encoded metadata obtained by encoding metadata of each object is output from an encoding device.
[0079] For example, metadata includes position information indicating the position of an object in space, importance information indicating the degree of importance of the object, and an extension as information indicating the degree of diffusion of a sound image of the object.
[0080] Here, the spread indicating the degree of diffusion of the sound image is an arbitrary angle from 0 degrees to 180 degrees, and the encoding device can specify the spread having a different value for each frame of the audio signal with respect to each object.
[0081] In addition, the position of the object is represented by the horizontal direction angle azimuth, the vertical direction angle elevation and the distance radius. Specifically, the position information of the object is configured by the values of the horizontal direction angle azimuth, the vertical direction angle elevation and the distance radius.
[0082] For example, consider a three-dimensional coordinate system such as Figure 2 As shown, in the three-dimensional coordinate system, the position of the user who enjoys the sound of the object output from the speaker not shown is determined as the origin O, and Figure 2 The upper right direction, the upper left direction, and the upward direction are determined as the x-axis, the y-axis, and the z-axis perpendicular to each other. At this time, if the position of an object is represented as the position OBJ11, the sound image can be positioned at the position OBJ11 in the three-dimensional coordinate system.
[0083] Furthermore, if a linear line interconnecting the position OBJ11 and the origin O is represented as a line L, the xy plane defined by the linear line L and the x-axis is Figure 2 The horizontal angle θ (azimuth) in the horizontal direction is the horizontal direction angle azimuth representing the horizontal position of the object at position OBJ11, and the horizontal direction angle azimuth has any value that satisfies the following: -180 degrees ≤ azimuth ≤ 180 degrees
[0084] For example, the positive direction of the x-axis is determined as an azimuth angle of 0 degrees, and the negative direction of the x-axis is determined as an azimuth angle of +180 degrees or -180 degrees. In addition, the counterclockwise direction around the origin O is determined as the positive direction of the azimuth angle, and the clockwise direction around the origin O is determined as the negative direction of the azimuth angle.
[0085] Furthermore, the angle defined by the linear line L and the xy plane, ie, Figure 2The vertical angle γ (elevation angle) in is the vertical angle elevation angle representing the vertical position of the object located at position OBJ11, and the vertical angle elevation angle has any value that satisfies the following: -90 degrees ≤ elevation angle ≤ 90 degrees. For example, the position on the xy plane is elevation angle = 0 degrees, and Figure 2 The upward direction in is the positive direction of the vertical angle elevation, and Figure 2 The downward direction in is the negative direction of the vertical direction angle elevation.
[0086] Furthermore, the length of linear line L, that is, the distance from origin O to position OBJ11, is the distance radius to the user, and the distance radius has a value of 0 or greater. Specifically, the distance radius has a value that satisfies the following: 0 ≤ (radius) ≤ ∞. In the following description, the distance radius is also referred to as the distance in the radial direction.
[0087] It should be noted that in VBAP, the distance radius from all speakers or objects to the user is equal, and it is a general method to perform calculations by normalizing the distance radius to 1.
[0088] The position information of the object included in the metadata in this manner is configured by the values of the horizontal direction angle azimuth, the vertical direction angle elevation, and the distance radius.
[0089] In the following description, the horizontal direction angle azimuth, the vertical direction angle elevation and the distance radius are also referred to as azimuth, elevation and radius respectively.
[0090] Furthermore, in a decoding device that receives a bit stream including encoded audio data and encoded metadata, after performing decoding of the encoded audio data and the encoded metadata, a rendering process for extending a sound image is performed in response to an extended value included in the metadata.
[0091] Specifically, the decoding apparatus first determines the position in the space indicated by the position information included in the metadata of the object as the position p. As described above, the position p corresponds to Figure 1 Position p in.
[0092] Then, the decoding apparatus sets 18 extension vectors p1 to p18 so that, for example, the position p is set to position p=center position p0, as shown in FIG. Figure 3 As shown in , these expansion vectors are symmetric in the up-down and left-right directions on the unit sphere around the center position p0. It should be noted that Figure 3 In, with Figure 1 Corresponding parts in the case of FIG are denoted by the same reference numerals, and descriptions of these parts will be appropriately omitted.
[0093] exist Figure 3In the example, the five speakers SP1 to SP5 are arranged on a spherical surface of a unit sphere with a radius of 1 and an origin O as the center, and the position p indicated by the position information is the center position p0. In the following description, the position p is also specifically referred to as the object position p, and the vector whose starting point is the origin O and whose end point is the object position p is also referred to as the vector p. In addition, the vector whose starting point is the origin O and whose end point is the center position p0 is also referred to as the vector p0.
[0094] exist Figure 3 In , its starting point is the origin O and the arrow mark drawn by the dotted line represents the extension vector. However, although in Figure 3 There are actually 18 expansion vectors in , but for Figure 3 The visibility only plots the eight expansion vectors.
[0095] Here, each of the expansion vectors p1 to p18 is a vector whose end point position is located within the circular region R11 on the unit sphere centered at the center position p0. Specifically, the angle defined by the vector p0 and the expansion vector whose end point position is located on the circumference of the circle represented by the region R11 is the angle indicated by the expansion.
[0096] Therefore, as the value of the spread increases, the end point position of each spread vector is set at a position farther from the center position p0. In other words, the region R11 increases in size.
[0097] Region R11 represents the spread of the sound image from the object's position. In other words, region R11 is the region indicating the spread of the object's sound image. Furthermore, since the object's sound is considered to emanate from the entire object, region R11 can be considered to represent the object's shape. In the following description, the region indicating the spread of the object's sound image, like region R11, is also referred to as the region indicating the spread of the sound image.
[0098] Furthermore, in the case where the value of the extension is 0, the end position of the 18 extension vectors p1 to p18 is equal to the center position p0.
[0099] It should be noted that in the following description, the end point positions of the expansion vectors p1 to p18 are also specifically referred to as positions p1 to p18 , respectively.
[0100] After determining the expansion vectors that are symmetrical in the vertical and horizontal directions on the unit sphere as described above, the decoding device calculates the VBAP gain for each of the speakers in the channel using VBAP, with respect to vector p and the expansion vectors, that is, with respect to position p and each of positions p1 to p18. At this time, the VBAP gain of the speaker is calculated so that the sound image is localized at each of the positions, such as position p and position p1.
[0101] The decoding device then adds the VBAP gains calculated for each speaker position. Figure 3 In the example of , the VBAP gains of the position p calculated with respect to the speaker SP1 and the positions p1 to p18 are added.
[0102] Furthermore, after the addition process calculated for each speaker, the decoding device normalizes the VBAP gain. Specifically, the normalization is performed so that the sum of the squares of the VBAP gains of all speakers becomes 1.
[0103] Then, the decoding device multiplies the audio signal of the object by the VBAP gain of the speaker obtained by normalization to obtain an audio signal of each speaker, and provides the audio signal obtained with respect to each speaker to the speaker so that the speaker outputs sound.
[0104] So, for example, in Figure 3 In the example of , the sound image is localized so that the sound is output from the entire region R11. In other words, the sound image is extended to the entire region R11.
[0105] exist Figure 3 In the example shown in FIG1 , when the processing for extending the sound image is not performed, the sound image of the object is localized at position p, so in this case, the sound is basically output from speakers SP2 and SP3. In contrast, when the processing for extending the sound image is performed, the sound image is extended to the entire region R11, so when the sound is reproduced, the sound is output from speakers SP1 to SP4.
[0106] Incidentally, when performing such processing for extending the sound image as described above, the amount of processing required for rendering increases compared to an alternative approach where the processing is not performed. Consequently, the number of objects that can be processed by the decoding device may decrease, or a decoding device with a small hardware renderer may be unable to perform rendering.
[0107] Therefore, in the case of performing processing for extending a sound image when rendering, it is desirable to be able to perform rendering with as small a processing amount as possible.
[0108] Furthermore, due to the constraint that the 18 expansion vectors are symmetrical in the vertical and horizontal directions on the unit sphere around the center position p0 = position p, processing that takes into account the directivity (radiation direction) of the sound of the object or the shape of the object cannot be performed. Therefore, sufficiently high-quality sound cannot be obtained.
[0109] Furthermore, the MPEG-H 3D Audio standard specifies a process for extending the sound image during rendering. Therefore, if the hardware scale of the renderer is small, this process cannot be performed. In other words, audio reproduction cannot be performed.
[0110] Furthermore, in the MPEG-H 3D Audio standard, the processing of switching the execution of rendering cannot be performed so that sound with the highest quality can be obtained with the amount of processing allowed by the hardware scale of the renderer.
[0111] Taking into account the above-mentioned circumstances, this technology makes it possible to reduce the amount of processing required during rendering. Furthermore, this technology makes it possible to obtain sufficiently high-quality sound by expressing the directionality or shape of an object. Furthermore, this technology makes it possible to select appropriate processing for rendering in response to the hardware scale of a renderer, etc., in order to obtain the highest-quality sound within the allowable processing volume.
[0112] The outline of the present technology is described below.
[0113] <Reduction in processing volume>
[0114] First, the reduction in processing volume at the time of rendering will be described.
[0115] In a common VBAP process (rendering process) in which the sound image is not extended, processes A1 to A3 described in detail below are performed:
[0116] (Processing A1)
[0117] The VBAP gains that are multiplied by the audio signals are calculated for the three speakers.
[0118] (Processing A2)
[0119] Normalization is performed so that the sum of the squares of the VBAP gains of the three speakers becomes 1.
[0120] (Processing A3)
[0121] The audio signal of the object is multiplied by the VBAP gain.
[0122] Here, in the process A3, since the multiplication process of multiplying the audio signal by the VBAP gain is performed for each of the three speakers, such a multiplication process just described is performed three times at most.
[0123] On the other hand, in the VBAP process (rendering process) that performs a process for extending a sound image, processes B1 to B5 described in detail below are performed:
[0124] (Processing B1)
[0125] The VBAP gains to be multiplied by the audio signals for each of the three speakers are calculated for the vector p.
[0126] (Processing B2)
[0127] The VBAP gains to be multiplied by the audio signals for each of the three speakers are calculated for the 18 extension vectors.
[0128] (Processing B3)
[0129] The VBAP gains calculated for the vector are summed for each loudspeaker.
[0130] (Processing B4)
[0131] Normalization is performed so that the sum of the squares of the VBAP gains of all speakers becomes 1.
[0132] (Processing B5)
[0133] The audio signal of the object is multiplied by the VBAP gain.
[0134] When the process for extending the sound image is performed, since the number of speakers that output sounds is three or more, the multiplication process in the process B5 is performed three or more times.
[0135] Therefore, if a case where the processing for extending the sound image is performed is compared with another case where the processing for extending the sound image is not performed, when the processing for extending the sound image is performed, the processing amount is increased by a certain amount, especially by the processing B2 and B3, and the processing amount in the processing B5 is also greater than the processing amount in the processing A3.
[0136] Therefore, the present technology makes it possible to reduce the amount of processing in the above-mentioned process B5 by quantizing the sum of the VBAP gains of the vectors determined with respect to each speaker.
[0137] Specifically, such processing as described below is performed by the present technology. It should be noted that the sum (addition value) of VBAP gains calculated for each vector such as vector p or a spread vector determined for each speaker is also referred to as a VBAP gain addition value.
[0138] First, after executing processes B1 to B3 and obtaining a VBAP gain addition value for each speaker, the VBAP gain addition value is binarized. In the binarization, for example, the VBAP gain addition value for each speaker is set to one of 0 and 1.
[0139] As a method for binarizing the VBAP gain added value, any method such as rounding, upper limit (round-up), lower limit (truncation), or threshold processing may be employed.
[0140] After binarizing the VBAP gain sum in this manner, the aforementioned process B4 is performed based on the binarized VBAP gain sum. Therefore, the final VBAP gain for each speaker is a gain other than 0. In other words, if the VBAP gain sum is binarized, the final VBAP gain for each speaker is 0 or a predetermined value.
[0141] For example, if the sum of the VBAP gains of three speakers is 1 and the sum of the VBAP gains of another speaker is 0 as a result of binarization, the final value of the VBAP gains of the three speakers is 1 / 3. (1 / 2) .
[0142] After the final VBAP gain of the speaker is obtained in this manner, a process of multiplying the audio signal of the speaker by the final VBAP gain is performed as process B5 ′ instead of the above-described process B5 .
[0143] If binarization is performed in this manner, the final value of the VBAP gain of each speaker becomes either 0 or a predetermined value. Therefore, in process B5′, multiplication processing only needs to be performed once, thereby reducing the processing amount. In other words, while process B5 requires multiplication processing to be performed three or more times, process B5′ requires multiplication processing to be performed only once.
[0144] It should be noted that although the case of binarizing the VBAP gain added value is given here as an example, the VBAP gain added value may be quantized into one of three or more values in other ways.
[0145] For example, if the VBAP gain addition value is one of three values, after executing the above-described processes B1 to B3 and obtaining the VBAP gain addition value for each speaker, the VBAP gain addition value is quantized to one of 0, 0.5, and 1. Then, processes B4 and B5' are executed. In this case, the number of multiplication processes in process B5' is a maximum of two.
[0146] When the VBAP gain added value is x-value converted in this manner, i.e., when the VBAP gain added value is quantized into one of x gains (where x is equal to or greater than 2), the maximum number of times the multiplication process is performed in process B5' becomes (x-1).
[0147] It should be noted that while the foregoing description describes an example in which the VBAP gain addition value is quantized to reduce the amount of processing when processing for extending the sound image is performed, the amount of processing can be similarly reduced by quantizing the VBAP gain even when processing for extending the sound image is not performed. Specifically, quantizing the VBAP gain for each speaker determined for vector p can reduce the number of times the audio signal multiplication process using the normalized VBAP gain is performed.
[0148] <Processing for Representing the Shape of an Object and the Directionality of its Sound>
[0149] Now, a process of expressing the shape of an object and the directionality of the sound of the object by the present technology is described.
[0150] Hereinafter, five methods including an extended three-dimensional vector method, an extended center vector method, an extended end vector method, an extended radiation vector method, and an arbitrary extended vector method are described.
[0151] (Extended 3D Vector Method)
[0152] First, the extended three-dimensional vector method is described.
[0153] In the extended 3D vector method, an extended 3D vector, which is a 3D vector, is stored in the bitstream and transmitted along with the bitstream. Here, it is assumed that the extended 3D vector is stored, for example, in metadata for each frame of the audio signal for each object. In this case, the extension indicating the degree of diffuseness of the sound image is not stored in the metadata.
[0154] For example, the extended three-dimensional vector is a three-dimensional vector including three factors: s3_azimuth indicating the degree of spread of the sound image in the horizontal direction, s3_elevation indicating the degree of spread of the sound image in the vertical direction, and s3_radius indicating the depth in the radial direction of the sound image.
[0155] Specifically, the extended three-dimensional vector = (s3_azimuth, s3_elevation, s3_radius).
[0156] Here, s3_azimuth indicates the spread angle of the sound image in the horizontal direction of position p, that is, in the direction of the aforementioned horizontal direction angle azimuth. Specifically, s3_azimuth indicates the angle defined by vector p (vector p0) and a vector extending from the origin o toward the end of the horizontal direction side of the region indicating the spread of the sound image.
[0157] Similarly, s3_elevation indicates the spread angle of the sound image in the vertical direction at position p, that is, in the direction of the aforementioned vertical angle elevation. Specifically, s3_elevation indicates the angle defined by vector p (vector pO) and a vector extending from the origin o toward the vertical end of the region indicating the spread of the sound image. Furthermore, s3_radius indicates the depth in the direction of the aforementioned distance radius, that is, in the direction of the normal to the unit sphere.
[0158] Note that s3_azimuth, s3_elevation, and s3_radius have values equal to or greater than 0. Furthermore, although the extended three-dimensional vector here is information indicating a relative position to the position p indicated by the position information of the object, the extended three-dimensional vector may alternatively be information indicating an absolute position.
[0159] In the extended three-dimensional vector method, rendering is performed using such extended three-dimensional vectors as described above.
[0160] Specifically, in the extended three-dimensional vector method, the extended value is calculated by calculating the expression (1) given below based on the extended three-dimensional vector:
[0161] [Expression 1]
[0162] Extension: max(s3_azimuth,s3_elevation) ...(1)
[0163] It should be noted that max(a, b) in Expression (1) indicates a function that returns the higher value of the values a and b. Therefore, the higher value of s3_azimuth and s3_elevation is determined as the extended value.
[0164] Then, based on the extended values obtained in this manner and the position information included in the metadata, 18 extended vectors p1 to p18 are calculated similarly to the case of the MPEG-H 3D audio standard.
[0165] Therefore, the position p of the object indicated by the position information included in the metadata is determined as the center position p0, and 18 expansion vectors p1 to p18 are determined so that they are symmetrical in the left-right direction and the up-down direction on the unit sphere centered at the center position p0.
[0166] Furthermore, in the extended three-dimensional vector method, a vector pO whose starting point is the origin O and whose end point is the center position pO is determined as an extended vector p0.
[0167] In addition, each expansion vector is represented by a horizontal direction angle azimuth, a vertical direction angle elevation and a range radius. In the following, the horizontal direction angle azimuth and the vertical direction angle elevation of the expansion vector pi (where i = 0 to 18) are represented as a(i) and e(i), respectively.
[0168] After the extended vectors p0 to p18 are obtained in this manner, the extended vectors p1 to p18 are changed into (corrected into) final extended vectors based on the ratio between s3_azimuth and s3_elevation.
[0169] Specifically, in the case where s3_azimuth is greater than s3_elevation, calculation of the following expression (2) is performed to change e(i) which is the elevation angle of the spread vectors p1 to p18 into e′(i):
[0170] [Expression 2]
[0171] e'(i)=e(0)+(e(i)–e(0))×s3_elevation / s3_azimuth ...(2)
[0172] It should be noted that for the extended vector p0, no correction of the elevation angle is performed.
[0173] In contrast, in the case where s3_azimuth is smaller than s3_elevation, calculation of the following expression (3) is performed to change a(i) which is the azimuth angle of the extended vectors p1 to p18 into a′(i):
[0174] [Expression 3]
[0175] a'(i)=a(0)+(a(i)–a(0))×s3_azimuth / s3_elevation…(3)
[0176] It should be noted that for the extended vector p0, no correction of the azimuth angle is performed.
[0177] The process of determining the larger one of s3_azimuth and s3_elevation as the extension to determine the extension vector in the manner described above is the following process: the area indicating the spread of the sound image on the unit sphere is temporarily set to a circle of a radius defined by the angle of the larger one of s3_azimuth and s3_elevation to determine the extension vector by a process similar to the conventional process.
[0178] Furthermore, the subsequent processing of correcting the extension vector by expression (2) or expression (3) in response to the magnitude relationship between s3_azimuth and s3_elevation is the following processing: correcting the area indicating the diffusion of the sound image, i.e., the extension vector, so that the area indicating the diffusion of the sound image on the unit sphere becomes the area defined by the original s3_azimuth and s3_elevation specified by the extension three-dimensional vector.
[0179] Therefore, the above-described processing then all becomes processing of calculating an expansion vector indicating an area of spread of a sound image on a unit sphere, which has a circular shape or an elliptical shape, based on the expansion three-dimensional vector, ie, based on s3_azimuth and s3_elevation.
[0180] After the extension vectors are obtained in this manner, the extension vectors p0 to p18 are thereafter used to perform the above-described process B2, process B3, process B4, and process B5' to generate audio signals to be supplied to the speakers.
[0181] Note that in process B2, the VBAP gain for each speaker is calculated for each of the 19 extension vectors p0 to p18. Since extension vector p0 is vector p, the calculation of the VBAP gain for extension vector p0 can be considered to be the execution of process B1. Furthermore, after process B3, the VBAP gain sum is quantized as needed.
[0182] By setting the area indicating the spread of the sound image to an area of an arbitrary shape by the expanded three-dimensional vector in this way, the shape of the object and the directionality of the sound of the object can be expressed, so that higher quality sound can be obtained by rendering.
[0183] Furthermore, although an example in which the higher value of the values s3_azimuth and s3_elevation is used as the extended value is described here, the lower value of the values s3_azimuth and s3_elevation may be used as the extended value.
[0184] In this case, when s3_azimuth is greater than s3_elevation, the azimuth angle a(i) of each extension vector is corrected, but when s3_azimuth is less than s3_elevation, the elevation angle e(i) of each extension vector is corrected.
[0185] Furthermore, although an example is given here in which the extension vectors p0 to p18, ie, 19 extension vectors determined in advance, are determined and the VBAP gain is calculated for these extension vectors, the number of extension vectors to be calculated is variable.
[0186] In the case just described, the number of spread vectors to be generated can be determined, for example, in response to the ratio between s3_azimuth and s3_elevation. According to this processing just described, for example, in the case of an object that is horizontally elongated and the vertical diffusion of the sound of the object is small, if the spread vectors arranged in parallel in the vertical direction are ignored and the spread vectors are basically arranged in parallel in the horizontal direction, the horizontal diffusion of the sound can be appropriately expressed.
[0187] (Extended Center Vector Method)
[0188] Now, the extended center vector method is described.
[0189] In the extended center vector method, an extended center vector, a three-dimensional vector, is stored in the bitstream and transmitted along with the bitstream. Here, it is assumed that the extended center vector is stored, for example, in the metadata of each audio signal frame for each object. In this case, the spread indicating the degree of diffuseness of the sound image is also stored in the metadata.
[0190] The expansion center vector is a vector indicating the center position pO of the area where the sound image of the object is diffused. For example, the expansion center vector is a three-dimensional vector composed of the following three factors: an azimuth indicating the horizontal direction angle of the center position pO, an elevation indicating the vertical direction angle of the center position pO, and a radius indicating the distance in the radial direction from the center position pO.
[0191] Specifically, the expansion center vector = (azimuth, elevation, radius).
[0192] When rendering, the position indicated by the extension center vector is determined as the center position p0, and the extension vectors p0 to p18 are calculated as the extension vectors. Figure 4 As depicted, the extended vector p0 is a vector p0 whose starting point is the origin O and whose end point is the center position pO. It should be noted that Figure 4 In, with Figure 3 Corresponding parts in the case of FIG. 1 are denoted by the same reference numerals, and their descriptions are appropriately omitted.
[0193] In addition, Figure 4 In the example, the dotted arrows represent the expansion vectors, and similarly in Figure 4 In order to make the figure easier to see, only nine expansion vectors are depicted.
[0194] However, in Figure 3 In the example depicted, position p = center position PO, Figure 4In the example of FIG. 1 , the center position p0 is a position different from the position p. In this example, it can be seen that the region R21 indicating the spread of the sound image and centered at the center position p0 is different from the position p. Figure 3 In the example of Figure 4 on the left side of the .
[0195] If an arbitrary position can be specified as the center position pO of the region indicating the diffusion of the sound image by expanding the center vector in this way, the directionality of the sound of the object can be expressed with higher accuracy.
[0196] In the extended center vector method, if extended vectors p0 to p18 are obtained, then processing B1 is performed on the vector p, and processing B2 is performed on the extended vectors p0 to p18.
[0197] Note that in process B2, the VBAP gain may be calculated for each of the 19 extension vectors, or may be calculated only for the extension vectors p1 to p18 excluding the extension vector p0. Hereinafter, a description will be given assuming that the VBAP gain is calculated also for the extension vector p0.
[0198] Furthermore, after calculating the VBAP gain of each vector, processing B3, processing B4, and processing B5' are performed to generate an audio signal to be supplied to the speaker. It should be noted that after processing B3, quantization of the VBAP gain addition value is performed as needed.
[0199] By using such an extended center vector method as described above, sufficiently high-quality sound can also be obtained by rendering.
[0200] (Extended End Vector Method)
[0201] Now, the extended end vector method is described.
[0202] In the extended end vector method, an extended end vector, a five-dimensional vector, is stored in the bitstream and transmitted along with the bitstream. Here, it is assumed that the extended end vector is stored in the metadata of each audio signal frame for each object. In this case, the spread indicating the degree of diffuseness of the sound image is not stored in the metadata.
[0203] For example, the expansion end vector is a vector indicating the area of diffusion of the sound image of the indication object and is a vector composed of the following five factors: expansion left end azimuth, expansion right end azimuth, expansion upper end elevation angle, expansion lower end elevation angle, and expansion radius.
[0204] Here, the extended left end azimuth and the extended right end azimuth, which constitute the extended end vector, respectively represent the values of the horizontal angular azimuths, which indicate the absolute positions of the left and right ends of the region indicating the spread of the sound image in the horizontal direction. In other words, the extended left end azimuth and the extended right end azimuth respectively indicate the degree of spread of the sound image in the left and right directions relative to the center position p0 of the region indicating the spread of the sound image.
[0205] Meanwhile, the "extended upper elevation angle" and "extended lower elevation angle" indicate vertical angle values, respectively, indicating the absolute positions of the upper and lower ends of the area where the sound image spreads in the vertical direction. In other words, the "extended upper elevation angle" and "extended lower elevation angle" indicate angles indicating the degree of upward and downward spread of the sound image relative to the center position p0 of the area where the sound image spreads. Furthermore, the "expanded radius" indicates the radial depth of the sound image.
[0206] It should be noted that although the extended end vector here is information indicating an absolute position in space, the extended end vector may alternatively be information indicating a relative position to the position p indicated by the position information of the object.
[0207] In the extended end vector method, rendering is performed using such extended end vectors as described above.
[0208] Specifically, in the extended end vector method, the following expression (4) is calculated based on the extended end vector to calculate the center position pO:
[0209] [Expression 4]
[0210] Azimuth: (extended left azimuth + extended right azimuth) / 2
[0211] Elevation angle: (extended upper elevation angle + extended lower elevation angle) / 2
[0212] Radius: Extension radius
[0213] …(4)
[0214] Specifically, the horizontal direction angle azimuth of the center position p0 is the middle (average) angle between the extended left end azimuth and the extended right end azimuth, and the vertical direction angle elevation of the center position p0 is the middle (average) angle between the extended upper end elevation and the extended lower end elevation. In addition, the distance radius of the center position p0 is the extended radius.
[0215] Therefore, in the extended end vector method, the center position p0 may be a position different from the position p of the object indicated by the position information.
[0216] Furthermore, in the extended end vector method, the extended value is calculated by calculating the following expression (5):
[0217] [Expression 5]
[0218] Extension: max((extended left end azimuth - extended right end azimuth) / 2, (extended upper end elevation - extended lower end elevation) / 2)
[0219] …(5)
[0220] It should be noted that max(a, b) in Expression (5) represents a function that returns the higher of the values a and b. Therefore, the higher of the values (extended left end azimuth - extended right end azimuth) / 2, which is an angle corresponding to the radius in the horizontal direction in the area of the spread of the sound image of the indicated object indicated by the expansion end vector, and (extended upper end azimuth - extended right end azimuth) / 2, which is an angle corresponding to the radius in the vertical direction in the area, is determined as the expansion value.
[0221] Then, based on the center position pO (vector pO) and the extended value obtained in this way, 18 extended vectors p1 to p18 are calculated similarly to the case of the MPEG-H 3D Audio standard.
[0222] Therefore, the 18 expansion vectors p1 to p18 are determined so that they are symmetrical in the up-down direction and the left-right direction on the unit sphere centered at the center position pO.
[0223] Furthermore, in the extended end vector method, a vector pO whose starting point is the origin O and whose end point is the center position pO is determined as an extended vector p0.
[0224] In the extended end vector method, similar to the case of the extended three-dimensional vector method, each extended vector is represented by a horizontal direction angle azimuth, a vertical direction angle elevation, and a range radius. In other words, the horizontal direction angle azimuth and vertical direction angle elevation of the extended vector pi (where i = 0 to 18) are represented by a(i) and e(i), respectively.
[0225] After the expansion vectors p0 to p18 are obtained in this manner, the expansion vectors p1 to p18 are changed (corrected) based on the ratio between (expanded left end azimuth - expanded right end azimuth) and (expanded upper end elevation angle - expanded lower end elevation angle) to determine the final expansion vector.
[0226] Specifically, if (extended left end azimuth - extended right end azimuth) is greater than (extended upper end elevation angle - extended lower end elevation angle), calculation of Expression (6) given below is performed, and e(i) which is the elevation angle of each extended vector p1 to p18 is changed to e'(i):
[0227] [Expression 6]
[0228] e'(i)=e(0)+(e(i)-e(0))×(extended upper elevation angle-extended lower elevation angle) / (extended left azimuth angle-extended right azimuth angle) ...(6)
[0229] It should be noted that for the extended vector p0, no correction of the elevation angle is performed.
[0230] On the other hand, when (extended left end azimuth angle - extended right end azimuth angle) is smaller than (extended upper end elevation angle - extended lower end elevation angle), calculation of Expression (7) given below is performed, and a(i) which is the azimuth angle of each extended vector p1 to p18 is changed to a'(i):
[0231] [Expression 7]
[0232] a'(i) = a(0) + (a(i) – a(0)) × (extended left end azimuth angle – extended right end azimuth angle) / (extended upper end elevation angle – extended lower end elevation angle) ...(7)
[0234] It should be noted that for the extended vector p0, no correction of the azimuth angle is performed.
[0235] It should be noted that the calculation method of the extended vector as described above is basically similar to the calculation method in the case of the extended three-dimensional vector method.
[0236] Therefore, all of the above-described processing thereafter is processing of calculating, from the spread end vector, a spread vector indicating an area where the circular or elliptical sound image spreads on the unit sphere defined by the spread end vector.
[0237] After the extended vector is obtained in this manner, the above-described processes B1 , B2 , B3 , B4 , and B5 ′ are performed using the vector p and the extended vectors p0 to p18 , thereby generating an audio signal to be supplied to the speaker.
[0238] It should be noted that in process B2, the VBAP gain of each speaker is calculated for 19 extension vectors. In addition, after process B3, quantization of the VBAP gain addition value is performed as needed.
[0239] By setting the area indicating the spread of the sound image to an arbitrary shaped area with a center position p0 located at an arbitrary position by the expansion end vector in this way, the shape of the object and the directionality of the sound of the object can be expressed, thereby obtaining higher quality sound through rendering.
[0240] In addition, although an example is described here in which the higher value of the values (extended left end azimuth angle - extended right end azimuth angle) / 2 and (extended upper end elevation angle - extended lower end elevation angle) / 2 is used as the extended value, the lower value of these values can also be used as the extended value.
[0241] Furthermore, although the case where the VBAP gain is calculated for the spreading vector p0 is described as an example, the VBAP gain may not be calculated for the spreading vector p0. The following description is given assuming that the VBAP gain is also calculated for the spreading vector p0.
[0242] Alternatively, similar to the case of the extended three-dimensional vector method, the number of extended vectors to be generated may be calculated, for example, in response to the ratio between (extended left end azimuth angle - extended right end azimuth angle) and (extended upper end elevation angle - extended lower end elevation angle).
[0243] (Diffuse Radiation Vector Method)
[0244] Furthermore, the extended radiation vector method is described.
[0245] In the extended radiation vector method, an extended radiation vector, a three-dimensional vector, is stored in the bitstream and transmitted along with the bitstream. Here, it is assumed that the extended radiation vector is stored in the metadata of each audio signal frame for each object, for example. In this case, the spread indicating the degree of diffuseness of the sound image is also stored in the metadata.
[0246] The extended radiation vector is a vector indicating the relative position of the center position p0 of the region indicating the spread of the sound image of the object to the position p of the object. For example, the extended radiation vector is a three-dimensional vector composed of the following three factors: the azimuth angle indicating the horizontal direction angle of the center position p0, the elevation angle indicating the vertical direction angle of the center position p0, and the radius indicating the distance in the radial direction from the center position p0, as viewed from the position p.
[0247] In other words, diffuse radiation vector = (azimuth, elevation, radius).
[0248] When rendering, the position indicated by the vector obtained by adding the extended radiant vector and the vector p is determined as the center position p0, and the extended vectors p0 to p18 are calculated as extended vectors. Here, for example, Figure 5 As shown in FIG, the extended vector p0 is a vector pO whose starting point is the origin O and whose end point is the center position pO. It should be noted that Figure 5 In, with Figure 5 Corresponding parts in the case of are denoted by the same reference numerals, and descriptions of these parts will be appropriately omitted.
[0249] In addition, Figure 5In the example, the dotted arrows represent the expansion vectors, and similarly in Figure 5 In order to make the figure easier to see, only nine expansion vectors are depicted.
[0250] However, in Figure 3 In the example depicted, position p = center position PO, and Figure 5 In the depicted example, the center position pO is a position different from the position p. In this example, the end position of the vector obtained by vector addition of the vector p and the extended radiation vector indicated by the arrow mark B11 is the center position pO.
[0251] Furthermore, it can be recognized that the region R31 indicating the spread of the sound image and centered at the center position p0 exceeds the position p of the target position. Figure 3 The left side of the example is shifted to Figure 5 on the left side of the .
[0252] If an arbitrary position can be specified as the center position pO of the area indicating the spread of the sound image using the expanded radiation vector and the position p in this way, the directionality of the sound of the object can be expressed more accurately.
[0253] In the extended radiation vector method, if extended vectors p0 to p18 are obtained, processing B1 is performed on the vector p, and processing B2 is performed on the extended vectors p0 to p18.
[0254] Note that in process B2, VBAP gains may be calculated for 19 extension vectors, or may be calculated only for extension vectors p1 to p18 excluding extension vector p0. In the following description, it is assumed that VBAP gains are also calculated for extension vector p0.
[0255] Furthermore, if the VBAP gain of each vector is calculated, processes B3, B4, and B5' are performed to generate an audio signal to be supplied to the speaker. Note that after process B3, quantization of each VBAP gain addition value is performed as needed.
[0256] Furthermore, by such a diffuse radiation vector method as described above, sufficiently high-quality sound can be obtained by rendering.
[0257] (Arbitrary Extension Vector Method)
[0258] Subsequently, the arbitrary extension vector method is described.
[0259] In the arbitrary spread vector method, spread vector quantity information indicating the number of spread vectors used to calculate the VBAP gain and spread vector position information indicating the end point position of each spread vector are stored in the bitstream and transmitted along with the bitstream. Here, it is assumed that the spread vector quantity information and spread vector position information are stored in metadata for each audio signal frame of each object, for example. In this case, the spread indicating the degree of diffuseness of the sound image is not stored in the metadata.
[0260] At the time of rendering processing, based on the extended vector position information of each piece, a vector whose starting point is the origin O and whose end point is the position indicated by the extended vector position information is calculated as an extended vector.
[0261] After that, process B1 is performed on vector p, and process B2 is performed on each expanded vector. Furthermore, after calculating the VBAP gain for each vector, processes B3, B4, and B5' are performed to generate an audio signal to be supplied to the speaker. It should be noted that after process B3, each VBAP gain sum is quantized as needed.
[0262] According to such an arbitrary extension vector method as described above, the diffusion to which the sound image is to be extended and the shape of the diffusion can be arbitrarily specified, and thus sufficiently high-quality sound can be obtained by rendering.
[0263] <Switch of Processing>
[0264] In the present technology, it is possible to select appropriate processing as processing when performing rendering in response to the hardware scale of the renderer or the like, and obtain the highest quality sound within a spread of an allowable processing amount.
[0265] Specifically, in this technology, in order to enable switching between multiple processes, an index for process switching is stored in the bitstream and sent from the encoding device to the decoding device together with the bitstream. In other words, the index value for switching the process is added to the bitstream syntax.
[0266] For example, the following processing is performed in response to the value of the index value index.
[0267] Specifically, when the index value index=0, the decoding device, more specifically, a renderer in the decoding device, performs rendering similar to that performed in the case of the conventional MPEG-H 3D Audio standard.
[0268] On the other hand, for example, when the index value is 1, among combinations of indices indicating 18 extension vectors according to the conventional MPEG-H 3D Audio standard, a predetermined combination of indices is stored in the bitstream and transmitted along with the bitstream. In this case, the renderer calculates the VBAP gain associated with the extension vector indicated by each index stored in the bitstream and transmitted along with the bitstream.
[0269] In addition, for example, when the index value index = 2, information indicating the number of extension vectors to be used in the process and an index indicating which of the 18 extension vectors indicated by the extension vector to be used for processing according to the conventional MPEG-H 3D Audio standard are stored in the bitstream and transmitted together with the bitstream.
[0270] Furthermore, for example, when the index value is 3, the rendering process is performed according to the aforementioned extended vector method, and for example, when the index value is 4, the VBAP gain addition value is binarized during the rendering process. Furthermore, for example, when the index value is 5, the rendering process is performed according to the aforementioned extended center vector method.
[0271] Furthermore, an index value index for switching a process in the encoding device may not be specified, but a process may be selected by a renderer in the decoding device.
[0272] In the case just described, for example, switching the processing based on the importance information included in the metadata of the object seems to be a recommended idea. Specifically, for example, for an object whose importance indicated by the importance information is high (equal to or higher than a predetermined value), the processing indicated by the index value index = 0 is performed. For an object whose importance indicated by the importance information is low (lower than a predetermined value), the processing indicated by the index value index = 4 can be performed.
[0273] By appropriately switching the processing when rendering is performed in this manner, it is possible to obtain the highest quality sound within the spread of the allowable processing amount in response to the hardware scale of the renderer and the like.
[0274] <Example of Configuration of Audio Processing Device>
[0275] Subsequently, more specific embodiments of the above-mentioned present technology are described.
[0276] Figure 6 is a diagram depicting an example of the configuration of an audio processing device to which the present technology is applied.
[0277] Speakers 12-1 to 12-M corresponding to M channels are connected to Figure 6The audio processing device 11 depicted in . The audio processing device 11 generates audio signals of different channels based on an audio signal and metadata of an object provided from the outside, and supplies the audio signals to the speakers 12 - 1 to 12 -M so that the speakers 12 - 1 to 12 -M reproduce sounds.
[0278] It should be noted that in the following description, where there is no need to particularly distinguish the speakers 12-1 to 12-M from one another, each of them is simply referred to as a speaker 12. Each speaker 12 is a sound output unit that outputs sound based on an audio signal supplied thereto.
[0279] The speakers 12 are arranged so as to surround a user who enjoys content, etc. For example, the speakers 12 are arranged on the above-mentioned unit sphere.
[0280] The audio processing device 11 includes an acquisition unit 21 , a vector calculation unit 22 , a gain calculation unit 23 , and a gain adjustment unit 24 .
[0281] The acquisition unit 21 acquires the audio signal of the object and metadata of each frame of the audio signal of each object from the outside. For example, the audio data and metadata are obtained by decoding the encoded audio data and encoded metadata included in the bit stream output from the encoding device by a decoding device.
[0282] The acquisition unit 21 supplies the acquired audio signal to the gain adjustment unit 24, and supplies the acquired metadata to the vector calculation unit 22. Here, the metadata includes, for example, position information indicating the position of the object, importance information indicating the degree of importance of each object, expansion indicating the spatial spread of the sound image of the object, etc., as needed.
[0283] The vector calculation unit 22 calculates an extended vector based on the metadata supplied thereto from the acquisition unit 21, and supplies the extended vector to the gain calculation unit 23. In addition, as needed, the vector calculation unit 22 supplies the position of each object indicated by the position information included in the metadata, that is, the vector p indicating the position p, to the gain calculation unit 23.
[0284] The gain calculation unit 23 calculates the VBAP gain of the speaker 12 corresponding to each channel by VBAP based on the expansion vector and the vector p supplied from the vector calculation unit 22, and supplies the VBAP gain to the gain adjustment unit 24. Furthermore, the gain calculation unit 23 includes a quantization unit 31 for quantizing the VBAP gain of each speaker.
[0285] The gain adjustment unit 24 performs gain adjustment on the audio signal of the object supplied from the acquisition unit 21 based on each VBAP gain supplied from the gain calculation unit 23 , and supplies the audio signals of M channels obtained as a result of the gain adjustment to the speaker 12 .
[0286] The gain adjustment unit 24 includes amplification units 32-1 to 32-M. The amplification units 32-1 to 32-M multiply the audio signal supplied from the acquisition unit 21 by the VBAP gain supplied from the gain calculation unit 23 and supply the audio signal obtained by the multiplication to the speakers 12-1 to 12-M to reproduce the sound.
[0287] It should be noted that in the following description, in the case where there is no need to particularly distinguish the amplification units 32 - 1 to 32 -M from one another, each of them is also referred to simply as the amplification unit 32 .
[0288] <Description of Reproduction Processing>
[0289] Now, describe Figure 6 The operation of the audio processing device 11 is depicted in FIG.
[0290] If an audio signal and metadata of an object are supplied from the outside, the audio processing device 11 performs a reproduction process to reproduce the sound of the object.
[0291] In the following, refer to Figure 7 The flowchart of _ describes the reproduction process of the audio processing device 11. It should be noted that this reproduction process is performed for each frame of the audio signal.
[0292] At step S11 , the acquisition unit 21 acquires the audio signal and metadata of one frame of the object from the outside, and supplies the audio signal to the amplification unit 32 , while it supplies the metadata to the vector calculation unit 22 .
[0293] At step S12, the vector calculation unit 22 performs an extended vector calculation process based on the metadata supplied from the acquisition unit 21, and supplies the extended vector obtained as a result of the extended vector calculation process to the gain calculation unit 23. In addition, the vector calculation unit 22 also supplies the vector p to the gain calculation unit 23 as needed.
[0294] It should be noted that although the details of the extended vector calculation processing are described below, in the extended vector calculation processing, the extended vector is calculated by the extended three-dimensional vector method, the extended center vector method, the extended end vector method, the extended radiation vector method or the arbitrary extended vector method.
[0295] At step S13 , the gain calculation unit 23 calculates the VBAP gain of each speaker 12 based on the position information indicating the position of the speaker 12 stored in advance and the vector p and the expansion vector supplied from the vector calculation unit 22 .
[0296] Specifically, for each of the expansion vectors and the vector p, the VBAP gain of each speaker 12 is calculated. Thus, for each of the expansion vectors and the vector p, the VBAP gain of one or more speakers 12 located near the position of the object (i.e., near the position indicated by the vector) is obtained. It should be noted that while the VBAP gain of the expansion vector is necessarily calculated, if the vector p is not supplied from the vector calculation unit 22 to the gain calculation unit 23 by the processing at step S12, the VBAP gain of the vector p is not calculated.
[0297] At step S14, the gain calculation unit 23 adds the VBAP gains calculated for each vector to calculate a VBAP gain addition value for each speaker 12. Specifically, the addition value (sum) of the VBAP gains of the vectors calculated for the same speaker 12 is calculated as the VBAP gain addition value.
[0298] At step S15 , the quantization unit 31 decides whether to perform binarization of the VBAP gain addition value.
[0299] For example, whether or not to perform binarization may be decided based on the index value index described above, or may be decided based on the degree of importance of the object indicated by importance information as metadata.
[0300] If the decision is performed based on the index value index, for example, the index value index read from the bitstream may be provided to the gain calculation unit 23. Alternatively, if the decision is performed based on the importance information, the importance information may be provided from the vector calculation unit 22 to the gain calculation unit 23.
[0301] If it is decided at step S15 that binarization is to be performed, then at step S16, the quantization unit 31 binarizes the added value of the VBAP gain (ie, the VBAP gain added value) decided for each speaker 12. Thereafter, the process proceeds to step S17.
[0302] In contrast, if it is decided at step S15 not to perform binarization, the processing at step S16 is skipped and the process proceeds to step S17.
[0303] At step S17 , the gain calculation unit 23 normalizes the VBAP gain of each speaker 12 so that the sum of the squares of the VBAP gains of all the speakers 12 may become 1.
[0304] Specifically, the added value of the VBAP gain decided for each speaker 12 is normalized so that the square sum of all the added values may become 1. The gain calculation unit 23 supplies the VBAP gain of the speaker 12 obtained by normalization to the amplification unit 32 corresponding to each speaker 12 .
[0305] At step S18 , the amplification unit 32 multiplies the audio signal supplied from the acquisition unit 21 by the VBAP gain supplied from the gain calculation unit 23 , and supplies the resultant value to the speaker 12 .
[0306] Then at step S19, the amplification unit 32 causes the speaker 12 to reproduce sound based on the audio signal supplied thereto, thereby ending the reproduction process. Thus, the sound image of the object is localized in the desired partial space in the reproduction space.
[0307] In the manner described above, the audio processing device 11 calculates the expansion vector based on the metadata, calculates the VBAP gain for each vector for each speaker 12, and determines and normalizes the added value of the VBAP gain for each speaker 12. By calculating the VBAP gain with respect to the expansion vector in this manner, the spatial spread of the sound image of the object, particularly the shape of the object or the directionality of the sound, can be expressed, and higher quality sound can be obtained.
[0308] Furthermore, by binarizing the added value of the VBAP gain as needed, not only can the amount of processing during rendering be reduced, but appropriate processing can also be performed in response to the processing capability (hardware scale) of the audio processing device 11 to obtain the highest possible quality sound.
[0309] <Description of Extended Vector Calculation Processing>
[0310] Here, refer to Figure 8 The flowchart to describe Figure 7 The processing at step S12 corresponds to the extended vector calculation processing.
[0311] At step S41 , the vector calculation unit 22 decides whether to calculate an extended vector based on the extended three-dimensional vector.
[0312] For example, with Figure 7 Similar to the case at step S15, which method to use to calculate the extended vector may be decided based on the index value index, or may be decided based on the importance degree of the object indicated by the importance information.
[0313] If it is decided at step S41 that the extended vector is to be calculated based on the extended three-dimensional vector, that is, if it is decided that the extended vector is to be calculated by the extended three-dimensional method, the process proceeds to step S42.
[0314] At step S42, the vector calculation unit 22 performs an extended vector calculation process based on the extended three-dimensional vector and supplies the resulting vector to the gain calculation unit 23. It should be noted that details of the extended vector calculation process based on the extended three-dimensional vector are described below.
[0315] After the extended vector is calculated, the extended vector calculation process ends, after which the process proceeds to Figure 7 Step S13.
[0316] On the other hand, if it is decided at step S41 not to calculate the extended vector based on the extended three-dimensional vector, the process proceeds to step S43.
[0317] At step S43 , the vector calculation unit 22 decides whether to calculate an extended vector based on the extended center vector.
[0318] If it is decided at step S43 that the extended vector is to be calculated based on the extended center vector, that is, if it is decided that the extended vector is to be calculated by the extended center vector method, the process proceeds to step S44.
[0319] At step S44, the vector calculation unit 22 performs an extended vector calculation process based on the extended center vector, and supplies the resultant vector to the gain calculation unit 23. It should be noted that the details of the extended vector calculation process based on the extended center vector are described below.
[0320] After the extended vector is calculated, the extended vector calculation process ends, after which the process proceeds to Figure 7 Step S13.
[0321] On the other hand, if it is decided at step S43 not to calculate the extended vector based on the extended center vector, the process proceeds to step S45.
[0322] At step S45 , the vector calculation unit 22 decides whether to calculate an extended vector based on the extended end vector.
[0323] If it is decided at step S45 that the extended vector is to be calculated based on the extended end vector, that is, if it is decided that the extended vector is to be calculated by the extended end vector method, the process proceeds to step S46.
[0324] At step S46, the vector calculation unit 22 performs an extended vector calculation process based on the extended side vector, and supplies the resultant vector to the gain calculation unit 23. It should be noted that the details of the extended vector calculation process based on the extended side vector are described below.
[0325] After the extended vector is calculated, the extended vector calculation process ends, after which the process proceeds to Figure 7Step S13.
[0326] Furthermore, if it is decided at step S45 not to calculate the extended vector based on the extended end vector, the process proceeds to step S47.
[0327] At step S47 , the vector calculation unit 22 decides whether to calculate an extension vector based on the extended radiation vector.
[0328] If it is decided at step S47 that the extended vector is to be calculated based on the extended radiation vector, that is, if it is decided that the extended vector is to be calculated by the extended radiation vector method, the process proceeds to step S48.
[0329] At step S48, the vector calculation unit 22 performs an extended vector calculation process based on the extended radiation vector, and supplies the resultant vector to the gain calculation unit 23. It should be noted that details of the extended vector calculation process based on the extended radiation vector are described below.
[0330] After the extended vector is calculated, the extended vector calculation process ends, after which the process proceeds to Figure 7 Step S13.
[0331] On the other hand, if it is decided at step S47 not to calculate the extended vector based on the extended radiant vector, that is, if it is decided to calculate the extended vector by the extended radiant vector method, the process proceeds to step S49.
[0332] At step S49, the vector calculation unit 22 performs extended vector calculation processing based on the extended vector position information, and supplies the resulting vector to the gain calculation unit 23. It should be noted that details of the extended vector calculation processing based on the extended vector position information are described below.
[0333] After the extended vector is calculated, the extended vector calculation process ends, after which the process proceeds to Figure 7 Step S13.
[0334] The audio processing device 11 calculates the extension vector by an appropriate method among various methods in this manner. By calculating the extension vector by an appropriate method in this manner, the highest quality sound can be obtained within the range of the allowable processing amount in response to the hardware scale of the renderer, etc.
[0335] <Description of Extended Vector Calculation Processing Based on Extended Three-Dimensional Vector>
[0336] Now, refer to Figure 8 Details of the processing corresponding to the processing at steps S42, S44, S46, S48, and S49 described above will be described.
[0337] First, refer to Figure 9 The flowchart to describe Figure 8 Step S42 corresponds to the extended vector calculation processing based on the extended three-dimensional vector.
[0338] At step S81, the vector calculation unit 22 determines, as the object position p, the position indicated by the position information included in the metadata supplied from the acquisition unit 21. In other words, the vector representing the position p is the vector p.
[0339] At step S82, the vector calculation unit 22 calculates the extension based on the extended three-dimensional vector included in the metadata supplied from the acquisition unit 21. Specifically, the vector calculation unit 22 calculates Expression (1) given above to calculate the extension.
[0340] At step S83 , the vector calculation unit 22 calculates extended vectors p0 to p18 based on the vector p and the extension.
[0341] Here, vector p is determined as vector p0 representing the center position p0, and vector p is determined as is as the extended vector p0. Furthermore, similar to the MPEG-H 3D Audio standard, the extended vectors p1 to p18 are calculated to be symmetrical in the vertical and horizontal directions within an area centered on the center position p0 and defined by the angle indicated by the extension on a unit sphere.
[0342] At step S84 , the vector calculation unit 22 makes a decision based on whether the extended three-dimensional vector satisfies s3_azimuth≧s3_elevation, that is, whether s3_azimuth is greater than s3_elevation.
[0343] If it is determined at step S84 that s3_azimuth≥s3_elevation is satisfied, then at step S85, the vector calculation unit 22 changes the elevation angles of the extension vectors p1 to p18. Specifically, the vector calculation unit 22 performs the calculation of Expression (2) described above to correct the elevation angles of the extension vectors to obtain the final extension vectors.
[0344] After obtaining the final extended vector, the vector calculation unit 22 supplies the extended vectors p0 to p18 to the gain calculation unit 23, thereby ending the extended vector calculation process based on the extended three-dimensional vector. Figure 8 The processing at step S42 of the embodiment ends therewith, so the subsequent processing proceeds to Figure 7 Step S13.
[0345] On the other hand, if it is determined at step S84 that s3_azimuth≥s3_elevation is not satisfied, then at step S86, the vector calculation unit 22 changes the azimuth of the extended vectors p1 to p18. Specifically, the vector calculation unit 22 performs the calculation of the expression (3) given above to correct the azimuth of the extended vector, thereby obtaining the final extended vector.
[0346] After obtaining the final extended vector, the vector calculation unit 22 supplies the extended vectors p0 to p18 to the gain calculation unit 23, thereby ending the extended vector calculation process based on the extended three-dimensional vector. Figure 8 The processing at step S42 is completed, so the subsequent processing proceeds to Figure 7 Step S13.
[0347] The audio processing device 11 calculates each extended vector by the extended three-dimensional vector method in such a manner as described above. Therefore, it becomes possible to express the shape of the object and the directionality of the sound of the object, and obtain higher-quality sound.
[0348] <Description of the Extended Vector Calculation Processing Based on the Extended Center Vector>
[0349] Now, refer to Figure 10 The flowchart to describe Figure 8 Step S44 corresponds to the extended vector calculation process based on the extended center vector.
[0350] It should be noted that the processing at step S111 is the same as Figure 9 The processing at step S81 is similar, so its description is omitted.
[0351] At step S112 , the vector calculation unit 22 calculates the extended vectors p0 to p18 based on the extended center vector and the extension included in the metadata supplied from the acquisition unit 21 .
[0352] Specifically, the vector calculation unit 22 sets the position indicated by the expansion center vector as the center position p0, and sets the vector representing the center position p0 as the expansion vector p0. Furthermore, the vector calculation unit 22 determines the expansion vectors p1 to p18 so that they are symmetrically positioned in the vertical and horizontal directions within an area centered on the center position p0 and defined by the angle indicated by the expansion on a unit sphere. The expansion vectors p1 to p18 are determined in a manner substantially similar to that of the MPEG-H 3D Audio standard.
[0353] The vector calculation unit 22 supplies the extended vectors p0 to p18 and the vector p obtained by the above-mentioned processing to the gain calculation unit 23, thereby ending the extended vector calculation processing based on the extended center vector. Figure 8 The processing at step S44 ends, and thereafter the processing proceeds to Figure 7 Step S13.
[0354] The audio processing device 11 calculates the vector p and the extended vector by the extended center vector method in such a manner as described above. Therefore, it becomes possible to express the shape of the object and the directionality of the sound of the object, and obtain higher quality sound.
[0355] It should be noted that, in the extended vector calculation process based on the extended center vector, the extended vector p0 may not be supplied to the gain calculation unit 23. In other words, the VBAP gain may not be calculated with respect to the extended vector p0.
[0356] <Description of Extended Vector Calculation Processing Based on Extended End Vector>
[0357] In addition, refer to Figure 11 The flowchart to describe Figure 8 Step S46 corresponds to the extended vector calculation processing based on the extended end vector.
[0358] It should be noted that the processing at step S141 is the same as Figure 9 The processing at step S81 is similar, so its description is omitted.
[0359] At step S142, the vector calculation unit 22 calculates the center position pO, that is, the vector pO, based on the extended end vector included in the metadata supplied from the acquisition unit 21. Specifically, the vector calculation unit 22 calculates Expression (4) given above to calculate the center position pO.
[0360] At step S143, the vector calculation unit 22 calculates the extension based on the extension end vector. Specifically, the vector calculation unit 22 calculates the expression (5) given above to calculate the extension.
[0361] At step S144 , the vector calculation unit 22 calculates the extension vectors p0 to p18 based on the center position p0 and the extension.
[0362] Here, vector p0 representing center position p0 is set as is as the extension vector p0. Furthermore, similar to the MPEG-H 3D Audio standard, extension vectors p1 to p18 are calculated so that they are symmetrically positioned in the vertical and horizontal directions within the area centered on center position p0 and defined by the angle indicated by the extension on a unit sphere.
[0363] At step S145, the vector calculation unit 22 determines whether (extended left end azimuth angle - extended right end azimuth angle) ≥ (extended upper end elevation angle - extended lower end elevation angle), that is, whether (extended left end azimuth angle - extended right end azimuth angle) is greater than (extended upper end elevation angle - extended lower end elevation angle).
[0364] If it is determined at step S145 that (extended left end azimuth angle - extended right end azimuth angle) ≥ (extended upper end elevation angle - extended lower end elevation angle) is satisfied, then at step S146, the vector calculation unit 22 changes the elevation angles of the extended vectors p1 to p18. Specifically, the vector calculation unit 22 performs the calculation of Expression (6) given above to correct the elevation angles of the extended vectors to obtain the final extended vectors.
[0365] After obtaining the final extended vector, the vector calculation unit 22 supplies the extended vectors p0 to p18 and the vector p to the gain calculation unit 23, thereby ending the extended vector calculation process based on the extended end vector. Figure 8 The processing at step S46 ends, and thereafter the processing proceeds to Figure 7 Step S13.
[0366] On the other hand, if it is determined at step S145 that (extended left end azimuth angle - extended right end azimuth angle) ≥ (extended upper end elevation angle - extended lower end elevation angle) is not satisfied, then at step S147, the vector calculation unit 22 changes the azimuth angles of the extended vectors p1 to p18. Specifically, the vector calculation unit 22 performs the calculation of Expression (7) given above to correct the azimuth angles of the extended vectors to obtain the final extended vectors.
[0367] After obtaining the final extended vector, the vector calculation unit 22 supplies the extended vectors p0 to p18 and the vector p to the gain calculation unit 23, thereby ending the extended vector calculation process based on the extended end vector. Figure 8 The processing at step S46 ends, and thereafter the processing proceeds to Figure 7 Step S13.
[0368] The audio processing device 11 calculates the extended vector by the extended end vector method as described above. Therefore, it becomes possible to express the shape of the object and the directionality of the sound of the object, and obtain higher quality sound.
[0369] It should be noted that, in the extended vector calculation process based on the extended end vector, the extended vector p0 may not be supplied to the gain calculation unit 23. In other words, the VBAP gain may not be calculated with respect to the extended vector p0.
[0370] <Description of Extended Vector Calculation Processing Based on Extended Radiation Vector>
[0371] Now, refer to Figure 12 The flowchart to describe Figure 8 Step S48 corresponds to the extended vector calculation process based on the extended radiation vector.
[0372] It should be noted that the processing at step S171 is the same as Figure 9 The processing at step S81 is similar to that at step S81, and therefore, the description of this processing is omitted.
[0373] At step S172 , the vector calculation unit 22 calculates the extension vectors p0 to p18 based on the extension and extension radiance vectors included in the metadata supplied from the acquisition unit 21 .
[0374] Specifically, the vector calculation unit 22 sets the position indicated by the vector obtained by adding the vector p representing the object position p to the radiation vector as the center position p0. The vector indicating this center position p0 is the vector p0, and the vector calculation unit 22 sets the vector p0 as it is as the expansion vector p0.
[0375] Furthermore, the vector calculation unit 22 determines the extension vectors p1 to p18 so that they are positioned symmetrically in the vertical and horizontal directions within a region centered at the center position p0 and defined by the angle indicated by the extension on the unit sphere. The extension vectors p1 to p18 are determined substantially similarly to the case of the MPEG-H 3D Audio standard.
[0376] The vector calculation unit 22 supplies the expansion vectors p0 to p18 and the vector p obtained by the above process to the gain calculation unit 23, thereby ending the expansion vector calculation process based on the expansion radiation vector. Figure 8 The processing at step S48 is completed, so the subsequent processing proceeds to Figure 7 Step S13.
[0377] The audio processing device 11 calculates the vector p and the expansion vector by the extended radiation vector method in such a manner as described above. Therefore, it becomes possible to express the shape of the object and the directionality of the sound of the object, and obtain higher quality sound.
[0378] It should be noted that, in the extended vector calculation process based on the extended radiation vector, the extended vector p0 may not be provided to the gain calculation unit 23. In other words, the VBAP gain may not be calculated with respect to the extended vector p0.
[0379] <Description of Extended Vector Calculation Processing Based on Extended Vector Position Information>
[0380] Now, refer to Figure 13 The flowchart to describe Figure 8Step S49 corresponds to the extended vector calculation processing based on the extended vector position information.
[0381] It should be noted that the processing at step S201 is the same as Figure 9 The processing at step S81 is similar, so its description is omitted.
[0382] At step S202 , the vector calculation unit 22 calculates an extended vector based on the extended vector number information and the extended vector position information included in the metadata supplied from the acquisition unit 21 .
[0383] Specifically, the vector calculation unit 22 calculates a vector having a start point at the origin O and an end point at a position indicated by the extended vector position information as an extended vector. Here, the number of extended vectors equal to the number indicated by the extended vector number information is calculated.
[0384] The vector calculation unit 22 supplies the extended vector and the vector p obtained by the above process to the gain calculation unit 23, thereby ending the extended vector calculation process based on the extended vector position information. Figure 8 The processing at step S49 is completed, so the subsequent processing proceeds to Figure 7 Step S13.
[0385] The audio processing device 11 calculates the vector p and the expansion vector by the arbitrary expansion vector method in such a manner as described above. Therefore, it becomes possible to express the shape of the object and the directionality of the sound of the object, and obtain higher-quality sound.
[0386] <Second embodiment>
[0387] <Reduction in rendering processing>
[0388] Incidentally, as described above, VBAP is known as a technology for controlling the localization of a sound image using a plurality of speakers, that is, a technology for performing rendering processing.
[0389] In VBAP, by outputting sound from three speakers, the sound image can be localized at any point on the inside of a triangle formed by the three speakers. In the following, in particular, a triangle formed by such three speakers is referred to as a mesh.
[0390] Since VBAP performs rendering processing for each object, the amount of processing required for rendering is large when the number of objects is large (e.g., in a game). Therefore, a renderer with limited hardware may not be able to render all objects, and thus may only be able to reproduce the sounds of a limited number of objects. This may impair the rendering or sound quality when reproducing the sound.
[0391] Therefore, the present technology makes it possible to reduce the processing amount of rendering processing while suppressing degradation in presentation or sound quality.
[0392] In the following, such a technique as just described is described.
[0393] In the normal VBAP process, that is, in the rendering process, the processes A1 to A3 described above are performed on each object to generate an audio signal for a speaker.
[0394] Since the number of speakers for which VBAP gains are generally calculated is three and the VBAP gain of each speaker is calculated for each sample constituting the audio signal, in the multiplication processing in process A3, multiplication is performed a number of times equal to (number of samples of the audio signal × 3).
[0395] In contrast, in the present technology, the processing amount of rendering processing is reduced by performing equal gain processing on the VBAP gain, that is, quantization processing of the VBAP gain and mesh number switching processing for changing the number of meshes to be used in VBAP gain calculation in an appropriate combination.
[0396] (Quantization processing)
[0397] First, the quantization process is described. Here, as examples of the quantization process, the binarization process and the ternary process are described.
[0398] When binarization is performed as the quantization process, after the process A1 is performed, the VBAP gain of each speaker obtained by the process A1 is binarized. In the binarization, for example, the VBAP gain of each speaker is represented by one of 0 and 1.
[0399] It should be noted that the method for binarizing the VBAP gain may be any method such as rounding, capping (rounding up), flooring (truncating), or thresholding.
[0400] After the VBAP gain is binarized in this manner, processing A2 and processing A3 are performed to generate an audio signal for the speaker.
[0401] At this time, in process A2, since normalization is performed based on the binarized VBAP gain, similarly to the quantization of the extension vector described above, the final VBAP gain of the speaker becomes a value other than 0. In other words, if the VBAP gain is binarized, the value of the final VBAP gain of the speaker is 0 or a predetermined value.
[0402] Therefore, in the multiplication process in process A3, multiplication can be performed (the number of samples of the audio signal×1) times, and thus the processing amount of the rendering process can be significantly reduced.
[0403] Similarly, after process A1, the obtained VBAP gain of the speaker can be ternary. In such a case as just described, the VBAP gain of each speaker obtained by process A1 is ternary to one of 0, 0.5, and 1. Then, processes A2 and A3 are performed thereafter to generate the audio signal of the speaker.
[0404] Therefore, since the number of multiplication times in the multiplication process in the process A3 becomes the maximum value (the number of samples of the audio signal×2), the processing amount of the rendering process can be significantly reduced.
[0405] It should be noted that although the description given here uses the case where the VBAP gain is binarized or ternary as an example, the VBAP gain can be quantized to 4 or more values. To summarize this, for example, the VBAP gain is quantized so that it has one of x gains equal to or greater than 2, or in other words, if the VBAP gain is quantized x times, the number of multiplication processes in process A3 becomes the maximum value (x-1).
[0406] By quantizing the VBAP gain in this manner, the amount of processing required for rendering can be reduced. By reducing the amount of processing required for rendering in this manner, even when the number of objects is large, rendering can be performed on all objects, thereby minimizing degradation in rendering or sound quality during sound reproduction. In other words, the amount of processing required for rendering can be reduced while minimizing degradation in rendering or sound quality.
[0407] (Grid quantity switching process)
[0408] Now, the mesh number switching process is described.
[0409] In VBAP, as described above, for example, with reference to Figure 1 , a vector p indicating the position p of the sound image of the object of the processing target is expressed by the linear sum of vectors I1 to I3 pointing in the directions of the three speakers SP1 to SP3, and the coefficients g1 to g3 multiplied by the vectors are the VBAP gains of the speakers. Figure 1 In the example of FIG, a triangular region TR11 surrounded by the speakers SP1 to SP3 forms a mesh.
[0410] When calculating the VBAP gain, specifically, the following expression (8) is used to calculate the inverse matrix L of the triangular mesh. 123 -1 And the calculation of the position p of the sound image of the object determines the three coefficients g1 to g3:
[0411] [Expression 8]
[0412]
[0413] It should be noted that p1, p2, and p3 in Expression (8) indicate the Cartesian coordinate system (ie, Figure 2 The x-coordinate, y-coordinate, and z-coordinate of the position of the sound image of the object (on the three-dimensional coordinate system depicted in ).
[0414] In addition, I 11 , I 12 and I 13 are the values of the x component, y component and z component when the vector I1 pointing to the first speaker SP1 constituting the grid is decomposed into components on the x-axis, y-axis and z-axis, and correspond to the x coordinate, y coordinate and z coordinate of the first speaker SP1, respectively.
[0415] Similarly, I 21 , I 22 and I 23 are the values of the x-component, y-component, and z-component, respectively, when the vector I2 pointing to the second speaker SP2 constituting the grid is decomposed into the component on the x-axis, the component on the y-axis, and the component on the z-axis. 31 , I 32 and I 33 1 and 2 are values of an x-component, a y-component, and a z-component, respectively, when a vector I3 directed toward the third speaker SP3 constituting the mesh is decomposed into a component on the x-axis, a component on the y-axis, and a component on the z-axis.
[0416] Furthermore, as represented by the following expression (9), the conversion from p1, p2, and p3 of the three-dimensional coordinate system of the position p to the coordinates θ, γ, and r of the spherical coordinate system is defined, where r = 1, as shown in the following expression (9). Here, θ, γ, and r are the horizontal direction angle azimuth, vertical direction angle elevation, and distance radius described above, respectively.
[0417] [Expression 9]
[0418] [p1 p2 p3]=[cos(θ)×cos(γ)sin(θ)×cos(γ)sin(γ)]…(9)
[0419] As described above, in the space on the content reproduction side, that is, in the reproduction space, multiple speakers are arranged on a unit sphere, and one grid is composed of three speakers from the multiple speakers. Furthermore, the multiple grids substantially cover the entire surface of the unit sphere, with no gaps left between the multiple grids. Furthermore, the grids are determined so that they do not overlap with each other.
[0420] In VBAP, if sound is output from two or three speakers arranged on the surface of a unit sphere so as to form a grid including the position p of the object, the sound image can be localized at the position p, and therefore, the VBAP gain of the speakers other than the speakers forming the grid is 0.
[0421] Therefore, when calculating the VBAP gain, a grid including the position p of the object can be specified to calculate the VBAP gain of the speakers constituting the grid. For example, it can be determined from the calculated VBAP gain whether the predetermined grid is the grid including the position p.
[0422] Specifically, if the VBAP gains of the three speakers calculated for the grid are all equal to or greater than 0, the grid is a grid that includes the object's position p. Conversely, if at least one of the VBAP gains of the three speakers has a negative value, the calculated VBAP gain is not a correct VBAP gain because the object's position p is outside the grid formed by the speakers.
[0423] Therefore, when calculating the VBAP gain, meshes are selected one by one as meshes of the processing target, and calculation of Expression (8) given above is performed on the meshes of the processing target to calculate the VBAP gain of each speaker constituting the mesh.
[0424] Then, based on the calculation result of the VBAP gain, it is decided whether the grid of the processing target is a grid containing the position p of the object. If it is decided that the grid of the processing target is a grid that does not include the position p, the next grid is determined as the new processing target grid, and similar processing is performed on the grid.
[0425] On the other hand, if the mesh targeted for processing is a mesh including the position p of the object, the VBAP gains of the speakers constituting the mesh are determined as the calculated VBAP gains, and the VBAP gains of the other speakers are set to 0. Thus, the VBAP gains of all speakers are obtained.
[0426] In this manner, in the rendering process, the process for calculating the VBAP gain and the process for specifying the mesh including the position p are simultaneously performed.
[0427] Specifically, in order to obtain a correct VBAP gain, the following processes are repeated: successively selecting a mesh of a processing target until all VBAP gains of speakers constituting the mesh indicate a value equal to or higher than 0; and calculating the VBAP gain of the mesh.
[0428] Therefore, in the rendering process, the processing amount of the process required to specify the mesh including the position p in order to obtain the correct VBAP gain increases as the number of meshes on the surface of the unit sphere increases.
[0429] Therefore, in this technology, instead of all speakers in the actual reproduction environment being used to form (compose) a mesh, only some of the speakers are used to form the mesh, thereby reducing the total number of meshes and the amount of processing required during rendering. Specifically, in this technology, a mesh number switching process is performed to change the total number of meshes.
[0430] Specifically, for example, in a 22-channel speaker system, Figure 14 As depicted in FIG, a total of 22 speakers including speakers SPK1 to SPK22 are provided as speakers of different channels on the surface of the unit sphere. It should be noted that in Figure 14 In the equation, the origin O and Figure 2 corresponds to the origin O depicted in .
[0431] In the case where 22 speakers are arranged on the surface of the unit sphere in this manner, if a grid is formed using all 22 speakers so that they cover the unit sphere surface, the total number of grids on the unit sphere is 40.
[0432] On the contrary, Figure 15 As depicted in FIG, for example, it is assumed that among a total of 22 speakers SPK1 to SPK22, only six speakers, namely, speakers SPK1, SPK6, SPK7, SPK10, SPK19, and SPK20, are used to form a mesh. Figure 15 In the figure, the same reference numerals are used to represent Figure 14 , and therefore their descriptions are appropriately omitted.
[0433] exist Figure 15 In the example of , since only a total of 6 speakers out of 22 speakers are used to form a grid, the total number of grids on the unit sphere is 8, and the total number of grids can be significantly reduced. Figure 15 In the example depicted in Figure 14 Compared to the case where all 22 speakers depicted in FIG are used to form a mesh, the amount of processing when calculating the VBAP gain can be reduced to 8 / 40 times, and the amount of processing can be significantly reduced.
[0434] In this example, it should be noted that since the entire surface of the unit sphere is covered by eight grids without any gaps, the sound image can be localized at any position on the surface of the unit sphere. However, since the area of each grid decreases as the total number of grids arranged on the surface of the unit sphere increases, the localization of the sound image can be controlled with higher precision as the total number of grids increases.
[0435] If the total number of grids is changed through the grid number switching process, when selecting speakers to form the changed number of grids, it is desirable to select speakers whose positions in the vertical direction (up and down direction) (i.e., their positions in the direction of the vertical angle elevation) are different from each other as viewed by a user at the origin O. In other words, it is desirable to use three or more speakers, including speakers located at different heights, to form the changed number of grids. The reason for this is to suppress degradation of the three-dimensional sense (i.e., the presence of sound).
[0436] For example, consider Figure 16 In the case where some or all of the five speakers including the speakers SP1 to SP5 arranged on the surface of the unit sphere are used to form a grid, it should be noted that Figure 16 In the figure, the same reference numerals are used to represent Figure 3 , and therefore their descriptions are omitted.
[0437] exist Figure 16 In the example depicted in FIG, in which all five speakers SP1 to SP5 are used to form a mesh covering the surface of the unit sphere, the number of meshes is 3. Specifically, three areas including a triangular area surrounded by the speakers SP1 to SP3, another triangular area surrounded by the speakers SP2 to SP4, and yet another triangular area surrounded by the speakers SP2, SP4, and SP5 form the mesh.
[0438] On the other hand, if only speakers SP1, SP2, and SP5 are used, the mesh does not form a triangular shape but a two-dimensional arc. In this case, the sound image of the object can be positioned only on the arc connecting the speakers SP1 and SP2 of the unit sphere or the arc connecting the speakers SP2 and SP5 of the unit sphere.
[0439] In this way, if all speakers forming the mesh are speakers at the same height in the vertical direction (ie, speakers on the same floor), there is degradation because the heights of the localization positions of all sound images of the object become the same height.
[0440] Therefore, it is desirable to form one or more grids using three or more speakers, including speakers whose positions in the vertical direction (vertical direction) are different from each other, so that the degradation in existence can be suppressed.
[0441] exist Figure 16In the example of , for example, if the speaker SP1 and the speakers SP3 to SP5 are used from among the speakers SP1 to SP5, two grids can be formed so that they cover the entire unit sphere surface. In this example, the speakers SP1 and SP5 and the speakers SP3 and SP4 are located at different heights from each other.
[0442] In this case, for example, a region in a triangle shape surrounded by the speakers SP1 , SP3 , and SP5 and another region in a triangle shape surrounded by the speakers SP3 to SP5 are formed as a mesh.
[0443] Furthermore, in this example, two areas including a triangular-shaped area surrounded by the speakers SP1 , SP3 , and SP4 and another triangular-shaped area surrounded by the speakers SP1 , SP4 , and SP5 may also be formed as a mesh.
[0444] In the above two examples, since the sound image can be localized at any position on the surface of the unit sphere, the degradation of the existence can be suppressed. In addition, in order to form a grid so that the entire unit sphere surface is covered by multiple grids, it is certainly desirable to use a so-called top speaker located just above the user. For example, the top speaker is Figure 14 The speaker pictured is the SPK19.
[0445] By performing mesh number switching processing to change the total number of meshes in a manner similar to the quantization process described above, the amount of processing required for rendering can be reduced, and furthermore, degradation of rendering or sound quality during sound reproduction can be suppressed to a minimum. In other words, the amount of processing required for rendering can be reduced while also suppressing degradation of rendering or sound quality.
[0446] Selecting whether or not such a mesh number switching process is to be performed or to which number the total number of meshes is to be set in the mesh number switching process can be regarded as selecting the total number of meshes to be used for calculating the VBAP gain.
[0447] (Combination of quantization processing and grid number switching processing)
[0448] In the foregoing description, as a technique for reducing the processing amount of the rendering process, the quantization process and the mesh number switching process are described.
[0449] On the renderer side that performs rendering processing, some processing described as quantization processing or mesh number switching processing may be fixedly used, or such processing may be switched or may be appropriately combined.
[0450] For example, which processing to perform in combination may be determined based on the total number of objects (hereinafter referred to as the object number), importance information included in metadata of the objects, sound pressure of the audio signals of the objects, etc. In addition, a combination of processing, that is, switching of processing, may be performed for each object or each frame of the audio signal.
[0451] For example, in the case of performing switching of processing in response to the number of objects, such processing as described below can be performed.
[0452] For example, when the number of subjects is equal to or greater than 10, the binarization process of the VBAP gain is performed for all subjects. In contrast, when the number of subjects is less than 10, only the processes A1 to A3 described above are performed as usual.
[0453] By performing processing as usual when the number of objects is small, but performing binarization processing in this way when the number of objects is large, rendering can be sufficiently performed even by a renderer with a small hardware scale, and sound of the highest possible quality can be obtained.
[0454] Furthermore, when switching of the process is performed in response to the number of objects, mesh number switching processing may be performed in response to the number of objects to appropriately change the total number of meshes.
[0455] In this case, for example, when the number of objects is equal to or greater than 10, the total number of meshes may be set to 8, and when the number of objects is less than 10, the total number of meshes may be set to 40. Furthermore, the total number of meshes may be changed in a plurality of stages in response to the number of objects such that the total number of meshes decreases as the number of objects increases.
[0456] By varying the total number of meshes in response to the number of objects in this way, the amount of processing can be adjusted in response to the hardware scale of the renderer, resulting in the highest possible sound quality.
[0457] Furthermore, in the case where switching of processing is performed based on importance information included in metadata of an object, the following processing may be performed.
[0458] For example, when the importance information of the object has the highest value indicating the highest importance degree, only processes A1 to A3 are performed as usual, but in the case where the importance information of the object has a value other than the highest value, binarization processing of the VBAP gain is performed.
[0459] In addition, for example, the number of grids can be switched in response to the value of the importance information of the object to appropriately change the total number of grids. In this case, as the importance of the object increases, the total number of grids can be increased, and the total number of grids can be changed in multiple stages.
[0460] In these examples, the processing can be switched for each object based on its importance information. In the processing described here, the sound quality of highly important objects is improved, while the sound quality of less important objects is reduced, thereby reducing the amount of processing. Therefore, when simultaneously reproducing the sounds of objects of varying importance, the auditory deterioration of sound quality is minimized to reduce the amount of processing. This technology can be considered to strike a good balance between ensuring sound quality and reducing the amount of processing.
[0461] In this way, when switching of processing is performed for each object based on the importance information of the object, the total number of objects can be increased as the importance degree of the object increases, or quantization processing can be avoided when the importance degree of the object is high.
[0462] In addition, for objects with a low degree of importance, that is, for objects whose importance information values are lower than a predetermined value, the total number of grids may be increased or quantization processing may not be performed for objects located close to objects with a higher degree of importance (that is, objects whose importance information values are equal to or higher than a predetermined value).
[0463] Specifically, for an object whose importance information indicates the highest value, the total number of meshes is set to 40, but for an object whose importance information does not indicate the highest value, the total number of meshes is reduced.
[0464] In this case, for objects whose importance information is not the highest, the total number of grids can be increased as the distance between the object and the object with the highest importance information decreases. Generally, because users listen particularly carefully to the sound of objects with high importance, if the sound quality of other objects located near these objects is low, the user will perceive the overall sound quality of the content as poor. Therefore, by also determining the total number of grids for objects located near objects with high importance, the highest possible sound quality can be achieved, thereby suppressing any auditory degradation in sound quality.
[0465] In addition, the processing can be switched in response to the sound pressure of the audio signal of the object. Here, the sound pressure of the audio signal can be determined by calculating the square root of the mean square value of the sample values of the samples in the frame of the rendering target of the audio signal. Specifically, the sound pressure RMS can be determined by calculating the following expression (10):
[0466] [Expression 10]
[0467]
[0468] It should be noted that, in expression (10), N represents the number of samples constituting a frame of the audio signal, and x nRepresents the sample value of the nth (where n=0, ..., N-1) sample in the frame.
[0469] In the case where the processing is switched in response to the sound pressure RMS of the audio signal obtained in this manner, the following processing can be performed.
[0470] For example, when the sound pressure RMS of the audio signal of the object is -6dB or a larger value relative to 0dB (which is the full scale of the sound pressure RMS), only processing A1 to A3 are performed as usual, but when the sound pressure RMS of the object is lower than -6dB, binarization processing of the VBAP gain is performed.
[0471] Generally, when sounds have high sound pressure, sound quality degradation is likely to be prominent, and such sounds are often the sounds of objects with high importance. Therefore, here, sound quality degradation is prevented for objects with high RMS sound pressure, while binarization is performed for objects with low RMS sound pressure, reducing the overall processing volume. This allows for efficient rendering even with a small hardware renderer, and achieves the highest possible sound quality.
[0472] Alternatively, the mesh number switching process may be performed in response to the RMS sound pressure of the audio signal of the object, so that the total number of meshes is appropriately changed. In this case, for example, the total number of meshes may be increased as the RMS sound pressure of the object increases, and the total number of meshes may be changed in multiple stages.
[0473] Furthermore, a combination of quantization processing or grid number switching processing may be selected in response to the number of objects, importance information, and sound pressure RMS.
[0474] Specifically, the VBAP gain can be calculated by processing according to the selected result based on the number of objects, importance information, and sound pressure RMS, whether to perform quantization processing, how many gains the VBAP gain is to be quantized in the quantization processing (i.e., the quantization number in the quantization processing), and the total number of grids to be used for calculating the VBAP gain. In this case, for example, such processing as given below can be performed.
[0475] For example, if the number of objects is 10 or more, the total number of meshes is set to 10, and binarization is performed. In this case, since the number of objects is large, the processing volume is reduced by reducing the total number of meshes and performing binarization. Therefore, even if the hardware scale of the renderer is small, it is possible to render all objects.
[0476] Meanwhile, when the number of objects is less than 10 and the value of importance information is the highest value, only processes A1 to A3 are performed as usual. Therefore, for objects with a high degree of importance, sound can be reproduced without deteriorating the sound quality.
[0477] If the number of objects is less than 10, the importance information value is not the highest, and the RMS sound pressure is equal to or higher than -30 dB, the total number of grids is set to 10, and ternary processing is performed. This reduces the amount of processing required during rendering to the extent that, for sounds with high sound pressure, even though the importance level is low, the sound quality degradation of the sound is not noticeable.
[0478] Furthermore, if the number of objects is less than 10, the importance information value is not the highest, and the RMS sound pressure is less than -30 dB, the total number of grids is set to 5, and binarization is performed. This makes it possible to significantly reduce the amount of processing required for rendering sounds with low importance and low sound pressure.
[0479] In this way, when the number of objects is large, the amount of processing required during rendering is reduced, allowing rendering to be performed for all objects. However, when the number of objects is small to a certain extent, appropriate processing is selected and rendering is performed for each object. Therefore, while ensuring sound quality and reducing the amount of processing required for each object are well balanced, overall sound reproduction with sufficient sound quality is achieved with a small amount of processing.
[0480] <Example of Configuration of Audio Processing Device>
[0481] Now, an audio processing device that performs rendering processing while appropriately performing the above-mentioned quantization processing, mesh number switching processing, and the like is described. Figure 17 is a diagram depicting an example of a specific configuration of such an audio processing device as just described. Figure 17 In the figure, the same reference numerals are used to represent Figure 6 Those corresponding parts in the case of , so their descriptions are appropriately omitted.
[0482] Figure 17 The audio processing device 61 depicted in FIG includes an acquisition unit 21, a gain calculation unit 23, and a gain adjustment unit 71. The gain calculation unit 23 receives the metadata and audio signal of the object provided from the acquisition unit 21, calculates the VBAP gain of each of the speakers 12 for each object, and provides the calculated VBAP gain to the gain adjustment unit 71.
[0483] Furthermore, the gain calculation unit 23 includes a quantization unit 31 that performs quantization of the VBAP gain.
[0484] The gain adjustment unit 71 multiplies the audio signal supplied from the acquisition unit 21 by the VBAP gain of each speaker 12 supplied from the gain calculation unit 23 for each object to generate an audio signal for each speaker 12 , and supplies the audio signal to the speaker 12 .
[0485] <Description of Reproduction Processing>
[0486] Then, describe Figure 17 Specifically, referring to the operation of the audio processing device 61 described in Figure 18 The reproduction processing by the audio processing device 61 is described with reference to the flowchart of FIG.
[0487] It should be noted that in this example, it is assumed that the audio signal and metadata of one object or each of a plurality of objects are supplied to the acquisition unit 21 for each frame, and the reproduction process is performed for each frame of the audio signal of each object.
[0488] At step S231, the acquisition unit 21 acquires the audio signal and metadata of the object from the outside, and supplies the audio signal to the gain calculation unit 23 and the gain adjustment unit 71, while supplying the metadata to the gain calculation unit 23. In addition, the acquisition unit 21 also acquires the number information of the objects (i.e., information on the number of objects) for which sounds are to be reproduced simultaneously in the frame that is the processing target, and supplies this information to the gain calculation unit 23.
[0489] At step S232 , the gain calculation unit 23 decides whether the number of objects is equal to or greater than 10 based on the information indicating the number of objects supplied from the acquisition unit 21 .
[0490] If it is decided at step S232 that the number of objects is equal to or greater than 10, then at step S233 the gain calculation unit 23 sets the total number of meshes to be used when calculating the VBAP gain to 10. In other words, the gain calculation unit 23 selects 10 as the total number of meshes.
[0491] Furthermore, the gain calculation unit 23 selects a predetermined number of speakers 12 from all the speakers 12 in response to the selected total number of grids so that a number of grids equal to the total number are formed on the surface of the unit sphere. The gain calculation unit 23 then determines 10 grids on the surface of the unit sphere formed by the selected speakers 12 as grids to be used when calculating the VBAP gain.
[0492] At step S234, the gain calculation unit 23 calculates the VBAP gain of each speaker 12 through VBAP based on the position information representing the positions of the speakers 12 constituting the 10 grids determined at step S233 and the position information included in the metadata provided from the acquisition unit 21 and representing the position of the object.
[0493] Specifically, the gain calculation unit 23 continuously performs the calculation of Expression (8) using the mesh determined at step S233 in sequence as the mesh of the processing target to calculate the VBAP gain of the speaker 12. At this time, new meshes are continuously determined as the mesh of the processing target until the VBAP gains calculated for the three speakers 12 constituting the mesh of the processing target all indicate values equal to or greater than 0, so as to continuously calculate the VBAP gain.
[0494] At step S235 , the quantization unit 31 binarizes the VBAP gain of the speaker 12 obtained at step S234 , and the process proceeds to step S246 .
[0495] If it is determined at step S232 that the number of objects is less than 10, the process proceeds to step S236.
[0496] At step S236, the gain calculation unit 23 determines whether the value of the importance information of the object included in the metadata provided by the acquisition unit 21 is the highest value. For example, if the value of the importance information is "7" indicating the highest importance level, it is determined that the importance information indicates the highest value.
[0497] If it is decided at step S236 that the importance information indicates the highest value, the process proceeds to step S237.
[0498] At step S237, the gain calculation unit 23 calculates the VBAP gain of each speaker 12 based on the position information indicating the position of the speaker 12 and the position information included in the metadata supplied from the acquisition unit 21, and the process then proceeds to step S246. Here, a mesh formed from all the speakers 12 is continuously determined as a mesh of a processing target, and the VBAP gain is calculated by calculating Expression (8).
[0499] On the other hand, if it is determined at step S236 that the importance information does not indicate the highest value, then at step S238, the gain calculation unit 23 calculates the sound pressure RMS of the audio signal supplied from the acquisition unit 21. Specifically, the calculation of Expression (10) given above is performed for the frame of the audio signal serving as the processing target to calculate the sound pressure RMS.
[0500] At step S239 , the gain calculation unit 23 decides whether the sound pressure RMS calculated at step S238 is equal to or higher than −30 dB.
[0501] If it is determined at step S239 that the sound pressure RMS is equal to or higher than -30 dB, the processing at steps S240 and S241 is performed. Note that the processing at steps S240 and S241 is similar to the processing at steps S233 and S234, respectively, and thus their description is omitted.
[0502] At step S242 , the quantization unit 31 ternaryizes the VBAP gain of each speaker 12 obtained at step S241 , and the process proceeds to step S246 .
[0503] On the other hand, if it is determined in step S239 that the sound pressure RMS is lower than -30 dB, the process proceeds to step S243.
[0504] At step S243 , the gain calculation unit 23 sets the total number of meshes to be used when calculating the VBAP gain to five.
[0505] Furthermore, the gain calculation unit 23 selects a predetermined number of speakers 12 from all the speakers 12 in response to the selected total number of meshes "5", and determines five meshes on the unit sphere surface formed by the selected speakers 12 as meshes to be used when calculating the VBAP gain.
[0506] After determining the grid to be used when calculating the VBAP gain, the processing at steps S244 and S245 is performed, and the processing proceeds to step S246. It should be noted that the processing at steps S244 and S245 is similar to the processing at steps S234 and S235, so their description is omitted.
[0507] After the processing at step S235 , S237 , S242 , or S245 is performed and the VBAP gain of the speaker 12 is obtained, the processing at steps S246 to S248 is performed, thereby ending the reproduction processing.
[0508] It should be noted that since the processing at steps S246 to S248 is respectively the same as that at the above reference Figure 7 The processes at steps S17 to S19 described are similar, so their descriptions are omitted.
[0509] However, more specifically, the reproduction process is performed substantially simultaneously for each object, and at step S248, the audio signal of the speaker 12 obtained for each object is provided to the speaker 12. Specifically, the speaker 12 reproduces the sound based on the signal obtained by adding the audio signals of the object. Therefore, the sound of all objects is output simultaneously.
[0510] The audio processing device 61 selectively performs quantization processing and mesh number switching processing for each object as appropriate. This can reduce the processing amount of the rendering process while suppressing degradation of presentation or sound quality.
[0511] <Modification 1 to Second Embodiment>
[0512] <Configuration Example of Audio Processing Device>
[0513] Furthermore, in the description of the second embodiment, although an example is described in which quantization processing or grid number switching processing is selectively performed when processing for extending the sound image is not performed, further, quantization processing or grid number switching processing may be selectively performed when processing for extending the sound image is performed.
[0514] In such a case, the audio processing device 11 uses, for example, Figure 19 It should be noted that in Figure 19 In the figure, the same reference numerals are used to represent Figure 6 or Figure 17 Those corresponding parts in the case of , so their descriptions are appropriately omitted.
[0515] Figure 19 The audio processing device 11 depicted in includes an acquisition unit 21 , a vector calculation unit 22 , a gain calculation unit 23 , and a gain adjustment unit 71 .
[0516] The acquisition unit 21 acquires the audio signal and metadata of one or more objects, and provides the acquired audio signal to the gain calculation unit 23 and the gain adjustment unit 71, and provides the acquired metadata to the vector calculation unit 22 and the gain calculation unit 23. In addition, the gain calculation unit 23 includes a quantization unit 31.
[0517] <Description of Reproduction Processing>
[0518] Now, refer to Figure 20 The flowchart is described by Figure 19 The reproduction process performed by the audio processing device 11 depicted in FIG.
[0519] It should be noted that in this example, it is assumed that for one or more objects, the audio signal and metadata of the object are supplied to the acquisition unit 21 for each frame, and the reproduction process is performed for each frame of the audio signal of each object.
[0520] In addition, since the processing at steps S271 and S272 is respectively Figure 7The processing at steps S11 and S12 is similar, so their descriptions are omitted. However, at step S271, the audio signal acquired by the acquisition unit 21 is provided to the gain calculation unit 23 and the gain adjustment unit 71, and the metadata acquired by the acquisition unit 21 is provided to the vector calculation unit 22 and the gain calculation unit 23.
[0521] When the processing at steps S271 and S272 is executed, an extended vector or an extended vector and vector p are obtained.
[0522] At step S273, the gain calculation unit 23 performs VBAP gain calculation processing to calculate the VBAP gain of each speaker 12. It should be noted that although the details of the VBAP gain calculation processing are described below, in the VBAP gain calculation processing, quantization processing or grid number switching processing is selectively performed to calculate the VBAP gain of each speaker 12.
[0523] After the processing at step S273 is executed and the VBAP gain of the speaker 12 is obtained, the processing at steps S274 to S276 is executed and the reproduction processing ends. However, since those processes are respectively similar to Figure 7 the processing at steps S17 to S19, their descriptions are omitted. However, more specifically, the reproduction processing is performed for the objects substantially simultaneously, and at step S276, the audio signal of the speaker 12 obtained for each object is provided to the speaker 12. Therefore, the sounds of all objects are output from the speaker 12 simultaneously.
[0524] The audio processing device 11 appropriately performs quantization processing or grid number switching processing for each object selectively in the above-described manner. Thus, even when performing processing for extending the sound image, the processing amount of the rendering processing can be reduced while suppressing deterioration of the presentation or sound quality.
[0525] <Description of VBAP gain calculation processing>
[0526] Now, refer to Figure 21 the flowchart to describe the VBAP gain calculation processing corresponding to the processing at step S273 of Figure 20
[0527] It should be noted that since the processing at steps S301 to S303 is respectively similar to Figure 18 the processing at steps S232 to S234, their descriptions are omitted. However, at step S303, the VBAP gain of each speaker 12 is calculated for each vector of the extended vector or the extended vector and vector p.
[0528] At step S304, the gain calculation unit 23 adds the VBAP gain of each speaker 12 calculated for the vector to calculate the VBAP gain addition value. Figure 7 The processing is similar to the processing at step S14.
[0529] At step S305, the quantization unit 31 binarizes the VBAP gain addition value of each speaker 12 obtained by the process at step S304, and then the calculation process ends, and the process proceeds to step S306. Figure 20 Step S274.
[0530] On the other hand, if it is determined at step S301 that the number of objects is less than 10, the processing at steps S306 and S307 is performed.
[0531] It should be noted that since the processing at steps S306 and S307 is respectively Figure 18 The processing at step S236 and step S237 is similar, so their description is omitted. However, at step S307, the VBAP gain of each speaker 12 is calculated for each of the vector of the expansion vector or the expansion vector and the vector p.
[0532] Furthermore, after the processing at step S307 is performed, the processing at step S308 is performed, and the VBAP gain calculation processing ends, after which the processing proceeds to Figure 20 However, since the processing at step S308 is similar to that at step S304, the description thereof is omitted.
[0533] Furthermore, if it is determined at step S306 that the importance information does not indicate the highest value, the processing at steps S309 to S312 is performed. Figure 18 The processing at steps S238 to S241 of is similar, so their description is omitted. However, at step S312, the VBAP gain of each speaker 12 is calculated for each of the vector of the expansion vector or the expansion vector and the vector p.
[0534] After the VBAP gain of the speaker 12 is obtained for the vector, the process at step S313 is performed to calculate the VBAP gain addition value. However, since the process at step S313 is similar to the process at step S304, its description is omitted.
[0535] At step S314, the quantization unit 31 ternaryizes the VBAP gain added value of each speaker 12 obtained by the process at step S313, and the VBAP gain calculation ends, after which the process proceeds to step S314. Figure 20 Step S274.
[0536] Furthermore, if it is determined at step S310 that the sound pressure RMS is lower than -30 dB, the process at step S315 is performed, and the total number of grids to be used when calculating the VBAP gain is set to 5. It should be noted that the process at step S315 is the same as that at step S316. Figure 18 The processing at step S243 is similar, so its description is omitted.
[0537] After the grid to be used when calculating the VBAP gain is determined, the processing at steps S316 to S318 is performed, and the VBAP gain calculation processing ends, after which the processing proceeds to Figure 20 It should be noted that the processing at steps S316 to S318 is similar to the processing at steps S303 to S305, and thus their description is omitted.
[0538] The audio processing device 11 selectively performs quantization processing or grid number switching processing for each object as appropriate in the manner described above. This makes it possible to reduce the processing amount of rendering processing while suppressing degradation of presentation or sound quality when performing processing for extending the sound image.
[0539] Incidentally, although the above series of processes can be executed by hardware, they can also be executed by software in other ways. In the case where the series of processes are executed by software, the program constituting the software is installed in a computer. Here, the computer includes a dedicated computer incorporated into hardware, for example, a general-purpose personal computer that can execute various functions by installing various programs, etc.
[0540] Figure 22 : is a block diagram depicting an example of the configuration of hardware of a computer that executes the series of processes described above according to a program.
[0541] In the computer, a CPU (Central Processing Unit) 501 , a ROM (Read Only Memory) 502 , and a RAM (Random Access Memory) 503 are connected to one another via a bus 504 .
[0542] The input / output interface 505 is also connected to the bus 504 . An input unit 506 , an output unit 507 , a recording unit 508 , a communication unit 509 , and a drive 510 are connected to the input / output interface 505 .
[0543] The input unit 506 is composed of a keyboard, a mouse, a microphone, an image pickup element, etc. The output unit 507 is composed of a display unit, a speaker, etc. The recording unit 508 is composed of a hard disk, a nonvolatile memory, etc. The communication unit 509 is composed of a network interface, etc. The drive 510 drives a removable recording medium 511 (such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory).
[0544] In the computer configured in such a manner as described above, the CPU 501 loads a program recorded in, for example, the recording unit 508 into the RAM 503 through the input / output interface 505 and the bus 504 and executes the program to perform a series of processes described above.
[0545] The program executed by the computer (CPU 501) can be recorded on and provided as a removable recording medium 511 (e.g., such as a package medium). In addition, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.
[0546] In the computer, the program can be installed into the recording unit 508 through the input / output interface 505 by loading the removable recording medium 511 into the drive 510. Alternatively, the program can be received by the communication unit 509 through a wired or wireless transmission medium and installed into the recording unit 508. Alternatively, the program can be pre-installed into the ROM 502 or the recording unit 508.
[0547] It should be noted that the program executed by the computer may be a program that executes processing in time series according to the order described in this specification or a program that executes processing in parallel or at the timing of calling the program, or the like.
[0548] Furthermore, the embodiments of the present technology are not limited to the embodiments described above and can be changed in various ways without departing from the subject matter of the present technology.
[0549] For example, the present technology may adopt a configuration of cloud computing, by which one function is shared and collaboratively processed by multiple devices via a network.
[0550] Furthermore, the steps described with reference to the flowcharts described hereinabove may be executed by a single device, or may be executed in a shared manner by a plurality of devices.
[0551] Furthermore, in the case where one step includes a plurality of processes, the plurality of processes included in one step may be executed by a single device or may be executed in a shared manner by a plurality of devices.
[0552] The present technology may also take the following configurations.
[0553] (1) An audio processing device comprising:
[0554] an acquisition unit configured to acquire metadata including position information indicating a position of an audio object and sound image information composed of a vector of at least two dimensions or more and representing a spread of a sound image from the position;
[0555] a vector calculation unit configured to calculate an expansion vector indicating a position in the area based on a horizontal direction angle and a vertical direction angle of the area representing the spread of the sound image determined by the sound image information; and
[0556] A gain calculation unit is configured to calculate a gain of each of audio signals provided to two or more sound output units located near the position indicated by the position information based on the expansion vector.
[0557] (2) The audio processing device according to (1), wherein
[0558] The vector calculation unit calculates the extension vector based on a ratio between the horizontal direction angle and the vertical direction angle.
[0559] (3) The audio processing device according to (1) or (2), wherein:
[0560] The vector calculation unit calculates a predetermined number of extended vectors.
[0561] (4) The audio processing device according to (1) or (2), wherein:
[0562] The vector calculation unit calculates an arbitrarily variable number of extended vectors.
[0563] (5) The audio processing device according to (1), wherein
[0564] The sound image information is a vector indicating the center position of the area.
[0565] (6) The audio processing device according to (1), wherein
[0566] The sound image information is a two-dimensional or more dimensional vector indicating the degree of diffusion of the sound image from the center of the area.
[0567] (7) The audio processing device according to (1), wherein
[0568] The sound image information is a vector indicating a relative position of the center position of the area viewed from the position indicated by the position information.
[0569] (8) The audio processing device according to any one of (1) to (7), wherein
[0570] The gain calculation unit:
[0571] calculating a gain for each extension vector of each of the sound output units,
[0572] calculating an added value of the gains calculated for the expansion vectors of each of the sound output units,
[0573] quantizing the added value into a gain of two or more values with respect to each of the sound output units, and
[0574] A final gain with respect to each of the sound output units is calculated based on the quantized added value.
[0575] (9) The audio processing device according to (8), wherein
[0576] The gain calculation unit selects the number of grids to be used for calculating the gain, each of the grids being an area surrounded by three of the sound output units, and the gain calculation unit calculates the gain of each of the extended vectors based on a result of the selection of the number of grids and the extended vector.
[0577] (10) The audio processing device according to (9), wherein
[0578] The gain calculation unit selects the number of grids to be used for calculating the gain, whether to perform quantization, and the quantization number of the added value at the time of quantization, and
[0579] A final gain is calculated in response to the selection result.
[0580] (11) The audio processing device according to (10), wherein
[0581] The gain calculation unit selects the number of grids to be used for calculating the gain, whether to perform quantization, and the quantization amount based on the number of the audio objects.
[0582] (12) The audio processing device according to (10) or (11), wherein
[0583] The gain calculation unit selects the number of meshes to be used for calculating the gain, whether to perform quantization, and the quantization amount based on the degree of importance of the audio object.
[0584] (13) The audio processing device according to (12), wherein
[0585] The gain calculation unit selects the number of meshes to be used for calculating the gain such that the number of meshes to be used for calculating the gain increases as the position of the audio object is located closer to an audio object having a high degree of importance.
[0586] (14) The audio processing device according to any one of (10) to (13), wherein
[0587] The gain calculation unit selects the number of grids to be used for calculating the gain, whether to perform quantization, and the quantization amount based on the sound pressure of the audio signal of the audio object.
[0588] (15) The audio processing device according to any one of (9) to (14), wherein
[0589] The gain calculation unit selects three or more sound output units among a plurality of sound output units including sound output units located at different heights from each other in response to a selection result of the number of grids, and calculates gains based on one or more grids formed by the selected sound output units.
[0590] (16) An audio processing method comprising the following steps:
[0591] acquiring metadata including position information indicating a position of an audio object and sound image information composed of a vector of at least two dimensions or more and representing a spread of a sound image from the position;
[0592] calculating an expansion vector indicating a position in the area based on a horizontal direction angle and a vertical direction angle of the area representing the spread of the sound image determined by the sound image information; and
[0593] A gain of each of audio signals supplied to two or more sound output units located near the position indicated by the position information is calculated based on the expansion vector.
[0594] (17) A program causing a computer to execute the following processing, the processing comprising the following steps:
[0595] acquiring metadata including position information indicating a position of an audio object and sound image information composed of a vector of at least two dimensions or more and representing a spread of a sound image from the position;
[0596] calculating an expansion vector indicating a position in the area based on a horizontal direction angle and a vertical direction angle of the area representing the spread of the sound image determined by the sound image information; and
[0597] A gain of each of audio signals supplied to two or more sound output units located near the position indicated by the position information is calculated based on the expansion vector.
[0598] (18) An audio processing device comprising:
[0599] an acquiring unit configured to acquire metadata including position information indicating a position of an audio object; and
[0600] A gain calculation unit is configured to select the number of grids to be used for calculating the gain of the audio signal to be provided to the sound output unit, each of the grids being surrounded by three sound output units, and the gain calculation unit calculates the gain based on the selection result of the number of grids and the position information.
[0601] [Reference Mark List]
[0602] 11 audio processing device, 21 acquisition unit, 22 vector calculation unit, 23 gain calculation unit, 24 gain adjustment unit, 31 quantization unit, 61 audio processing device, 71 gain adjustment unit.
Claims
1. An audio processing device, comprising: an acquisition unit configured to acquire metadata including position information indicating a position of an audio object and sound image information composed of a three-dimensional vector and indicating a spread of the sound image from the position; The vector computing unit is configured as follows: Determine a three-dimensional vector representing the position of the audio object, wherein a starting point of the three-dimensional vector is an origin in a three-dimensional coordinate system, and the origin of the three-dimensional coordinate system is given by the position of the user; and calculating at least one expansion vector indicating a position in the area based on the three-dimensional vector and a ratio between a horizontal direction angle and a vertical direction angle of the area representing the diffusion of the sound image determined by the sound image information, wherein the number of the expansion vectors is predetermined; and a gain calculation unit configured to calculate a gain of each of the audio signals provided to the two or more sound output units located near the position indicated by the position information based on the at least one extension vector, Wherein, the gain calculation unit: calculating a gain of each expansion vector for each of the sound output units, calculating an added value of the gains calculated with respect to the expansion vector of each of the sound output units, normalize the added value, and A final gain of each of the sound output units is calculated based on the normalized added value.
2. An audio processing method, comprising the following steps: Acquire metadata including position information indicating a position of an audio object and sound image information composed of a three-dimensional vector and indicating a spread of the sound image from the position; Determine a three-dimensional vector representing the position of the audio object, wherein a starting point of the three-dimensional vector is an origin in a three-dimensional coordinate system, and the origin of the three-dimensional coordinate system is given by the position of the user; and calculating at least one expansion vector indicating a position in the area based on the three-dimensional vector and a ratio between a horizontal direction angle and a vertical direction angle of the area representing the diffusion of the sound image determined by the sound image information, wherein the number of the expansion vectors is predetermined; and calculating a gain of each of audio signals provided to two or more sound output units located near the position indicated by the position information based on the at least one expansion vector, Wherein, calculating the gain includes: calculating a gain of each expansion vector for each of the sound output units, calculating an added value of the gains calculated with respect to the expansion vector of each of the sound output units, normalize the added value, and A final gain of each of the sound output units is calculated based on the normalized added value.
3. A computer-readable storage medium having stored thereon a program for causing a computer to execute the following processing, the processing comprising the following steps: Acquire metadata including position information indicating a position of an audio object and sound image information composed of a three-dimensional vector and indicating a spread of the sound image from the position; Determine a three-dimensional vector representing the position of the audio object, wherein a starting point of the three-dimensional vector is an origin in a three-dimensional coordinate system, and the origin of the three-dimensional coordinate system is given by the position of the user; and calculating at least one expansion vector indicating a position in the area based on the three-dimensional vector and a ratio between a horizontal direction angle and a vertical direction angle of the area representing the diffusion of the sound image determined by the sound image information, wherein the number of the expansion vectors is predetermined; and calculating a gain of each of audio signals provided to two or more sound output units located near the position indicated by the position information based on the at least one expansion vector, Wherein, calculating the gain includes: calculating a gain of each expansion vector for each of the sound output units, calculating an added value of the gains calculated with respect to the expansion vector of each of the sound output units, normalize the added value, and A final gain of each of the sound output units is calculated based on the normalized added value.
