Audio processing apparatus, method, and program

Abstract

To make it possible to obtain sound of higher quality.SOLUTION: An acquisition unit acquires an audio signal and metadata of an object. A vector calculation unit calculates, on the basis of a horizontal direction angle and a vertical direction angle included in the metadata of the object and indicative of an extent of a sound image, a spread vector indicative of a position in a region indicative of the extent of the sound image. A gain calculation unit calculates, on the basis of the spread vector, a VBAP gain of the audio signal in regard to each speaker by VBAP. The present technology can be applied to an audio processing apparatus.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present technology relates to an audio processing device, method, and program, and more particularly to an audio processing device, method, and program that enable higher quality audio to be obtained. [Background technology]

[0002] 2. Description of the Related Art Conventionally, vector base amplitude panning (VBAP) is known as a technique for controlling the position of a sound image using a plurality of speakers (see, for example, Non-Patent Document 1).

[0003] In VBAP, by outputting sound from three speakers, the sound image can be localized at any point inside the triangle formed by the three speakers.

[0004] However, in the real world, sound images are not localized at a single point, but are localized in a subspace with a certain degree of spread. For example, the human voice is emitted from the vocal cords, but the vibrations propagate to the face and body, and as a result, the sound is thought to be emitted from the subspace of the entire human body.

[0005] Multiple Direction Amplitude Panning (MDAP) is a commonly known technology for localizing sound in such a partial space, that is, for expanding a sound image (see, for example, Non-Patent Document 2). MDAP is also used in the rendering processing unit of the MPEG (Moving Picture Experts Group)-H 3D Audio standard (see, for example, Non-Patent Document 3). [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Ville Pulkki, “Virtual Sound Source Positioning Using Vector Base Amplitude Panning”, Journal of AES, vol.45, no.6, pp.456-466, 1997 [Non-Patent Document 2] Ville Pulkki, "Uniform Spreading of Amplitude Panned Virtual Sources", Proc. 1999 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, New Paltz, New York, Oct. 17-20, 1999 [Non-Patent Document 3] ISO / IEC JTC1 / SC29 / WG11 N14747, August 2014, Sapporo, Japan, "Text of ISO / IEC 23008-3 / DIS, 3D Audio" Summary of the Invention [Problem to be solved by the invention]

[0007] However, the above-mentioned techniques have not been able to provide sufficiently high quality audio.

[0008] For example, in the MPEG-H 3D Audio standard, metadata for audio objects includes information called "spread" that indicates the degree of spread of a sound image, and a process to spread the sound image is performed based on this spread. However, the process to spread the sound image is restricted to having a sound image that is symmetrical up, down, left, and right with the position of the audio object at the center. This makes it impossible to perform processing that takes into account the directivity (radiation direction) of the sound from the audio object, and it is not possible to obtain sound of sufficiently high quality.

[0009] The present technology has been made in view of such circumstances, and makes it possible to obtain audio of higher quality. [Means for solving the problem]

[0010] An audio processing device according to one aspect of the present technology includes an acquisition unit that acquires metadata including position information indicating a position of an audio object and sound image information consisting of at least a two-dimensional vector indicating the spread of a sound image from the position, a vector calculation unit that calculates a spread vector indicating a position within an area representing the spread of the sound image determined by the sound image information, based on a horizontal angle and a vertical angle with respect to the area, and a gain calculation unit that calculates, based on the spread vector, respective gains of audio signals that are supplied to two or more audio output units located in the vicinity of the position indicated by the position information.

[0011] An audio processing method or program according to one aspect of the present technology includes the steps of acquiring metadata including position information indicating a position of an audio object and sound image information consisting of a vector of at least two dimensions and indicating the spread of a sound image from the position, calculating a spread vector indicating a position within the region based on a horizontal angle and a vertical angle with respect to a region indicating the spread of the sound image determined by the sound image information, and calculating, based on the spread vector, respective gains of audio signals supplied to two or more audio output units located in the vicinity of the position indicated by the position information.

[0012] In one aspect of the present technology, metadata is acquired that includes position information indicating the position of an audio object and sound image information consisting of at least a two-dimensional vector representing the spread of a sound image from the position, a spread vector indicating a position within the region representing the spread of the sound image determined by the sound image information is calculated based on the horizontal angle and vertical angle of the region, and gains of each of the audio signals supplied to two or more audio output units located in the vicinity of the position indicated by the position information are calculated based on the spread vector. Effect of the Invention

[0013] According to one aspect of the present technology, audio of higher quality can be obtained.

[0014] Note that the effects described herein are not necessarily limited to those described herein, and may be any of the effects described in this disclosure. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram illustrating VBAP. [Diagram 2] FIG. 2 is a diagram illustrating the position of a sound image. [Diagram 3] FIG. 13 is a diagram illustrating a spread vector. [Figure 4] FIG. 13 is a diagram illustrating a spread central vector method. [Diagram 5] FIG. 1 is a diagram illustrating a spread radiation vector method. [Figure 6] FIG. 1 illustrates an example of the configuration of a voice processing device. [Figure 7] 11 is a flowchart illustrating a playback process. [Figure 8] 13 is a flowchart illustrating a spread vector calculation process. [Figure 9] 13 is a flowchart illustrating a spread vector calculation process based on a spread three-dimensional vector. [Figure 10] 13 is a flowchart illustrating a spread vector calculation process based on a spread center vector. [Figure 11] 13 is a flowchart illustrating a spread vector calculation process based on a spread end vector. [Figure 12] 13 is a flowchart illustrating a spread vector calculation process based on a spread radial vector. [Figure 13] 13 is a flowchart illustrating a spread vector calculation process based on spread vector position information. [Figure 14] FIG. 13 is a diagram illustrating switching of the number of meshes. [Figure 15] FIG. 13 is a diagram illustrating switching of the number of meshes. [Figure 16] FIG. 13 is a diagram illustrating the formation of a mesh. [Figure 17] FIG. 1 illustrates an example of the configuration of a voice processing device. [Figure 18] 11 is a flowchart illustrating a playback process. [Figure 19] FIG. 1 illustrates an example of the configuration of a voice processing device. [Figure 20] 11 is a flowchart illustrating a playback process. [Figure 21] 13 is a flowchart illustrating a VBAP gain calculation process. [Figure 22] FIG. 1 illustrates an example of the configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, embodiments to which the present technology is applied will be described with reference to the drawings.

[0017] First Embodiment <About VBAP and sound image widening processing> The present technology makes it possible to obtain higher quality audio when rendering by acquiring an audio signal of an audio object and metadata such as position information of the audio object. Note that hereinafter, an audio object will also be simply referred to as an object.

[0018] In the following, we will first explain the process of expanding the sound image in VBAP and the MPEG-H 3D Audio standard.

[0019] For example, as shown in FIG. 1, assume that a user U11 viewing content such as audio-accompanying moving images and music is listening to three-channel audio output from three speakers SP1 to SP3 as the audio of the content.

[0020] In such a case, it is considered that a sound image is localized at a position p using information indicating the positions of the three speakers SP1 to SP3 that output the sounds of the respective channels.

[0021] For example, in a three-dimensional coordinate system with the position of the head of the user U11 as the origin O, the position p is represented by a three-dimensional vector (hereinafter also referred to as vector p) starting from the origin O. Furthermore, if the three-dimensional vectors starting from the origin O and pointing in the direction of the positions of the speakers SP1 to SP3 are vectors l1 to l3, the vector p can be represented by a linear sum of vectors l1 to l3.

[0022] That is, p = g1l1 + g2l2 + g3l3.

[0023] Here, by calculating the coefficients g1 to g3 by which vectors l1 to l3 are multiplied, and setting these coefficients g1 to g3 as the gains of the sounds output from speakers SP1 to SP3, respectively, the sound image can be localized at position p.

[0024] The method of calculating coefficients g1 to g3 using the position information of the three speakers SP1 to SP3 in this way and controlling the localization position of the sound image is called three-dimensional VBAP. In particular, the gains calculated for each speaker, such as coefficients g1 to g3, will be referred to as VBAP gains hereinafter.

[0025] 1, the sound image can be localized at any position within a triangular area TR11 on a spherical surface that includes the positions of the speakers SP1, SP2, and SP3. Here, the area TR11 is an area on the surface of a sphere that is centered on the origin O and passes through the positions of the speakers SP1 to SP3, and is a triangular area surrounded by the speakers SP1 to SP3.

[0026] By using this type of 3D VBAP, it becomes possible to localize a sound image at any position in space. For more information on VBAP, see, for example, "Ville Pulkki, "Virtual Sound Source Positioning Using Vector Base Amplitude Panning", Journal of AES, vol.45, no.6, pp.456-466, 1997".

[0027] Next, the process of expanding the sound image in the MPEG-H 3D Audio standard will be described.

[0028] In the MPEG-H 3D Audio standard, an encoding device outputs a bitstream obtained by multiplexing encoded audio data obtained by encoding the audio signal of each object and encoded metadata obtained by encoding the metadata of each object.

[0029] For example, the metadata includes position information indicating the spatial position of an object, importance information indicating the importance of the object, and spread, which is information indicating the degree of spread of the sound image of the object.

[0030] Here, spread, which indicates the degree of spread of the sound image, can be any angle between 0° and 180°, and the encoding device can specify a different value of spread for each object for each frame of the audio signal.

[0031] The position of an object is represented by a horizontal angle azimuth, a vertical angle elevation, and a distance radius. That is, the position information of an object consists of the values ​​of the horizontal angle azimuth, the vertical angle elevation, and the distance radius.

[0032] 2, for example, consider a three-dimensional coordinate system in which the position of a viewer listening to the sounds of each object output from a speaker (not shown) is taken as the origin O, and the upper right, upper left, and upward directions in the figure are taken as the mutually perpendicular x-axis, y-axis, and z-axis directions. In this case, if the position of one object is taken as position OBJ11, then the sound image should be localized at position OBJ11 in the three-dimensional coordinate system.

[0033] Furthermore, if the line connecting position OBJ11 and the origin O is called line L, then the horizontal angle θ (azimuth) in the diagram formed by line L and the x-axis on the xy plane becomes the horizontal angle azimuth that indicates the horizontal position of the object at position OBJ11, and the horizontal angle azimuth is set to any value that satisfies -180°≦azimuth≦180°.

[0034] For example, the positive direction along the x-axis is azimuth = 0°, and the negative direction along the x-axis is azimuth = +180° = -180°. Also, the counterclockwise direction around the origin O is the + azimuth direction, and the clockwise direction around the origin O is the - azimuth direction.

[0035] Furthermore, the angle between the straight line L and the xy plane, that is, the vertical angle γ (elevation angle) in the figure, becomes the vertical angle elevation that indicates the vertical position of the object at position OBJ11, and the vertical angle elevation is set to any value that satisfies -90°≦elevation≦90°. For example, the position of the xy plane is set to elevation=0°, and the upward direction in the figure is set to the + direction of the vertical angle elevation, and the downward direction in the figure is set to the - direction of the vertical angle elevation.

[0036] Moreover, the length of the straight line L, that is, the distance from the origin O to the position OBJ11, is set to the distance radius to the viewer, and the distance radius is set to a value equal to or greater than 0. In other words, the distance radius is set to a value that satisfies 0≦radius<∞. Hereinafter, the distance radius is also referred to as the distance in the radial direction.

[0037] In VBAP, the distance (radius) from all speakers and objects to the viewer is the same, and it is common to normalize the distance (radius) to 1 for calculations.

[0038] Thus, the position information of an object included in the metadata consists of the values ​​of the horizontal angle azimuth, the vertical angle elevation, and the distance radius.

[0039] Hereinafter, the horizontal angle azimuth, the vertical angle elevation, and the distance radius will be simply referred to as azimuth, elevation, and radius.

[0040] In addition, in a decoding device that receives a bitstream containing encoded audio data and encoded metadata, after the encoded audio data and encoded metadata are decoded, a rendering process is performed to widen the sound image according to the spread value contained in the metadata.

[0041] Specifically, the decoding device first determines that the spatial position indicated by the position information included in the metadata of the object is position p. This position p corresponds to the position p in FIG.

[0042] Next, the decoding device arranges 18 spread vectors p1 to p18 so as to be vertically and horizontally symmetrical on the unit sphere with the central position p0 as the center, with position p set to central position p0 as shown in Fig. 3. Note that in Fig. 3, the same reference numerals are used to designate parts corresponding to those in Fig. 1, and the description thereof will be omitted as appropriate.

[0043] 3, five speakers SP1 to SP5 are arranged on the spherical surface of a unit sphere of radius 1 centered at the origin O, and the position p indicated by the position information is set as the central position p0. Hereinafter, the position p will also be referred to as the object position p in particular, and a vector that starts at the origin O and ends at the object position p will also be referred to as the vector p. In addition, a vector that starts at the origin O and ends at the central position p0 will also be referred to as the vector p0.

[0044] In Figure 3, the dotted arrows starting from the origin O represent the spread vectors. However, although there are actually 18 spread vectors, only eight are drawn in Figure 3 to make the figure easier to see.

[0045] Here, the end positions of the spread vectors p1 to p18 are vectors located within an area R11 of a circle on the unit sphere centered at the central position p0. In particular, the angle between the spread vectors whose end positions are on the circumference of the circle represented by the area R11 and the vector p0 is the angle indicated by spread.

[0046] Therefore, the end positions of the spread vectors are located at positions farther away from the center position p0 as the spread value increases. In other words, the area R11 becomes larger.

[0047] This region R11 expresses the spread of the sound image from the position of the object. In other words, the region R11 is a region that indicates the range in which the sound image of the object spreads. Moreover, since the sound of the object is considered to be emitted from the entire object, it can also be said that the region R11 represents the shape of the object. Hereinafter, a region that indicates the range in which the sound image of the object spreads, such as the region R11, will also be referred to as a region that indicates the spread of the sound image.

[0048] Furthermore, when the spread value is 0, the end positions of each of the 18 spread vectors p1 to p18 are equal to the center position p0.

[0049] In the following, the end points of the spread vectors p1 to p18 will be referred to as positions p1 to p18, respectively.

[0050] In this way, when the spread vectors that are vertically and horizontally symmetrical on the unit sphere are determined, the decoding device calculates the VBAP gain for each speaker of each channel by VBAP for vector p and each spread vector, that is, for position p and each of positions p1 to p18. At this time, the VBAP gain for each speaker is calculated so that the sound image is localized at each position, such as position p and position p1.

[0051] Then, the decoding device adds up the VBAP gains calculated for each position for each speaker. For example, in the example of Fig. 3, the VBAP gains calculated for the speaker SP1 at the position p and at the positions p1 to p18 are added up.

[0052] Furthermore, the decoding device normalizes the VBAP gains after the addition process calculated for each speaker. That is, normalization is performed so that the sum of the squares of the VBAP gains of all speakers becomes one.

[0053] The decoding device then multiplies the audio signal of the object by the VBAP gain for each speaker obtained by normalization to obtain an audio signal for each speaker, and supplies the audio signal obtained for each speaker to the speaker to output audio.

[0054] 3, the sound image is localized as if the sound were being output from the entire region R11. In other words, the sound image spreads over the entire region R11.

[0055] 3, when the sound image widening process is not performed, the sound image of the object is localized at position p, and in this case, sound is substantially output from speakers SP2 and SP3. In contrast, when the sound image widening process is performed, the sound image spreads over the entire region R11, and during sound playback, sound is output from speakers SP1 to SP4.

[0056] However, when the above-mentioned sound image widening process is performed, the amount of processing required for rendering increases compared to when the sound image widening process is not performed. This can result in a decrease in the number of objects that can be handled by the decoding device, or in a decoding device equipped with a renderer with a small hardware scale that cannot perform rendering.

[0057] Therefore, when performing processing to widen the sound image during rendering, it is desirable to be able to perform the rendering with as small an amount of processing as possible.

[0058] In addition, the 18 spread vectors mentioned above are constrained to be vertically and horizontally symmetric on the unit sphere with the central position p0 = position p as the center, so processing cannot take into account the sound directivity (radiation direction) of the object or the shape of the object. As a result, it was not possible to obtain sound of sufficiently high quality.

[0059] Furthermore, the MPEG-H 3D Audio standard only specifies one way of expanding the sound image during rendering, so if the renderer hardware scale is small, it is not possible to expand the sound image, meaning that audio cannot be played back.

[0060] Furthermore, the MPEG-H 3D Audio standard did not allow for switching between different processes to achieve the highest quality audio possible within the processing load permitted by the hardware scale of the renderer.

[0061] In consideration of the above circumstances, this technology makes it possible to reduce the amount of processing required during rendering. This technology also makes it possible to obtain sufficiently high-quality audio by expressing the directionality and shape of an object. Furthermore, this technology selects an appropriate process for rendering depending on the hardware scale of the renderer, making it possible to obtain the highest quality audio within the range of the allowable amount of processing.

[0062] An overview of this technology will be described below.

[0063] <Reducing processing volume> First, the reduction in the amount of processing during rendering will be described.

[0064] In normal VBAP processing (rendering processing) that does not widen the sound image, the following specific processing A1 to A3 are performed.

[0065] (Process A1) Calculate the VBAP gain to be multiplied to the audio signal for three speakers (Process A2) Normalize the VBAP gains of the three speakers so that the sum of the squares is 1. (Process A3) Multiply the object's audio signal by the VBAP gain

[0066] Here, in process A3, the audio signal is multiplied by the VBAP gain for each of the three speakers, and therefore such multiplication is performed a maximum of three times.

[0067] On the other hand, in the VBAP process (rendering process) for widening the sound image, the following specific processes B1 to B5 are performed.

[0068] (Process B1) For a vector p, calculate the VBAP gain to multiply the audio signal for each of the three speakers. (Process B2) For each of the 18 spread vectors, calculate the VBAP gain to multiply the audio signal for each of the three speakers. (Process B3) For each speaker, add the VBAP gain calculated for each vector. (Process B4) Normalize the VBAP gain of all speakers so that the sum of the squares is 1. (Process B5) Multiply the object's audio signal by the VBAP gain

[0069] When processing to widen the sound image is performed, the number of speakers that output sound will be three or more, so that multiplication processing will be performed three or more times in process B5.

[0070] Therefore, when comparing the case where sound image widening processing is performed with the case where it is not performed, when sound image widening processing is performed, the processing amount increases particularly by the amount of processing B2 and processing B3, and even processing B5 requires more processing than processing A3.

[0071] Therefore, in the present technology, the sum of the VBAP gains of the vectors calculated for each speaker is quantized, thereby making it possible to reduce the amount of processing in the above-mentioned process B5.

[0072] Specifically, in this technology, the following process is performed. Note that, hereinafter, the sum (addition value) of the VBAP gains calculated for each vector, such as the vector p and the spread vector, calculated for each speaker will also be referred to as a VBAP gain addition value.

[0073] First, processes B1 to B3 are performed to obtain a VBAP gain addition value for each speaker. The VBAP gain addition value is then binarized. In the binarization, for example, the VBAP gain addition value for each speaker is set to either 0 or 1.

[0074] The VBAP gain addition value may be binarized by any method, such as rounding off, ceiling (rounding up), flooring (rounding down), or threshold processing.

[0075] Once the VBAP gain addition value has been binarized in this manner, the above-mentioned process B4 is then performed based on the binarized VBAP gain addition value. As a result, the final VBAP gain of each speaker will have only one value excluding 0. In other words, once the VBAP gain addition value has been binarized, the final VBAP gain value of each speaker will be either 0 or a predetermined value.

[0076] For example, if the result of binarization is that the VBAP gain addition value of three speakers becomes 1 and the VBAP gain addition value of the other speakers becomes 0, the final VBAP gain value of those three speakers will be 1 / 3. (1 / 2) It becomes.

[0077] Once the final VBAP gain of each speaker is obtained in this manner, thereafter, instead of the above-mentioned process B5, a process B5' is performed in which the audio signal of each speaker is multiplied by the final VBAP gain.

[0078] When binarization is performed as described above, the final VBAP gain value for each speaker becomes either 0 or a predetermined value, so that only one multiplication process needs to be performed in process B5', thereby reducing the amount of processing. In other words, while three or more multiplication processes must be performed in process B5, only one multiplication process needs to be performed in process B5'.

[0079] Although the case where the VBAP gain addition value is binarized has been described as an example here, the VBAP gain addition value may be quantized to three or more values.

[0080] For example, when the VBAP gain addition value is set to one of three values, the above-mentioned processes B1 to B3 are performed, and when the VBAP gain addition value is obtained for each speaker, the VBAP gain addition value is quantized to one of 0, 0.5, or 1. Then, processes B4 and B5' are performed. In this case, the number of multiplication processes in process B5' is a maximum of two.

[0081] In this way, when the VBAP gain addition value is converted into an x ​​value, that is, quantized to become one of x gains equal to or greater than 2, the number of multiplication processes in process B5' becomes a maximum of (x-1) times.

[0082] In the above, an example has been described in which the amount of processing is reduced by quantizing the VBAP gain addition value when processing to widen a sound image is performed, but even when processing to widen a sound image is not performed, the amount of processing can be reduced by quantizing the VBAP gain in a similar manner. In other words, if the VBAP gains of each speaker obtained for vector p are quantized, the number of times that the audio signal is multiplied by the normalized VBAP gain can be reduced.

[0083] Processing to represent object shapes and sound directionality Next, a process of expressing the shape of an object and the directivity of the sound of the object according to the present technology will be described.

[0084] The following describes five methods: the spread three-dimensional vector method, the spread central vector method, the spread edge vector method, the spread radial vector method, and the arbitrary spread vector method.

[0085] (spread 3-dimensional vector method) First, the spread three-dimensional vector method will be described.

[0086] In the spread 3D vector method, a spread 3D vector, which is a 3D vector, is stored in the bitstream and transmitted. Here, for example, the spread 3D vector is stored in the metadata of the frame of each audio signal for each object. In this case, spread, which indicates the degree of spread of the sound image, is not stored in the metadata.

[0087] For example, the spread three-dimensional vector is a three-dimensional vector consisting of three elements: s3_azimuth, which indicates the horizontal spread of the sound image; s3_elevation, which indicates the vertical spread of the sound image; and s3_radius, which indicates the radial depth of the sound image.

[0088] That is, spread 3D vector = (s3_azimuth, s3_elevation, s3_radius).

[0089] Here, s3_azimuth indicates the spread angle of the sound image in the horizontal direction from position p, that is, in the direction of the horizontal angle azimuth described above. Specifically, s3_azimuth indicates the angle between the vector from the origin O toward the horizontal end of the area indicating the spread of the sound image and vector p (vector p0).

[0090] Similarly, s3_elevation indicates the sound image spread angle in the vertical direction from position p, that is, in the direction of the vertical angle elevation described above. Specifically, s3_elevation indicates the angle between vector p (vector p0) and a vector pointing from origin O to the vertical end of the area indicating the spread of the sound image. Also, s3_radius indicates the direction of the distance radius described above, that is, the depth in the normal direction of the unit sphere.

[0091] Note that these s3_azimuth, s3_elevation, and s3_radius are set to values ​​equal to or greater than 0. Also, here, the spread three-dimensional vector is set as information indicating a relative position with respect to the position p indicated by the position information of the object, but the spread three-dimensional vector may be set as information indicating an absolute position.

[0092] In the spread 3D vector method, rendering is performed using such a spread 3D vector.

[0093] Specifically, in the spread three-dimensional vector method, the value of spread is calculated by calculating the following formula (1) based on the spread three-dimensional vector.

[0094]

number

[0095] In formula (1), max(a,b) represents a function that returns the larger value of a and b. Therefore, the larger value of s3_azimuth and s3_elevation is set as the value of spread.

[0096] Then, based on the spread value obtained in this manner and the position information included in the metadata, 18 spread vectors p1 to p18 are calculated in the same manner as in the MPEG-H 3D Audio standard.

[0097] Therefore, the position p of the object indicated by the position information included in the metadata is set to the central position p0, and 18 spread vectors p1 to p18 are calculated so that they are vertically and horizontally symmetrical on the unit sphere with the central position p0 as the center.

[0098] In the spread three-dimensional vector method, a vector p0 having the origin O as a start point and the center position p0 as an end point is set as a spread vector p0.

[0099] Each spread vector is expressed by a horizontal angle azimuth, a vertical angle elevation, and a distance radius. In the following, the horizontal angle azimuth and vertical angle elevation of a spread vector pi (where i=0 to 18) are represented as a(i) and e(i).

[0100] Once the spread vectors p0 to p18 have been obtained in this manner, the spread vectors p1 to p18 are then changed (corrected) based on the ratio of s3_azimuth to s3_elevation to form the final spread vector.

[0101] That is, when s3_azimuth is greater than s3_elevation, the calculation of the following equation (2) is performed, and e(i), which is the elevation of each of spread vectors p1 to p18, is changed to e'(i).

[0102]

number

[0103] It should be noted that no elevation correction is performed for the spread vector p0.

[0104] On the other hand, if s3_azimuth is less than s3_elevation, the calculation of the following equation (3) is performed, and a(i), which is the azimuth of each of spread vectors p1 to p18, is changed to a'(i).

[0105]

number

[0106] It should be noted that no azimuth correction is performed for the spread vector p0.

[0107] As described above, the larger of s3_azimuth and s3_elevation is taken as spread, and the process of calculating the spread vector is a process in which the area indicating the spread of the sound image on the unit sphere is temporarily taken as a circle with a radius determined by the larger angle of s3_azimuth and s3_elevation, and the spread vector is calculated in the same manner as conventional processes.

[0108] Then, the process of correcting the spread vector using equation (2) or (3) depending on the magnitude relationship between s3_azimuth and s3_elevation is a process of correcting the area indicating the spread of the sound image, i.e., the spread vector, so that the area indicating the spread of the sound image on the unit sphere becomes the area determined by the original s3_azimuth and s3_elevation specified by the spread three-dimensional vector.

[0109] Therefore, these processes ultimately amount to calculation of a spread vector for a region indicating the spread of a circular or elliptical sound image on a unit sphere based on a spread three-dimensional vector, that is, s3_azimuth and s3_elevation.

[0110] Once the spread vectors are obtained in this manner, the spread vectors p0 to p18 are then used to perform the above-mentioned processes B2, B3, B4, and B5' to generate audio signals to be supplied to each speaker.

[0111] In process B2, the VBAP gain for each speaker is calculated for each of the 19 spread vectors p0 to p18. Here, since the spread vector p0 is the vector p, the process of calculating the VBAP gain for the spread vector p0 can be said to be the process B1. After process B3, the VBAP gain addition value is quantized as necessary.

[0112] In this way, by using the spread 3D vector to define the area indicating the spread of the sound image as an area of ​​any shape, it becomes possible to express the shape of an object and the directionality of the object's sound, and higher quality audio can be obtained by rendering.

[0113] Also, although an example has been described here in which the larger of s3_azimuth and s3_elevation is set as the value of the spread, the smaller of s3_azimuth and s3_elevation may be set as the value of the spread.

[0114] In this case, when s3_azimuth is greater than s3_elevation, the azimuth a(i) of each spread vector is corrected, and when s3_azimuth is less than s3_elevation, the elevation e(i) of each spread vector is corrected.

[0115] Furthermore, although an example has been described here in which spread vector p0 to spread vector p18, i.e., 19 predetermined spread vectors, are obtained and the VBAP gain is calculated for those spread vectors, the number of spread vectors calculated may be made variable.

[0116] In such a case, the number of spread vectors to be generated can be determined according to the ratio of s3_azimuth and s3_elevation, for example. According to such processing, for example, when an object is long horizontally and the sound of the object does not spread much in the vertical direction, the spread vectors arranged vertically can be omitted and each spread vector is arranged approximately horizontally, so that the sound spread in the horizontal direction can be appropriately expressed.

[0117] (Spread center vector method) Next, the spread center vector method will be explained.

[0118] In the spread center vector method, a spread center vector, which is a three-dimensional vector, is stored in the bitstream and transmitted. Here, for example, the spread center vector is stored in the metadata of the frame of each audio signal for each object. In this case, the spread, which indicates the degree of spread of the sound image, is also stored in the metadata.

[0119] The spread center vector is a vector that indicates the central position p0 of the area that indicates the spread of the sound image of the object. For example, the spread center vector is a three-dimensional vector consisting of three elements: azimuth, which indicates the horizontal angle of the central position p0; elevation, which indicates the vertical angle of the central position p0; and radius, which indicates the radial distance from the central position p0.

[0120] That is, the spread center vector = (azimuth, elevation, radius).

[0121] During rendering processing, the position indicated by this spread center vector is set as the center position p0, and spread vectors p0 to p18 are calculated as spread vectors. Here, the spread vector p0 is a vector p0 that starts at the origin O and ends at the center position p0, as shown in Fig. 4, for example. In Fig. 4, the same reference numerals are used to designate parts corresponding to those in Fig. 3, and their explanation will be omitted as appropriate.

[0122] In addition, in FIG. 4, the dotted arrows represent spread vectors, and in FIG. 4, only nine spread vectors are drawn to make the figure easier to see.

[0123] In the example shown in Fig. 3, position p=center position p0, but in the example shown in Fig. 4, center position p0 is a position different from position p. In this example, it can be seen that region R21 indicating the spread of the sound image centered on center position p0 is shifted to the left in the figure with respect to position p, which is the position of the object, more so than in the example in Fig. 3.

[0124] In this way, if an arbitrary position can be specified by the spread center vector as the center position p0 of the area indicating the spread of the sound image, the sound directivity of the object can be expressed more accurately.

[0125] In the spread center vector method, once the spread vectors p0 to p18 are obtained, process B1 is then performed on the vector p, and process B2 is performed on the spread vectors p0 to p18.

[0126] In the process B2, the VBAP gain may be calculated for each of the 19 spread vectors, or the VBAP gain may be calculated only for the spread vectors p1 to p18 excluding the spread vector p0. In the following, the explanation will be continued assuming that the VBAP gain is also calculated for the spread vector p0.

[0127] After the VBAP gain of each vector is calculated, processing B3, processing B4, and processing B5' are performed to generate an audio signal to be supplied to each speaker. After processing B3, the VBAP gain addition value is quantized as necessary.

[0128] Even with the spread center vector method described above, it is possible to obtain sufficiently high quality audio by rendering.

[0129] (Spread end vector method) Next, the spread end vector method will be described.

[0130] In the spread edge vector method, a spread edge vector, which is a five-dimensional vector, is stored in the bitstream and transmitted. Here, for example, the spread edge vector is stored in the metadata of the frame of each audio signal for each object. In this case, the spread, which indicates the degree of spread of the sound image, is not stored in the metadata.

[0131] For example, the spread end vector is a vector that represents an area showing the spread of the sound image of an object, and is a vector consisting of five elements: spread left end azimuth, spread right end azimuth, spread top end elevation, spread bottom end elevation, and spread radius.

[0132] Here, the spread left end azimuth and spread right end azimuth constituting the spread end vector indicate the value of the horizontal angle azimuth indicating the absolute positions of the left and right ends in the horizontal direction in the area indicating the spread of the sound image, respectively. In other words, the spread left end azimuth and spread right end azimuth indicate the angle indicating the degree of spread of the sound image to the left and right from the center position p0 of the area indicating the spread of the sound image, respectively.

[0133] Moreover, the spread upper end elevation and the spread lower end elevation indicate the vertical angle elevation value that indicates the absolute positions of the upper and lower ends in the vertical direction in the area that indicates the spread of the sound image. In other words, the spread upper end elevation and the spread lower end elevation indicate the angles that indicate the degree of spread of the sound image in the upward and downward directions from the center position p0 of the area that indicates the spread of the sound image. Furthermore, the spread radius indicates the radial depth of the sound image.

[0134] Note that, here, the spread end vector is considered to be information indicating an absolute position in space, but the spread end vector may also be information indicating a relative position with respect to the position p indicated by the object's position information.

[0135] In the spread end vector method, rendering is performed using such spread end vectors.

[0136] Specifically, in the spread end vector method, the center position p0 is calculated by calculating the following equation (4) based on the spread end vector.

[0137]

number

[0138] That is, the horizontal angle azimuth indicating the center position p0 is set to the intermediate (average) angle between the spread left end azimuth and the spread right end azimuth, and the vertical angle elevation indicating the center position p0 is set to the intermediate (average) angle between the spread top end elevation and the spread bottom end elevation. Also, the distance radius indicating the center position p0 is set to the spread radius.

[0139] Therefore, in the spread end vector method, the center position p0 may be a position different from the position p of the object indicated by the position information.

[0140] In addition, in the spread end vector method, the spread value is calculated by calculating the following equation (5).

[0141]

number

[0142] In formula (5), max(a,b) indicates a function that returns the larger value of a and b. Therefore, the value of spread is determined to be the larger of (spread left end azimuth-spread right end azimuth) / 2, which is the angle corresponding to the horizontal radius in the area indicating the spread of the sound image of the object indicated by the spread end vector, and (spread top end elevation-spread bottom end elevation) / 2, which is the angle corresponding to the vertical radius.

[0143] Then, based on the spread value thus obtained and the central position p0 (vector p0), 18 spread vectors p1 to p18 are calculated in the same way as in the MPEG-H 3D Audio standard.

[0144] Therefore, 18 spread vectors p1 to p18 are obtained so as to be vertically and horizontally symmetrical on the unit sphere with the central position p0 as the center.

[0145] In the spread end vector method, a vector p0 having the origin O as a start point and the center position p0 as an end point is set as the spread vector p0.

[0146] In the spread end vector method, as in the spread 3D vector method, each spread vector is expressed by a horizontal angle azimuth, a vertical angle elevation, and a distance radius. That is, the horizontal angle azimuth and vertical angle elevation of a spread vector pi (where i=0 to 18) are a(i) and e(i), respectively.

[0147] Once spread vectors p0 to p18 are obtained in this manner, spread vectors p1 to p18 are then modified (corrected) based on the ratio of (spread left end azimuth-spread right end azimuth) to (spread top end elevation-spread bottom end elevation) to obtain the final spread vector.

[0148] In other words, if (spread left end azimuth - spread right end azimuth) is greater than (spread top end elevation - spread bottom end elevation), the calculation of the following equation (6) is performed, and e(i), which is the elevation of each of spread vectors p1 to p18, is changed to e'(i).

[0149]

number

[0150] It should be noted that no elevation correction is performed for the spread vector p0.

[0151] On the other hand, if (spread left end azimuth - spread right end azimuth) is less than (spread top end elevation - spread bottom end elevation), the calculation of the following equation (7) is performed, and the azimuth a(i) of each of spread vectors p1 to p18 is changed to a'(i).

[0152]

number

[0153] It should be noted that no azimuth correction is performed for the spread vector p0.

[0154] The above-described spread vector calculation method is basically the same as that in the spread three-dimensional vector method.

[0155] Therefore, these processes ultimately amount to calculation of a spread vector for a region indicating the spread of a circular or elliptical sound image on a unit sphere determined by the spread end vector, based on the spread end vector.

[0156] Once the spread vector is obtained in this manner, the vector p and spread vectors p0 to p18 are then used to perform the above-mentioned processes B1, B2, B3, B4, and B5' to generate audio signals to be supplied to each speaker.

[0157] In addition, in the process B2, the VBAP gain for each speaker is calculated for each of the 19 spread vectors. After the process B3, the VBAP gain addition value is quantized as necessary.

[0158] In this way, by using the spread end vector to define the area indicating the spread of the sound image as an area of ​​any shape with an arbitrary position as the center position p0, it becomes possible to express the shape of an object and the directionality of the object's sound, and higher quality audio can be obtained by rendering.

[0159] Also, although an example has been described in which the larger of (spread left end azimuth-spread right end azimuth) / 2 and (spread top end elevation-spread bottom end elevation) / 2 is set as the spread value, the smaller of these values ​​may also be set as the spread value.

[0160] Further, although the case where the VBAP gain is calculated for the spread vector p0 has been described as an example here, the VBAP gain may not be calculated for the spread vector p0. In the following, the description will be continued assuming that the VBAP gain is calculated for the spread vector p0 as well.

[0161] Also, as in the case of the spread three-dimensional vector method, the number of spread vectors to be generated may be determined according to the ratio of, for example, (spread left end azimuth-spread right end azimuth) to (spread upper end elevation-spread lower end elevation).

[0162] (spread radiation vector method) Also, the spread radiation vector method will be explained.

[0163] In the spread radiation vector method, a spread radiation vector, which is a three-dimensional vector, is stored in the bitstream and transmitted. Here, for example, the spread radiation vector is stored in the metadata of the frame of each audio signal for each object. In this case, the metadata also stores spread, which indicates the degree of spread of the sound image.

[0164] The spread radiation vector is a vector that indicates the relative position of the center position p0 of the area indicating the spread of the sound image of the object with respect to the object's position p. For example, the spread radiation vector is a three-dimensional vector consisting of three elements: azimuth, which indicates the horizontal angle from position p to the center position p0, elevation, which indicates the vertical angle from position p to the center position p0, and radius, which indicates the radial distance from the center position p0.

[0165] That is, the spread radiation vector = (azimuth, elevation, radius).

[0166] During rendering processing, the position indicated by the vector obtained by adding this spread radial vector and vector p is set as the center position p0, and spread vectors p0 to p18 are calculated as spread vectors. Here, spread vector p0 is a vector p0 whose start point is the origin O and whose end point is the center position p0, as shown in Fig. 5, for example. In Fig. 5, the same reference numerals are used to designate parts corresponding to those in Fig. 3, and their explanation will be omitted as appropriate.

[0167] In addition, in FIG. 5, the dotted arrows represent spread vectors, and in FIG. 5, only nine spread vectors are drawn to make the figure easier to see.

[0168] In the example shown in Fig. 3, position p=center position p0, but in the example shown in Fig. 5, center position p0 is a position different from position p. In this example, the end position of the vector obtained by vector addition of vector p and the spread radial vector indicated by arrow B11 is the center position p0.

[0169] Also, it can be seen that an area R31 indicating the spread of a sound image centered on a central position p0 is shifted to the left in the figure with respect to the position p, which is the position of the object, compared to the example in FIG.

[0170] In this way, if it is possible to specify any position as the center position p0 of the area indicating the spread of the sound image using the spread radiation vector and position p, it becomes possible to express the sound directionality of the object more accurately.

[0171] In the spread radial vector method, once the spread vectors p0 to p18 are obtained, a process B1 is performed on the vector p, and a process B2 is performed on the spread vectors p0 to p18.

[0172] In the process B2, the VBAP gain may be calculated for each of the 19 spread vectors, or the VBAP gain may be calculated only for the spread vectors p1 to p18 excluding the spread vector p0. In the following, the explanation will be continued assuming that the VBAP gain is also calculated for the spread vector p0.

[0173] After the VBAP gain of each vector is calculated, processing B3, processing B4, and processing B5' are performed to generate an audio signal to be supplied to each speaker. After processing B3, the VBAP gain addition value is quantized as necessary.

[0174] Even with the spread radial vector method described above, it is possible to obtain sufficiently high quality audio by rendering.

[0175] (Arbitrary spread vector method) Next, the arbitrary spread vector method will be described.

[0176] In the arbitrary spread vector method, spread vector number information indicating the number of spread vectors for calculating the VBAP gain and spread vector position information indicating the end point position of each spread vector are stored in the bitstream and transmitted. Here, for example, the spread vector number information and spread vector position information are stored in the metadata of the frame of each audio signal for each object. In this case, the spread indicating the degree of spread of the sound image is not stored in the metadata.

[0177] During rendering processing, a vector having the origin O as its start point and the position indicated by the spread vector position information as its end point is calculated as a spread vector based on each spread vector position information.

[0178] Then, process B1 is performed on vector p, and process B2 is performed on each spread vector. After the VBAP gain of each vector is calculated, processes B3, B4, and B5' are performed to generate an audio signal to be supplied to each speaker. After process B3, the VBAP gain addition value is quantized as necessary.

[0179] In the above-described arbitrary spread vector method, it is possible to arbitrarily specify the range and shape of the sound image to be spread, so that sufficiently high quality sound can be obtained by rendering.

[0180] <About switching processes> With this technology, an appropriate process is selected during rendering depending on factors such as the hardware scale of the renderer, making it possible to obtain the highest quality audio within the allowable amount of processing.

[0181] That is, in this technology, in order to enable switching between a plurality of processes, an index for switching between processes is stored in a bitstream and transmitted from an encoding device to a decoding device. That is, an index for switching between processes is added to the bitstream syntax.

[0182] For example, the following processing is performed depending on the value of the index.

[0183] That is, when index=0, the decoding device, or more specifically, the renderer in the decoding device, performs rendering similar to that in the conventional MPEG-H 3D Audio standard.

[0184] Also, for example, when index=1, a predetermined combination of indexes among the combinations of indexes indicating each of the 18 spread vectors in the conventional MPEG-H 3D Audio standard is stored in the bitstream and transmitted. In this case, the renderer calculates the VBAP gain for the spread vectors indicated by the indexes stored in the bitstream and transmitted.

[0185] Furthermore, for example, when index=2, information indicating the number of spread vectors used in processing and an index indicating which of the 18 spread vectors in the conventional MPEG-H 3D Audio standard the spread vector used in processing is is stored in the bitstream and transmitted.

[0186] Also, for example, when index=3, the rendering process is performed using the arbitrary spread vector method described above, and when index=4, the rendering process binarizes the VBAP gain addition value described above. Furthermore, when index=5, the rendering process is performed using the spread center vector method described above.

[0187] Also, instead of specifying an index for switching processes in the encoding device, the process may be selected in a renderer in the decoding device.

[0188] In such a case, it is possible to switch the processing based on the importance information included in the metadata of the object. Specifically, for example, the processing indicated by the above-mentioned index index=0 may be performed for an object whose importance information indicates high importance (above a predetermined value), and the processing indicated by the above-mentioned index index=4 may be performed for an object whose importance information indicates low importance (below a predetermined value).

[0189] In this way, by appropriately switching the processing during rendering, it is possible to obtain the highest quality audio within the range of the allowable processing amount depending on the hardware scale of the renderer, etc.

[0190] <Configuration example of voice processing device> Next, a more specific embodiment of the present technology described above will be described.

[0191] FIG. 6 is a diagram showing an example of the configuration of a voice processing device to which the present technology is applied.

[0192] Speakers 12-1 to 12-M corresponding to M channels are connected to the audio processing device 11 shown in Fig. 6. The audio processing device 11 generates audio signals for each channel based on audio signals and metadata of objects supplied from the outside, and supplies these audio signals to the speakers 12-1 to 12-M to play back audio.

[0193] In the following description, when there is no need to particularly distinguish between the speakers 12-1 to 12-M, they will be simply referred to as speakers 12. These speakers 12 are audio output units that output audio based on supplied audio signals.

[0194] The speakers 12 are arranged so as to surround a user who is listening to content, etc. For example, the speakers 12 are arranged on the unit sphere described above.

[0195] The voice processing device 11 includes an acquisition unit 21, a vector calculation unit 22, a gain calculation unit 23, and a gain adjustment unit 24.

[0196] The acquisition unit 21 acquires audio signals of objects and metadata for each frame of the audio signal of each object from the outside. For example, the audio signals and metadata are obtained by decoding, by a decoding device, encoded audio data and encoded metadata included in a bitstream output from an encoding device.

[0197] The acquisition unit 21 supplies the acquired audio signal to the gain adjustment unit 24, and also supplies the acquired metadata to the vector calculation unit 22. Here, the metadata includes, as necessary, for example, position information indicating the position of an object, importance information indicating the importance of an object, spread indicating the degree of spread of a sound image of an object, and the like.

[0198] The vector calculation unit 22 calculates a spread vector based on the metadata supplied from the acquisition unit 21, and supplies the spread vector to the gain calculation unit 23. In addition, the vector calculation unit 22 also supplies the position p of the object indicated by the position information included in the metadata, i.e., the vector p indicating the position p, to the gain calculation unit 23 as necessary.

[0199] The gain calculation unit 23 calculates the VBAP gain of the speaker 12 corresponding to each channel by VBAP based on the spread vector and vector p supplied from the vector calculation unit 22, and supplies the VBAP gain to the gain adjustment unit 24. The gain calculation unit 23 also includes a quantization unit 31 that quantizes the VBAP gain of each speaker.

[0200] The gain adjustment unit 24 performs gain adjustment on the audio signal of the object supplied from the acquisition unit 21 based on each VBAP gain supplied from the gain calculation unit 23, and supplies the resulting audio signals of each of the M channels to the speaker 12.

[0201] The gain adjustment unit 24 includes an amplifier 32-1 to an amplifier 32-M. The amplifiers 32-1 to 32-M multiply the audio signal supplied from the acquisition unit 21 by the VBAP gain supplied from the gain calculation unit 23, and supply the resulting audio signal to the speakers 12-1 to 12-M to reproduce sound.

[0202] In the following description, the amplifiers 32-1 to 32-M will be simply referred to as amplifiers 32 unless there is a need to distinguish between them.

[0203] <Explanation of Regeneration Treatment> Next, the operation of the voice processing device 11 shown in FIG. 6 will be described.

[0204] When an audio signal and metadata of an object are supplied from the outside, the audio processing device 11 performs a playback process to play back the audio of the object.

[0205] Hereinafter, the reproduction process by the audio processing device 11 will be described with reference to the flowchart of Fig. 7. Note that this reproduction process is performed for each frame of the audio signal.

[0206] In step S11, the acquisition unit 21 acquires an audio signal and metadata for one frame of an object from outside, and supplies the audio signal to the amplification unit 32 and also supplies the metadata to the vector calculation unit 22.

[0207] In step S12, the vector calculation unit 22 performs a spread vector calculation process based on the metadata supplied from the acquisition unit 21, and supplies the resultant spread vector to the gain calculation unit 23. In addition, the vector calculation unit 22 also supplies the vector p to the gain calculation unit 23 as necessary.

[0208] Details of the spread vector calculation process will be described later, but in this spread vector calculation process, the spread vector is calculated using the above-mentioned spread three-dimensional vector method, spread center vector method, spread end vector method, spread radial vector method, or arbitrary spread vector method.

[0209] In step S13, the gain calculation unit 23 calculates the VBAP gain of each speaker 12 based on pre-stored placement position information indicating the placement position of each speaker 12 and the spread vector and vector p supplied from the vector calculation unit 22.

[0210] That is, for each vector, such as the spread vector and vector p, the VBAP gain of each speaker 12 is calculated. As a result, for each vector, such as the spread vector and vector p, the VBAP gain of one or more speakers 12 located near the position of the object, more specifically, near the position indicated by the vector, is obtained. Note that the VBAP gain of the spread vector is always calculated, but if the vector p is not supplied from the vector calculation unit 22 to the gain calculation unit 23 by the process of step S12, the VBAP gain of the vector p is not calculated.

[0211] In step S14, the gain calculation unit 23 calculates a VBAP gain addition value by adding the VBAP gains calculated for each vector for each speaker 12. That is, the addition value (sum) of the VBAP gains of each vector calculated for the same speaker 12 is calculated as the VBAP gain addition value.

[0212] In step S15, the quantization unit 31 determines whether or not to binarize the VBAP gain addition value.

[0213] For example, whether or not to perform binarization may be determined based on the above-mentioned index, or may be determined based on the importance of an object indicated by importance information as metadata.

[0214] When the determination is made based on the index, for example, the index read out from the bit stream may be supplied to the gain calculation unit 23. When the determination is made based on importance information, the importance information may be supplied from the vector calculation unit 22 to the gain calculation unit 23.

[0215] If it is determined in step S15 that binarization is to be performed, in step S16, the quantization unit 31 binarizes the sum of the VBAP gains calculated for each speaker 12, that is, the VBAP gain sum, and then the process proceeds to step S17.

[0216] On the other hand, if it is determined in step S15 that binarization is not to be performed, the process of step S16 is skipped and the process proceeds to step S17.

[0217] In step S17, the gain calculation unit 23 normalizes the VBAP gain of each speaker 12 so that the sum of the squares of the VBAP gains of all the speakers 12 becomes one.

[0218] That is, the sum of the VBAP gains calculated for each speaker 12 is normalized so that the sum of the squares of all the sums becomes 1. The gain calculation unit 23 supplies the VBAP gains of each speaker 12 obtained by the normalization to the amplification units 32 corresponding to those speakers 12.

[0219] In step S18, the amplifier 32 multiplies the audio signal supplied from the acquisition unit 21 by the VBAP gain supplied from the gain calculation unit 23, and supplies the result to the speaker 12.

[0220] Then, in step S19, the amplifier 32 causes the speaker 12 to reproduce sound based on the supplied audio signal, and the reproduction process ends. As a result, the sound image of the object is localized in the desired partial space in the reproduction space.

[0221] In this manner, the audio processing device 11 calculates the spread vector based on the metadata, calculates the VBAP gain of each vector for each speaker 12, and normalizes the sum of the VBAP gains for each speaker 12. By calculating the VBAP gain for the spread vector in this manner, the spread of the sound image of an object, particularly the shape of the object and the directivity of the sound, can be expressed, and audio of higher quality can be obtained.

[0222] Moreover, by binarizing the added value of the VBAP gain as necessary, not only can the amount of processing required during rendering be reduced, but appropriate processing can be performed according to the processing capabilities (hardware scale) of the audio processing device 11, thereby enabling audio of the highest possible quality to be obtained.

[0223] <Explanation of spread vector calculation process> Now, with reference to the flowchart in FIG. 8, the spread vector calculation process corresponding to the process in step S12 in FIG. 7 will be described.

[0224] In step S41, the vector calculation section 22 determines whether or not to calculate a spread vector based on the spread three-dimensional vector.

[0225] For example, the method by which the spread vector is calculated may be determined based on the index, as in step S15 of FIG. 7, or may be determined based on the importance of the object indicated by the importance information.

[0226] In step S41, if it is determined that the spread vector is calculated based on the spread three-dimensional vector, that is, if it is determined that the spread vector is calculated by the spread three-dimensional vector method, the process proceeds to step S42.

[0227] In step S42, the vector calculation section 22 performs a spread vector calculation process based on the spread three-dimensional vector, and supplies the obtained vector to the gain calculation section 23. The spread vector calculation process based on the spread three-dimensional vector will be described in detail later.

[0228] When the spread vector is calculated, the spread vector calculation process ends, and the process then proceeds to step S13 in FIG.

[0229] On the other hand, if it is determined in step S41 that the spread vector is not to be calculated based on the spread three-dimensional vector, the process proceeds to step S43.

[0230] In step S43, the vector calculation section 22 determines whether or not to calculate a spread vector based on the spread center vector.

[0231] If it is determined in step S43 that the spread vector is to be calculated based on the spread center vector, that is, if it is determined that the spread vector is to be calculated by the spread center vector method, the process proceeds to step S44.

[0232] In step S44, the vector calculation section 22 performs a spread vector calculation process based on the spread center vector, and supplies the obtained vector to the gain calculation section 23. The spread vector calculation process based on the spread center vector will be described in detail later.

[0233] When the spread vector is calculated, the spread vector calculation process ends, and the process then proceeds to step S13 in FIG.

[0234] On the other hand, if it is determined in step S43 that the spread vector is not to be calculated based on the spread center vector, the process proceeds to step S45.

[0235] In step S45, the vector calculation section 22 determines whether or not to calculate a spread vector based on the spread end vector.

[0236] If it is determined in step S45 that the spread vector is to be calculated based on the spread end vector, that is, if it is determined that the spread vector is to be calculated by the spread end vector method, the process proceeds to step S46.

[0237] In step S46, the vector calculation section 22 performs a spread vector calculation process based on the spread end vector, and supplies the obtained vector to the gain calculation section 23. The spread vector calculation process based on the spread end vector will be described in detail later.

[0238] When the spread vector is calculated, the spread vector calculation process ends, and the process then proceeds to step S13 in FIG.

[0239] If it is determined in step S45 that the spread vector is not to be calculated based on the spread end vector, the process proceeds to step S47.

[0240] In step S47, the vector calculation unit 22 determines whether or not to calculate a spread vector based on the spread radial vector.

[0241] If it is determined in step S47 that the spread vector is to be calculated based on the spread radial vector, that is, if it is determined that the spread vector is to be calculated by the spread radial vector method, the process proceeds to step S48.

[0242] In step S48, the vector calculation section 22 performs a spread vector calculation process based on the spread radiation vector, and supplies the obtained vector to the gain calculation section 23. The spread vector calculation process based on the spread radiation vector will be described in detail later.

[0243] When the spread vector is calculated, the spread vector calculation process ends, and the process then proceeds to step S13 in FIG.

[0244] Also, if it is determined in step S47 that the spread vector is not to be calculated based on the spread radial vector, that is, if it is determined that the spread vector is to be calculated by the arbitrary spread vector method, the process proceeds to step S49.

[0245] In step S49, the vector calculation section 22 performs a spread vector calculation process based on the spread vector position information, and supplies the obtained vector to the gain calculation section 23. The spread vector calculation process based on the spread vector position information will be described in detail later.

[0246] When the spread vector is calculated, the spread vector calculation process ends, and the process then proceeds to step S13 in FIG.

[0247] In this manner, the audio processing device 11 calculates the spread vector by an appropriate method among a plurality of methods. By calculating the spread vector by an appropriate method in this manner, it is possible to obtain the highest quality audio within the range of the allowable amount of processing depending on the hardware scale of the renderer, etc.

[0248] <Explanation of spread vector calculation process based on spread three-dimensional vector> Next, the processes corresponding to steps S42, S44, S46, S48, and S49 described with reference to FIG. 8 will be described in detail.

[0249] First, the spread vector calculation process based on the spread three-dimensional vector, which corresponds to step S42 in FIG. 8, will be described with reference to the flowchart in FIG.

[0250] In step S81, the vector calculation unit 22 sets the position indicated by the position information included in the metadata supplied from the acquisition unit 21 as the object position p. That is, the vector indicating the position p is set as the vector p.

[0251] In step S82, the vector calculation unit 22 calculates the spread based on the spread three-dimensional vector included in the metadata supplied from the acquisition unit 21. Specifically, the vector calculation unit 22 calculates the spread by calculating the above-mentioned formula (1).

[0252] In step S83, the vector calculation section 22 calculates spread vectors p0 to p18 based on the vector p and spread.

[0253] Here, vector p is set as vector p0 indicating center position p0, and vector p is directly set as spread vector p0. As for spread vectors p1 to p18, similarly to the case of the MPEG-H 3D Audio standard, each spread vector is calculated so as to be vertically and horizontally symmetrical within an area determined by the angle indicated by spread on the unit sphere centered on center position p0.

[0254] In step S84, the vector calculation section 22 determines whether or not s3_azimuth≧s3_elevation, that is, whether or not s3_azimuth is greater than s3_elevation, based on the spread three-dimensional vector.

[0255] If it is determined in step S84 that s3_azimuth≧s3_elevation, in step S85, the vector calculation unit 22 changes the elevation of the spread vectors p1 to p18. That is, the vector calculation unit 22 performs the calculation of the above-mentioned formula (2) to correct the elevation of each spread vector to obtain a final spread vector.

[0256] When the final spread vectors are obtained, the vector calculation unit 22 supplies the spread vectors p0 to p18 to the gain calculation unit 23, and the spread vector calculation process based on the spread three-dimensional vector is completed. Then, the process of step S42 in FIG. 8 is completed, and the process then proceeds to step S13 in FIG.

[0257] On the other hand, if it is determined in step S84 that s3_azimuth≧s3_elevation is not satisfied, then in step S86, the vector calculation unit 22 changes the azimuths of the spread vectors p1 to p18. That is, the vector calculation unit 22 performs the calculation of the above-mentioned formula (3) to correct the azimuth of each spread vector to obtain a final spread vector.

[0258] When the final spread vectors are obtained, the vector calculation unit 22 supplies the spread vectors p0 to p18 to the gain calculation unit 23, and the spread vector calculation process based on the spread three-dimensional vector is completed. Then, the process of step S42 in FIG. 8 is completed, and the process then proceeds to step S13 in FIG.

[0259] In this manner, the sound processing device 11 calculates each spread vector by the spread three-dimensional vector method. This makes it possible to express the shape of an object and the directivity of the sound of the object, thereby obtaining a higher quality sound.

[0260] <Explanation of spread vector calculation process based on spread center vector> Next, the spread vector calculation process based on the spread center vector, which corresponds to step S44 in FIG. 8, will be described with reference to the flowchart in FIG.

[0261] The process of step S111 is similar to the process of step S81 in FIG. 9, and therefore the description thereof will be omitted.

[0262] In step S112, the vector calculation unit 22 calculates the spread vectors p0 to p18 based on the spread center vector and the spread included in the metadata supplied from the acquisition unit 21.

[0263] Specifically, the vector calculation unit 22 sets the position indicated by the spread center vector as the center position p0, and sets the vector indicating the center position p0 as the spread vector p0. The vector calculation unit 22 also calculates spread vectors p1 to p18 so as to be symmetrical up, down, left, and right within an area determined by the angle indicated by spread on the unit sphere centered on the center position p0. These spread vectors p1 to p18 are basically calculated in the same manner as in the MPEG-H 3D Audio standard.

[0264] The vector calculation unit 22 supplies the vector p obtained by the above process and the spread vectors p0 to p18 to the gain calculation unit 23, and the spread vector calculation process based on the spread center vector is completed. Then, the process of step S44 in FIG. 8 is completed, and the process proceeds to step S13 in FIG. 7.

[0265] In this way, the voice processing device 11 calculates the vector p and each spread vector by the spread center vector method. This makes it possible to express the shape of an object and the directivity of the sound of the object, thereby obtaining voice of higher quality.

[0266] In the spread vector calculation process based on the spread center vector, the spread vector p0 may not be supplied to the gain calculation section 23. In other words, the VBAP gain may not be calculated for the spread vector p0.

[0267] <Explanation of spread vector calculation process based on spread end vector> Furthermore, the spread vector calculation process based on the spread end vector, which corresponds to step S46 in FIG. 8, will be described with reference to the flowchart in FIG.

[0268] The process of step S141 is similar to the process of step S81 in FIG. 9, and therefore the description thereof will be omitted.

[0269] In step S142, the vector calculation unit 22 calculates the center position p0, that is, the vector p0, based on the spread end vector included in the metadata supplied from the acquisition unit 21. Specifically, the vector calculation unit 22 calculates the center position p0 by calculating the above-mentioned formula (4).

[0270] In step S143, the vector calculation unit 22 calculates the spread based on the spread end vector. Specifically, the vector calculation unit 22 calculates the spread by calculating the above-mentioned formula (5).

[0271] In step S144, the vector calculation section 22 calculates the spread vectors p0 to p18 based on the center position p0 and the spread.

[0272] Here, vector p0 indicating the central position p0 is directly used as spread vector p0. As for spread vectors p1 to p18, similarly to the case of the MPEG-H 3D Audio standard, each spread vector is calculated so as to be vertically and horizontally symmetrical within an area determined by the angle indicated by spread on the unit sphere centered on the central position p0.

[0273] In step S145, the vector calculation unit 22 determines whether (spread left end azimuth-spread right end azimuth)≧(spread upper end elevation-spread lower end elevation), that is, whether (spread left end azimuth-spread right end azimuth) is greater than (spread upper end elevation-spread lower end elevation).

[0274] If it is determined in step S145 that (spread left end azimuth-spread right end azimuth)≧(spread upper end elevation-spread lower end elevation), in step S146, the vector calculation unit 22 changes the elevation of the spread vectors p1 to p18. That is, the vector calculation unit 22 performs the calculation of the above-mentioned formula (6) to correct the elevation of each spread vector to obtain a final spread vector.

[0275] When the final spread vector is obtained, the vector calculation unit 22 supplies the spread vectors p0 to p18 and the vector p to the gain calculation unit 23, and the spread vector calculation process based on the spread end vector is completed. Then, the process of step S46 in FIG. 8 is completed, and the process then proceeds to step S13 in FIG.

[0276] On the other hand, if it is determined in step S145 that (spread left end azimuth-spread right end azimuth)≧(spread upper end elevation-spread lower end elevation) is not true, then in step S147, the vector calculation unit 22 changes the azimuths of the spread vectors p1 to p18. That is, the vector calculation unit 22 performs the calculation of the above-mentioned formula (7) to correct the azimuth of each spread vector to obtain a final spread vector.

[0277] When the final spread vector is obtained, the vector calculation unit 22 supplies the spread vectors p0 to p18 and the vector p to the gain calculation unit 23, and the spread vector calculation process based on the spread end vector is completed. Then, the process of step S46 in FIG. 8 is completed, and the process then proceeds to step S13 in FIG.

[0278] In this way, the sound processing device 11 calculates each spread vector by the spread end vector method. This makes it possible to express the shape of an object and the directivity of the sound of the object, thereby obtaining a higher quality sound.

[0279] In addition, in the spread vector calculation process based on the spread end vector, the spread vector p0 may not be supplied to the gain calculation section 23. In other words, the VBAP gain may not be calculated for the spread vector p0.

[0280] <Explanation of spread vector calculation process based on spread radiation vector> Next, the spread vector calculation process based on the spread radial vector, which corresponds to step S48 in FIG. 8, will be described with reference to the flowchart in FIG.

[0281] The process of step S171 is similar to the process of step S81 in FIG. 9, and therefore the description thereof will be omitted.

[0282] In step S172, the vector calculation unit 22 calculates spread vectors p0 to p18 based on the object position p and the spread radial vector and spread included in the metadata supplied from the acquisition unit 21.

[0283] Specifically, the vector calculation unit 22 determines the position indicated by the vector obtained by adding the vector p indicating the object position p and the spread radial vector as the central position p0. The vector indicating this central position p0 is the vector p0, and the vector calculation unit 22 determines the vector p0 as it is as the spread vector p0.

[0284] Furthermore, the vector calculation unit 22 calculates spread vectors p1 to p18 so as to be symmetrical in the vertical and horizontal directions within an area determined by the angle indicated by spread on the unit sphere centered on the central position p0. These spread vectors p1 to p18 are basically calculated in the same manner as in the MPEG-H 3D Audio standard.

[0285] The vector calculation unit 22 supplies the vector p obtained by the above processing and the spread vectors p0 to p18 to the gain calculation unit 23, and the spread vector calculation processing based on the spread radiation vector is completed. Then, the processing of step S48 in Fig. 8 is completed, and the processing then proceeds to step S13 in Fig. 7.

[0286] In this manner, the sound processing device 11 calculates the vector p and each spread vector by the spread radiation vector method. This makes it possible to express the shape of an object and the directivity of the sound of the object, thereby obtaining a higher quality sound.

[0287] In addition, in the spread vector calculation process based on the spread radial vector, the spread vector p0 may not be supplied to the gain calculation unit 23. In other words, the VBAP gain may not be calculated for the spread vector p0.

[0288] <Explanation of spread vector calculation process based on spread vector position information> Next, the spread vector calculation process based on the spread vector position information, which corresponds to step S49 in FIG. 8, will be described with reference to the flowchart in FIG.

[0289] The process of step S201 is similar to the process of step S81 in FIG. 9, and therefore the description thereof will be omitted.

[0290] In step S202, the vector calculation unit 22 calculates a spread vector based on the spread vector number information and the spread vector position information included in the metadata supplied from the acquisition unit 21.

[0291] Specifically, the vector calculation unit 22 calculates a vector having the origin O as a start point and the position indicated by the spread vector position information as an end point as a spread vector. Here, the number of spread vectors calculated is the number indicated by the spread vector number information.

[0292] The vector calculation unit 22 supplies the vector p and the spread vector obtained by the above processing to the gain calculation unit 23, and the spread vector calculation processing based on the spread vector position information ends. Then, the processing of step S49 in Fig. 8 ends, and thereafter, the processing proceeds to step S13 in Fig. 7.

[0293] In this way, the voice processing device 11 calculates the vector p and each spread vector by the arbitrary spread vector method. This makes it possible to express the shape of an object and the directivity of the sound of the object, thereby obtaining voice of higher quality.

[0294] Second embodiment Reducing the amount of rendering processing As described above, VBAP is known as a technique for controlling the position of a sound image using a plurality of speakers, that is, a technique for performing a rendering process.

[0295] In VBAP, by outputting sound from three speakers, it is possible to localize the sound image at any point inside the triangle formed by those three speakers. In the following, such a triangle formed by three speakers will be called a mesh.

[0296] Rendering by VBAP is performed for each object, so when there are a large number of objects, such as in a game, the rendering process becomes very busy. Therefore, a renderer with a small hardware scale cannot render all objects, and as a result, only the sounds of a limited number of objects may be played. This can result in a loss of realism and sound quality when playing audio.

[0297] Therefore, the present technology makes it possible to reduce the amount of processing required for rendering while suppressing deterioration in the sense of realism and sound quality.

[0298] The present technology will be described below.

[0299] In normal VBAP processing, that is, rendering processing, the above-mentioned processes A1 to A3 are performed for each object, and audio signals for each speaker are generated.

[0300] In reality, the number of speakers for which the VBAP gain is calculated is three, and the VBAP gain for each speaker is calculated for each sample that constitutes the audio signal, so the multiplication process in process A3 involves (the number of samples of the audio signal × 3) multiplications.

[0301] In response to this, the present technology reduces the amount of processing required for rendering by appropriately combining equal gain processing for VBAP gain, i.e., quantization processing of VBAP gain, and mesh number switching processing that changes the number of meshes used when calculating VBAP gain.

[0302] (Quantization processing) First, the quantization process will be described. Here, binarization and ternarization will be described as examples of the quantization process.

[0303] When binarization is performed as the quantization process, after the process A1 is performed, the VBAP gain obtained for each speaker by the process A1 is binarized. In the binarization, for example, the VBAP gain of each speaker is set to either a value of 0 or 1.

[0304] The VBAP gain may be binarized by any method, such as rounding off, ceiling (rounding up), flooring (rounding down), or threshold processing.

[0305] Once the VBAP gain has been binarized in this manner, processing A2 and processing A3 are then carried out to generate audio signals for each speaker.

[0306] At this time, in process A2, normalization is performed based on the binarized VBAP gain, so just like in the above-mentioned quantization of the spread vector, the final VBAP gain for each speaker will have one value excluding 0. In other words, when the VBAP gain is binarized, the final VBAP gain value for each speaker will be either 0 or a predetermined value.

[0307] Therefore, in the multiplication process in process A3, it is sufficient to perform (the number of samples of the audio signal×1) multiplications, so that the amount of processing in the rendering process can be significantly reduced.

[0308] Similarly, after process A1, the VBAP gain obtained for each speaker may be converted to three values. In such a case, the VBAP gain obtained for each speaker by process A1 is converted to three values ​​and set to any one of 0, 0.5, and 1. Then, processes A2 and A3 are performed to generate audio signals for each speaker.

[0309] Therefore, the number of multiplications in the multiplication process in process A3 is a maximum of (the number of samples of the audio signal×2), making it possible to significantly reduce the amount of processing in the rendering process.

[0310] Note that, although the case where the VBAP gain is binarized or ternarized will be described as an example here, the VBAP gain may be quantized to a value equal to or greater than 4. In general, for example, if the VBAP gain is quantized to one of x gains equal to or greater than 2, that is, if the VBAP gain is quantized by a quantization number x, the number of multiplication processes in process A3 will be a maximum of (x-1) times.

[0311] By quantizing the VBAP gain as described above, the amount of processing in the rendering process can be reduced. If the amount of processing in the rendering process is reduced in this way, it becomes possible to render all objects even when there are a large number of objects, so that deterioration in the sense of realism and sound quality during audio playback can be minimized. In other words, it is possible to reduce the amount of processing in the rendering process while suppressing deterioration in the sense of realism and sound quality.

[0312] (Mesh number switching process) Next, the mesh number switching process will be described.

[0313] In VBAP, as described with reference to Fig. 1, for example, a vector p indicating the position p of the sound image of the object to be processed is expressed as a linear sum of vectors l1 to l3 pointing in the directions of three speakers SP1 to SP3, and coefficients g1 to g3 multiplied by these vectors are set as the VBAP gains of each speaker. In the example of Fig. 1, a triangular area TR11 surrounded by speakers SP1 to SP3 is one mesh.

[0314] When calculating the VBAP gain, the inverse matrix L of the triangular mesh is calculated using the following formula (8): 123 -1 Three coefficients g1 to g3 are calculated based on the position p of the sound image of the object.

[0315]

number

[0316] In equation (8), p1, p2, and p3 represent the x-coordinate, y-coordinate, and z-coordinate on the Cartesian coordinate system indicating the position p of the sound image of the object, that is, the three-dimensional coordinate system shown in FIG.

[0317] Also, 11 , l 12 , and l 13 are the values ​​of the x-, y-, and z-components when vector l1 pointing toward the first speaker SP1 that makes up the mesh is decomposed into x-, y-, and z-axis components, and correspond to the x-, y-, and z-coordinates of the first speaker SP1.

[0318] Similarly, l 21 , l 22 , and l 23 are the values ​​of the x-, y-, and z-components when the vector l2 directed toward the second speaker SP2 that constitutes the mesh is decomposed into the x-, y-, and z-axis components. 31 , l 32 , and l 33 are the values ​​of the x-, y-, and z-components when a vector l3 directed toward the third speaker SP3 that constitutes the mesh is resolved into x-, y-, and z-axis components.

[0319] Furthermore, the transformation from p1, p2, and p3 in the three-dimensional coordinate system of the position p to coordinates θ, γ, and r in the spherical coordinate system when r = 1 is defined as shown in the following equation (9), where θ, γ, and r are the horizontal angle azimuth, vertical angle elevation, and distance radius, respectively, as described above.

[0320]

number

[0321] As described above, in the space on the content playback side, that is, the playback space, multiple speakers are arranged on a unit sphere, and one mesh is composed of three speakers out of the multiple speakers. Basically, the entire surface of the unit sphere is covered with multiple meshes without any gaps. Moreover, each mesh is determined so that it does not overlap with another mesh.

[0322] In VBAP, if sound is output from two or three speakers that constitute a mesh containing the object's position p among the speakers arranged on the surface of a unit sphere, the sound image can be localized at position p, so the VBAP gain of speakers other than those that constitute that mesh will be 0.

[0323] Therefore, when calculating the VBAP gain, it is sufficient to identify one mesh that includes the position p of the object and calculate the VBAP gain of the speaker that constitutes that mesh. For example, whether or not a specific mesh is a mesh that includes the position p can be determined from the calculated VBAP gain.

[0324] That is, if all of the VBAP gains of the three speakers calculated for a mesh are equal to or greater than 0, the mesh is a mesh that includes the object's position p. Conversely, if any one of the VBAP gains of the three speakers is a negative value, the object's position p is located outside the mesh that includes those speakers, and the calculated VBAP gain is not a correct VBAP gain.

[0325] Therefore, when calculating the VBAP gain, each mesh is selected one by one in turn as the mesh to be processed, and the calculation of the above-mentioned equation (8) is performed for the mesh to be processed, and the VBAP gain of each speaker that constitutes the mesh is calculated.

[0326] Then, from the calculation results of these VBAP gains, it is determined whether the mesh to be processed is a mesh that includes the position p of the object, and if it is determined that the mesh does not include position p, the next mesh is set as the new mesh to be processed and similar processing is performed.

[0327] On the other hand, if it is determined that the mesh to be processed is a mesh that includes the object position p, the VBAP gains of the speakers that make up that mesh are set to the calculated VBAP gains, and the VBAP gains of the other speakers are set to 0. This results in the VBAP gains of all the speakers being obtained.

[0328] In this way, in the rendering process, the process of calculating the VBAP gain and the process of identifying the mesh that includes the position p are performed simultaneously.

[0329] That is, in order to obtain a correct VBAP gain, a mesh to be processed is selected and the process of calculating the VBAP gain of the mesh is repeated until a mesh is obtained in which all of the VBAP gains of the speakers constituting the mesh are equal to or greater than 0.

[0330] Therefore, in the rendering process, the more meshes there are on the surface of the unit sphere, the more processing is required to identify the mesh that contains position p, i.e., to obtain the correct VBAP gain.

[0331] Therefore, in this technology, instead of forming (configuring) a mesh using all the speakers in the actual playback environment, a mesh is formed using only a portion of all the speakers, thereby reducing the total number of meshes and the amount of processing during rendering. In other words, in this technology, a mesh number switching process is performed to change the total number of meshes.

[0332] Specifically, for example, in a 22-channel speaker system, a total of 22 speakers, speaker SPK1 to speaker SPK22, are arranged as speakers for each channel on the surface of a unit sphere as shown in Fig. 14. Note that the origin O in Fig. 14 corresponds to the origin O shown in Fig. 2.

[0333] In this way, when 22 speakers are arranged on the surface of the unit sphere, if a mesh is formed using all 22 speakers to cover the surface of the unit sphere, the total number of meshes on the unit sphere will be 40.

[0334] In contrast to this, suppose that a mesh is formed using only six speakers, namely, speaker SPK1, speaker SPK6, speaker SPK7, speaker SPK10, speaker SPK19, and speaker SPK20, out of a total of 22 speakers, speaker SPK1 to speaker SPK22, as shown in Fig. 15. Note that in Fig. 15, parts corresponding to those in Fig. 14 are given the same reference numerals, and their explanation will be omitted as appropriate.

[0335] In the example of Fig. 15, meshes are formed using only a total of 6 speakers out of 22 speakers, so the total number of meshes on the unit sphere is 8, which can significantly reduce the total number of meshes. As a result, in the example shown in Fig. 15, the amount of processing required to calculate the VBAP gain can be reduced to 8 / 40 times that required when comparing with the case in which meshes are formed using all 22 speakers shown in Fig. 14, and the amount of processing can be significantly reduced.

[0336] In this example, the entire surface of the unit sphere is also covered with eight meshes without any gaps, so it is possible to localize the sound image at any position on the surface of the unit sphere. However, the greater the total number of meshes on the surface of the unit sphere, the smaller the area of ​​each mesh becomes, so the greater the total number of meshes, the more precisely it is possible to control the localization of the sound image.

[0337] When the total number of meshes is changed by the mesh number switching process, it is desirable to select speakers to be used to form the new number of meshes from speakers that are different in the vertical direction (up and down direction) as seen by the user at the origin O, that is, in the direction of the vertical angle elevation. In other words, it is desirable to form the new number of meshes using three or more speakers, including speakers that are located at different heights. This is to suppress deterioration of the three-dimensional sound, that is, the sense of realism.

[0338] For example, consider the case where a mesh is formed using some or all of the five speakers SP1 to SP5 arranged on the surface of a unit sphere as shown in Fig. 16. Note that in Fig. 16, the same reference numerals are used to designate parts corresponding to those in Fig. 3, and their explanation will be omitted.

[0339] 16, when meshes that cover the surface of the unit sphere are formed using all five speakers SP1 to SP5, the number of meshes is three. That is, each of three regions is made into a mesh: a triangular region surrounded by speakers SP1 to SP3, a triangular region surrounded by speakers SP2 to SP4, and a triangular region surrounded by speakers SP2, SP4, and SP5.

[0340] In contrast, if only speakers SP1, SP2, and SP5 are used, the mesh will be a two-dimensional arc rather than a triangle. In this case, the sound image of the object can only be localized on the arc connecting speakers SP1 and SP2, or on the arc connecting speakers SP2 and SP5, in the unit sphere.

[0341] If the speakers used to form the mesh in this way were all at the same height in the vertical direction, i.e., speakers in the same layer, the positioning height of the sound images of all objects would be the same height, which would degrade the sense of realism.

[0342] Therefore, it is desirable to form one or more meshes using three or more speakers, including speakers located at different positions in the vertical direction (perpendicular direction), so as to suppress deterioration of the sense of realism.

[0343] 16, for example, by using the speaker SP1 and the speakers SP3 to SP5 among the speakers SP1 to SP5, two meshes can be formed so as to cover the entire surface of the unit sphere. In this example, the speakers SP1 and SP5, and the speakers SP3 and SP4 are located at different heights.

[0344] In this case, for example, two areas, a triangular area surrounded by the speakers SP1, SP3, and SP5, and a triangular area surrounded by the speakers SP3 to SP5, are each made into a mesh.

[0345] Additionally, in this example, it is also possible to make each of the two areas, the triangular area surrounded by speakers SP1, SP3, and SP4, and the triangular area surrounded by speakers SP1, SP4, and SP5, into a mesh.

[0346] In both of these two examples, the sound image can be localized at any position on the surface of the unit sphere, so that the deterioration of the sense of realism can be suppressed. In addition, in order to form meshes so that the entire surface of the unit sphere is covered with multiple meshes, it is advisable to always use a so-called top speaker located directly above the user. For example, the top speaker is the speaker SPK19 shown in FIG. 14.

[0347] By changing the total number of meshes by performing the mesh number switching process as described above, the amount of processing in the rendering process can be reduced, and deterioration of the sense of realism and sound quality during audio playback can be minimized, as in the case of the quantization process. In other words, the amount of processing in the rendering process can be reduced while suppressing deterioration of the sense of realism and sound quality.

[0348] Selecting whether or not to perform such mesh number switching processing, or selecting the total number of meshes to be used in the mesh number switching processing, can be said to be selecting the total number of meshes to be used in calculating the VBAP gain.

[0349] (Combination of quantization and mesh number switching) In the above, the quantization process and the mesh number switching process have been described as methods for reducing the amount of processing in the rendering process.

[0350] On the renderer side performing the rendering process, any of the processes described as the quantization process or the mesh number switching process may be used in a fixed manner, or these processes may be switched or appropriately combined.

[0351] For example, what processes are to be combined may be determined based on the total number of objects (hereinafter referred to as the number of objects), importance information included in the metadata of the objects, the sound pressure of the audio signals of the objects, etc. Furthermore, the combination of processes, that is, the switching of processes, may be performed for each object or for each frame of the audio signal.

[0352] For example, when switching between processes depending on the number of objects, the following process can be performed.

[0353] For example, when the number of objects is 10 or more, binarization processing is performed for the VBAP gain for all objects. On the other hand, when the number of objects is less than 10, only the above-mentioned processing A1 to A3 is performed for all objects as in the conventional method.

[0354] In this way, by performing conventional processing when there are a small number of objects, and performing binarization processing when there are a large number of objects, rendering can be performed adequately even with a renderer with a small hardware scale, and the highest possible quality audio can be obtained.

[0355] Furthermore, when switching processing depending on the number of objects, mesh number switching processing may be performed depending on the number of objects, and the total number of meshes may be changed appropriately.

[0356] In this case, for example, if the number of objects is 10 or more, the total number of meshes can be 8, and if the number of objects is less than 10, the total number of meshes can be 40. Also, the total number of meshes may be changed in multiple stages according to the number of objects, so that the total number of meshes decreases as the number of objects increases.

[0357] In this way, by changing the total number of meshes according to the number of objects, the amount of processing can be adjusted according to the hardware scale of the renderer, making it possible to obtain the highest possible quality of audio.

[0358] Furthermore, when the process is switched based on the importance information included in the metadata of the object, the following process can be performed.

[0359] For example, if the importance information of an object is the highest value indicating the highest importance, only processes A1 to A3 are performed as in the conventional method, and if the importance information of an object is a value other than the maximum value, binarization processing is performed on the VBAP gain.

[0360] Alternatively, for example, mesh number switching processing may be performed according to the value of the importance information of the object, and the total number of meshes may be appropriately changed. In this case, the higher the importance of the object, the larger the total number of meshes may be, and the total number of meshes may be changed in multiple stages.

[0361] In these examples, processing can be switched for each object based on the importance information of each object. In the processing described here, the sound quality can be increased for objects with high importance, and the sound quality can be decreased for objects with low importance to reduce the amount of processing. Therefore, when playing back the audio of objects with various levels of importance simultaneously, it is possible to minimize the deterioration of the sound quality as perceived by the ears and reduce the amount of processing, and it can be said that this is a method that strikes a good balance between ensuring sound quality and reducing the amount of processing.

[0362] In this way, when switching processing for each object based on the object's importance information, it is possible to increase the total number of meshes for objects with higher importance, or to not perform quantization processing when the object's importance is high.

[0363] Furthermore, in addition to this, for objects of low importance, i.e. objects whose importance information value is less than a predetermined value, the total number of meshes may be increased or quantization processing may not be performed for objects that are closer to objects of high importance, i.e. objects whose importance information value is equal to or greater than a predetermined value.

[0364] Specifically, for objects with the highest importance information, the total number of meshes is set to 40, and for objects with importance information that is not the highest, the total number of meshes is reduced.

[0365] In this case, for an object whose importance information is not the highest, the shorter the distance between that object and the object whose importance information is the highest, the greater the total number of meshes. Normally, a user pays particular attention to the sound of an object with high importance, so if the sound quality of other objects near that object is low, the user will feel that the sound quality of the entire content is poor. Therefore, by determining the total number of meshes so that the sound quality of objects located close to the object with high importance is as good as possible, the deterioration of the sound quality perceived by the ear can be suppressed.

[0366] Furthermore, the processing may be switched according to the sound pressure of the audio signal of the object. Here, the sound pressure of the audio signal can be found by calculating the square root of the mean square of the sample values ​​of each sample in the frame to be rendered of the audio signal. In other words, the sound pressure RMS can be found by the calculation of the following equation (10).

[0367]

number

[0368] In equation (10), N represents the number of samples that make up a frame of an audio signal, and x n indicates the sample value of the nth sample (where n=0,...,N-1) in the frame.

[0369] When switching processing depending on the sound pressure RMS of the audio signal obtained in this manner, the following processing can be performed.

[0370] For example, when the sound pressure RMS of the audio signal of an object is -6 dB or higher relative to the full scale of the sound pressure RMS of 0 dB, only processes A1 to A3 are performed as in the conventional method, and when the sound pressure RMS of the object is less than -6 dB, binarization processing is performed on the VBAP gain.

[0371] Generally, audio with high sound pressure is more susceptible to deterioration in sound quality, and such audio is often the audio of an important object. Therefore, we have made sure that the sound quality of objects with high RMS sound pressure does not deteriorate, and binarization is performed on objects with low RMS sound pressure, reducing the overall amount of processing. This allows rendering to be performed adequately even on a renderer with a small hardware scale, and still obtains the highest possible quality audio.

[0372] In addition, the total number of meshes may be appropriately changed by performing mesh number switching processing according to the sound pressure RMS of the audio signal of the object. In this case, for example, the total number of meshes may be increased for an object with a larger sound pressure RMS, and the total number of meshes may be changed in multiple stages.

[0373] Furthermore, a combination of quantization processing and mesh number switching processing may be selected according to the number of objects, importance information, and sound pressure RMS.

[0374] That is, based on the number of objects, the importance information, and the sound pressure RMS, it is possible to select whether or not to perform quantization processing, and to what gain the VBAP gain is to be quantized in the quantization processing, that is, the quantization number in the quantization processing, and the total number of meshes used in calculating the VBAP gain, and calculate the VBAP gain by processing according to the selection result. In such a case, for example, the following processing can be performed.

[0375] For example, if there are 10 or more objects, the total number of meshes for all objects is set to 10, and binarization processing is then performed. In this case, since there are a large number of objects, the total number of meshes is reduced and binarization processing is performed, thereby reducing the amount of processing. This makes it possible to render all objects even if the hardware scale of the renderer is small.

[0376] Also, when the number of objects is less than 10 and the importance information value is the highest, only the processes A1 to A3 are performed as in the conventional method. This makes it possible to play back audio for objects with high importance without degrading the sound quality.

[0377] If the number of objects is less than 10, the importance information value is not the highest value, and the sound pressure RMS is -30 dB or higher, the total number of meshes is set to 10, and ternarization processing is performed. This makes it possible to reduce the amount of processing during rendering for audio with low importance but high sound pressure to the extent that the deterioration of sound quality of the audio is not noticeable.

[0378] Furthermore, if the number of objects is less than 10, the importance information value is not the highest value, and the sound pressure RMS is less than -30 dB, the total number of meshes is set to 5, and binarization processing is performed. This makes it possible to sufficiently reduce the amount of processing during rendering for sounds with low importance and low sound pressure.

[0379] In this way, when there are many objects, the amount of rendering processing is reduced so that all objects can be rendered, and when there are a relatively small number of objects, an appropriate process is selected for each object and rendering is performed. This makes it possible to play back audio with sufficient sound quality with an overall small amount of processing, while maintaining a balance between ensuring sound quality and reducing the amount of processing for each object.

[0380] <Configuration example of voice processing device> Next, a description will be given of an audio processing device that performs rendering processing while appropriately performing the above-described quantization processing, mesh number switching processing, etc. Fig. 17 is a diagram showing a specific configuration example of such an audio processing device. Note that in Fig. 17, the same reference numerals are attached to parts corresponding to those in Fig. 6, and the description thereof will be omitted as appropriate.

[0381] 17 includes an acquisition unit 21, a gain calculation unit 23, and a gain adjustment unit 71. The gain calculation unit 23 receives metadata and audio signals of objects from the acquisition unit 21, calculates a VBAP gain for each object for each speaker 12, and supplies the VBAP gain to the gain adjustment unit 71.

[0382] Moreover, the gain calculation unit 23 includes a quantization unit 31 that quantizes the VBAP gain.

[0383] For each object, the gain adjustment unit 71 multiplies the VBAP gain for each speaker 12 supplied from the gain calculation unit 23 by the audio signal supplied from the acquisition unit 21 to generate an audio signal for each speaker 12 and supply it to the speaker 12.

[0384] <Explanation of Regeneration Treatment> Next, a description will be given of the operation of the audio processing device 61 shown in Fig. 17. That is, the playback process by the audio processing device 61 will be described with reference to the flowchart in Fig. 18.

[0385] In this example, it is assumed that the acquisition unit 21 is supplied with the audio signal and metadata of one or more objects for each frame, and the playback process is performed for each frame of the audio signal for each object.

[0386] In step S231, the acquisition unit 21 acquires an audio signal and metadata of an object from outside, supplies the audio signal to the gain calculation unit 23 and the gain adjustment unit 71, and supplies the metadata to the gain calculation unit 23. The acquisition unit 21 also acquires the number of objects that simultaneously play back audio in the frame being processed, i.e., information indicating the number of objects, and supplies the information to the gain calculation unit 23.

[0387] In step S232, the gain calculation unit 23 determines, based on the information indicating the number of objects supplied from the acquisition unit 21, whether the number of objects is 10 or more.

[0388] If it is determined in step S232 that the number of objects is 10 or more, in step S233, the gain calculation unit 23 sets the total number of meshes used when calculating the VBAP gain to 10. That is, the gain calculation unit 23 selects 10 as the total number of meshes.

[0389] Furthermore, the gain calculation unit 23 selects a predetermined number of speakers 12 from among all the speakers 12 so that the total number of meshes selected is equal to the total number of meshes formed on the surface of the unit sphere. Then, the gain calculation unit 23 sets the 10 meshes on the surface of the unit sphere formed by the selected speakers 12 as meshes to be used when calculating the VBAP gain.

[0390] In step S234, the gain calculation unit 23 calculates the VBAP gain of each speaker 12 using VBAP based on the placement position information indicating the placement position of each speaker 12 that constitutes the 10 meshes determined in step S233 and the position information indicating the position of the object included in the metadata supplied from the acquisition unit 21.

[0391] Specifically, the gain calculation unit 23 performs calculation of equation (8) in order with the meshes determined in step S233 as the meshes to be processed, thereby calculating the VBAP gain of each speaker 12. At this time, as described above, a new mesh is set as the mesh to be processed, and the VBAP gain is calculated until all of the VBAP gains calculated for the three speakers 12 constituting the mesh to be processed are equal to or greater than 0.

[0392] In step S235, the quantization unit 31 binarizes the VBAP gain of each speaker 12 obtained in step S234, and then the process proceeds to step S246.

[0393] If it is determined in step S232 that the number of objects is less than 10, the process proceeds to step S236.

[0394] In step S236, the gain calculation unit 23 determines whether or not the value of the importance information of the object included in the metadata supplied from the acquisition unit 21 is the highest value. For example, when the value of the importance information is a numerical value "7" indicating the highest importance, the importance information is determined to be the highest value.

[0395] If it is determined in step S236 that the importance information has the highest value, the process proceeds to step S237.

[0396] In step S237, the gain calculation unit 23 calculates the VBAP gain of each speaker 12 based on the arrangement position information indicating the arrangement position of each speaker 12 and the position information included in the metadata supplied from the acquisition unit 21, and then the process proceeds to step S246. Here, meshes formed by all the speakers 12 are sequentially treated as meshes to be processed, and the VBAP gain is calculated by the calculation of equation (8).

[0397] On the other hand, if it is determined in step S236 that the importance information is not the highest value, in step S238, the gain calculation unit 23 calculates the sound pressure RMS of the audio signal supplied from the acquisition unit 21. Specifically, the calculation of the above-mentioned formula (10) is performed for the frame of the audio signal to be processed, and the sound pressure RMS is calculated.

[0398] In step S239, the gain calculation unit 23 determines whether the sound pressure RMS calculated in step S238 is equal to or greater than −30 dB.

[0399] If it is determined in step S239 that the sound pressure RMS is equal to or greater than -30 dB, the process proceeds to steps S240 and S241. Note that the processes in steps S240 and S241 are similar to those in steps S233 and S234, and therefore will not be described here.

[0400] In step S242, the quantization unit 31 converts the VBAP gain of each speaker 12 obtained in step S241 into three values, and then the process proceeds to step S246.

[0401] If it is determined in step S239 that the sound pressure RMS is less than −30 dB, the process proceeds to step S243.

[0402] In step S243, the gain calculation unit 23 sets the total number of meshes used when calculating the VBAP gain to five.

[0403] In addition, the gain calculation unit 23 selects a predetermined number of speakers 12 from among all speakers 12 in accordance with the total number of selected meshes, which is “5”, and sets the five meshes on the surface of the unit sphere formed by the selected speakers 12 as the meshes to be used when calculating the VBAP gain.

[0404] After the mesh used in calculating the VBAP gain is determined, the processes of steps S244 and S245 are performed, and the process proceeds to step S246. Note that the processes of steps S244 and S245 are similar to the processes of steps S234 and S235, and therefore the description thereof will be omitted.

[0405] When the processing of step S235, step S237, step S242, or step S245 is performed and the VBAP gain of each speaker 12 is obtained, the processing of steps S246 to S248 is then performed and the reproduction processing ends.

[0406] The processes in steps S246 to S248 are similar to those in steps S17 to S19 described with reference to FIG. 7, and therefore will not be described further.

[0407] More specifically, the playback process is performed for each object substantially simultaneously, and in step S248, the audio signals of the speakers 12 obtained for each object are supplied to those speakers 12. That is, the speakers 12 play sounds based on a signal obtained by adding up the audio signals of the objects. As a result, the sounds of all the objects are output simultaneously.

[0408] In this way, the audio processing device 61 selectively performs quantization processing and mesh number switching processing for each object as appropriate. In this way, it is possible to reduce the amount of processing required for rendering while suppressing deterioration of the sense of realism and sound quality.

[0409] <Modification 1 of the second embodiment> <Configuration example of voice processing device> In addition, in the second embodiment, an example has been described in which quantization processing and mesh number switching processing are selectively performed when processing to widen a sound image is not performed, but quantization processing and mesh number switching processing may also be selectively performed when processing to widen a sound image is performed.

[0410] In such a case, the voice processing device 11 is configured, for example, as shown in Fig. 19. In Fig. 19, the parts corresponding to those in Fig. 6 or 17 are given the same reference numerals, and the description thereof will be omitted as appropriate.

[0411] The voice processing device 11 shown in FIG. 19 includes an acquisition unit 21, a vector calculation unit 22, a gain calculation unit 23, and a gain adjustment unit 71.

[0412] The acquisition unit 21 acquires audio signals and metadata of one or more objects, and supplies the acquired audio signals to the gain calculation unit 23 and the gain adjustment unit 71, and also supplies the acquired metadata to the vector calculation unit 22 and the gain calculation unit 23. In addition, the gain calculation unit 23 includes a quantization unit 31.

[0413] <Explanation of Regeneration Treatment> Next, the reproduction process performed by the audio processing device 11 shown in FIG. 19 will be described with reference to the flowchart in FIG.

[0414] In this example, it is assumed that the acquisition unit 21 is supplied with the audio signal and metadata of one or more objects for each frame, and the playback process is performed for each frame of the audio signal for each object.

[0415] 7, the processing in steps S271 and S272 is similar to the processing in steps S11 and S12 in Fig. 7, and therefore the description thereof will be omitted. However, in step S271, the audio signal acquired by the acquisition unit 21 is supplied to the gain calculation unit 23 and the gain adjustment unit 71, and the metadata acquired by the acquisition unit 21 is supplied to the vector calculation unit 22 and the gain calculation unit 23.

[0416] When the processes in steps S271 and S272 are performed, the spread vector, or the spread vector and the vector p, are obtained.

[0417] In step S273, the gain calculation unit 23 performs a VBAP gain calculation process to calculate a VBAP gain for each speaker 12. Note that, although details of the VBAP gain calculation process will be described later, in the VBAP gain calculation process, a quantization process and a mesh number switching process are selectively performed as appropriate, and a VBAP gain for each speaker 12 is calculated.

[0418] After the process of step S273 is performed to obtain the VBAP gain of each speaker 12, the processes of steps S274 to S276 are performed and the playback process ends, but since these processes are similar to the processes of steps S17 to S19 in Fig. 7, their description will be omitted. However, in more detail, the playback process is performed for each object approximately simultaneously, and in step S276, the audio signals of each speaker 12 obtained for each object are supplied to those speakers 12. Therefore, the speakers 12 output the sounds of all objects simultaneously.

[0419] In this way, the audio processing device 11 selectively performs quantization processing and mesh number switching processing for each object as appropriate. In this way, even when performing processing to widen the sound image, it is possible to reduce the amount of processing required for rendering while suppressing deterioration of the sense of realism and sound quality.

[0420] <Explanation of VBAP Gain Calculation Process> Next, the VBAP gain calculation process corresponding to the process in step S273 in FIG. 20 will be described with reference to the flowchart in FIG.

[0421] The processing in steps S301 to S303 is similar to the processing in steps S232 to S234 in Fig. 18, and therefore description thereof will be omitted. However, in step S303, a VBAP gain is calculated for each speaker 12 for the spread vector, or for each of the spread vector and vector p.

[0422] In step S304, the gain calculation unit 23 calculates a VBAP gain addition value by adding the VBAP gains calculated for each vector for each speaker 12. In step S304, the same process as in step S14 in FIG.

[0423] In step S305, the quantization unit 31 binarizes the VBAP gain addition value obtained for each speaker 12 in the process of step S304, and the VBAP gain calculation process ends. After that, the process proceeds to step S274 in FIG.

[0424] If it is determined in step S301 that the number of objects is less than 10, the processes in steps S306 and S307 are performed.

[0425] The processing in steps S306 and S307 is similar to that in steps S236 and S237 in Fig. 18, and therefore description thereof will be omitted. However, in step S307, a VBAP gain is calculated for each speaker 12 for the spread vector, or for each of the spread vector and vector p.

[0426] Also, after the processing of step S307 is performed, the processing of step S308 is performed and the VBAP gain calculation processing is terminated. Thereafter, the processing proceeds to step S274 in FIG. 20. However, since the processing of step S308 is similar to the processing of step S304, its description will be omitted.

[0427] Furthermore, if it is determined in step S306 that the importance information is not the highest value, then the processes of steps S309 to S312 are performed, but since these processes are similar to the processes of steps S238 to S241 in Fig. 18, their description will be omitted. However, in step S312, a VBAP gain is calculated for each speaker 12 for the spread vector, or for each vector of the spread vector and vector p.

[0428] In this way, when the VBAP gain for each speaker 12 is obtained for each vector, the process of step S313 is performed to calculate the VBAP gain addition value. However, since the process of step S313 is similar to the process of step S304, a description thereof will be omitted.

[0429] In step S314, the quantization unit 31 converts the VBAP gain addition value obtained for each speaker 12 in the process of step S313 into a ternary value, and the VBAP gain calculation process ends. After that, the process proceeds to step S274 in FIG.

[0430] Furthermore, if it is determined in step S310 that the sound pressure RMS is less than −30 dB, the process of step S315 is performed and the total number of meshes used in calculating the VBAP gain is set to 5. Note that the process of step S315 is similar to the process of step S243 in FIG. 18, and therefore a description thereof will be omitted.

[0431] When the mesh to be used in calculating the VBAP gain is determined, the processes of steps S316 to S318 are performed, the VBAP gain calculation process ends, and then the process proceeds to step S274 in Fig. 20. Note that the processes of steps S316 to S318 are similar to the processes of steps S303 to S305, and therefore a description thereof will be omitted.

[0432] In this way, the audio processing device 11 selectively performs quantization processing and mesh number switching processing for each object as appropriate. In this way, even when performing processing to widen the sound image, it is possible to reduce the amount of processing required for rendering while suppressing deterioration of the sense of realism and sound quality.

[0433] The above-mentioned series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed in a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, for example, capable of executing various functions by installing various programs.

[0434] FIG. 22 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes by a program.

[0435] In the computer, a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, and a RAM (Random Access Memory) 503 are interconnected via a bus 504.

[0436] An input / output interface 505 is further connected to the bus 504. An input unit 506, an output unit 507, a recording unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.

[0437] The input unit 506 includes a keyboard, a mouse, a microphone, an image sensor, etc. The output unit 507 includes a display, a speaker, etc. The recording unit 508 includes a hard disk, a non-volatile memory, etc. The communication unit 509 includes a network interface, etc. The drive 510 drives a removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0438] In a computer configured as described above, the CPU 501 loads, for example, a program recorded in the recording unit 508 into the RAM 503 via the input / output interface 505 and the bus 504, and executes the program, thereby performing the series of processes described above.

[0439] The program executed by the computer (CPU 501) can be provided by being recorded on a removable recording medium 511 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0440] In the computer, the program can be installed in the recording unit 508 via the input / output interface 505 by mounting the removable recording medium 511 in the drive 510. The program can also be received by the communication unit 509 via a wired or wireless transmission medium and installed in the recording unit 508. Alternatively, the program can be installed in the ROM 502 or the recording unit 508 in advance.

[0441] In addition, the program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or it may be a program in which processing is performed in parallel or at the required timing, such as when called.

[0442] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0443] For example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0444] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.

[0445] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices.

[0446] Furthermore, the present technology can also be configured as follows.

[0447] (1) an acquisition unit that acquires metadata including position information indicating a position of an audio object and sound image information that is composed of at least two-dimensional vectors and indicates a spread of a sound image from the position; a vector calculation unit that calculates a spread vector indicating a position within a region that represents the spread of a sound image determined by the sound image information, based on a horizontal angle and a vertical angle of the region; a gain calculation unit that calculates, based on the spread vector, gains of audio signals to be supplied to two or more audio output units located near the position indicated by the position information; An audio processing device comprising: (2) The vector calculation unit calculates the spread vector based on a ratio between the horizontal angle and the vertical angle. An audio processing device as described in (1). (3) The vector calculation unit calculates a predetermined number of the spread vectors. An audio processing device according to (1) or (2). (4) The vector calculation unit calculates a variable arbitrary number of the spread vectors. An audio processing device according to (1) or (2). (5) The sound image information is a vector indicating the center position of the region. An audio processing device as described in (1). (6) The sound image information is a vector of two or more dimensions that indicates the degree of spread of the sound image from the center of the region. An audio processing device as described in (1). (7) The sound image information is a vector indicating the relative position of the center position of the area as viewed from the position indicated by the position information. An audio processing device as described in (1). (8) The gain calculation unit Calculating the gain for each of the spread vectors for each of the audio output units; calculating an added value of the gains calculated for each of the spread vectors for each of the audio output units; quantizing the added value into two or more gain values ​​for each of the audio output units; Calculating the final gain for each of the audio output units based on the quantized sum. A voice processing device according to any one of (1) to (7). (9) The gain calculation unit selects the number of meshes to be used in the calculation of the gain, the meshes being an area surrounded by the three audio output units, and calculates the gain for each of the spread vectors based on the selection result of the number of meshes and the spread vector. The audio processing device according to (8). (10) The gain calculation unit selects the number of the meshes used in the calculation of the gain, whether or not to perform the quantization, and the quantization number of the added value at the time of the quantization, and calculates the final gain according to the selection result. The audio processing device according to (9). (11) The gain calculation unit selects the number of meshes used in the calculation of the gain, whether or not to perform the quantization, and the quantization number based on the number of the audio objects. The audio processing device according to (10). (12) The gain calculation unit selects the number of meshes used in the calculation of the gain, whether or not to perform the quantization, and the quantization number based on the importance of the audio object. The audio processing device according to (10) or (11). (13) The gain calculation unit selects the number of meshes used for the calculation of the gain such that the number of meshes used for the calculation of the gain increases for the audio object located closer to the audio object having a higher importance. The audio processing device according to (12). (14) The gain calculation unit selects the number of meshes used in the calculation of the gain, whether or not to perform the quantization, and the quantization number based on a sound pressure of the audio signal of the audio object. A voice processing device according to any one of (10) to (13). (15) The gain calculation unit selects, according to a selection result of the number of meshes, three or more of the audio output units including the audio output units located at different heights from each other among the plurality of audio output units, and calculates the gain based on one or more of the meshes formed from the selected audio output units. A voice processing device according to any one of (9) to (14). (16) Acquire metadata including position information indicating a position of an audio object and sound image information, which is composed of at least two-dimensional vectors and indicates a sound image spread from the position; calculating a spread vector indicating a position within a region representing a spread of a sound image determined by the sound image information, based on a horizontal angle and a vertical angle of the region; Calculating gains of audio signals to be supplied to two or more audio output units located near the position indicated by the position information, based on the spread vector. 1. An audio processing method comprising the steps of: (17) Acquire metadata including position information indicating a position of an audio object and sound image information, which is composed of at least two-dimensional vectors and indicates a sound image spread from the position; calculating a spread vector indicating a position within a region representing a spread of a sound image determined by the sound image information, based on a horizontal angle and a vertical angle of the region; Calculating gains of audio signals to be supplied to two or more audio output units located near the position indicated by the position information, based on the spread vector. A program that causes a computer to execute a process including steps. (18) an acquisition unit for acquiring metadata including position information indicating a position of an audio object; a gain calculation unit that selects a number of meshes, which are regions surrounded by three audio output units, to be used in calculating a gain of an audio signal to be supplied to the audio output units, and calculates the gain based on a selection result of the number of meshes and the position information; An audio processing device comprising: [Explanation of symbols]

[0448] 11 audio processing device, 21 acquisition unit, 22 vector calculation unit, 23 gain calculation unit, 24 gain adjustment unit, 31 quantization unit, 61 audio processing device, 71 gain adjustment unit

Claims

1. an acquisition unit that acquires metadata including position information indicating a position of an audio object and sound image information that is composed of at least two-dimensional vectors and indicates a sound image spread from the position; a vector calculation unit that calculates a spread vector indicating a position within a region that represents the spread of a sound image determined by the sound image information, based on a horizontal angle and a vertical angle of the region; a gain calculation unit that calculates, based on the spread vector, gains of each of the audio signals to be supplied to two or more audio output units located in the vicinity of the position indicated by the position information; An audio processing device comprising:

2. The vector calculation unit calculates the spread vector based on a ratio between the horizontal angle and the vertical angle. The audio processing device according to claim 1 .

3. The vector calculation unit calculates a predetermined number of the spread vectors. The audio processing device according to claim 1 .

4. The vector calculation unit calculates a variable arbitrary number of the spread vectors. The audio processing device according to claim 1 .

5. The sound image information is a vector indicating the center position of the region. The audio processing device according to claim 1 .

6. The sound image information is a vector of two or more dimensions that indicates the degree of spread of the sound image from the center of the region. The audio processing device according to claim 1 .

7. The sound image information is a vector indicating the relative position of the center position of the area as viewed from the position indicated by the position information. The audio processing device according to claim 1 .

8. The gain calculation unit Calculating the gain for each of the spread vectors for each of the audio output units; calculating an added value of the gains calculated for each of the spread vectors for each of the audio output units; quantizing the sum into two or more gain values ​​for each of the audio output units; Calculating the final gain for each of the audio output units based on the quantized sum. The audio processing device according to claim 1 .

9. The gain calculation unit selects the number of meshes to be used in the calculation of the gain, which are meshes that are regions surrounded by the three audio output units, and calculates the gain for each of the spread vectors based on the selection result of the number of meshes and the spread vector. The audio processing device according to claim 8.

10. The gain calculation unit selects the number of the meshes used in the calculation of the gain, whether or not to perform the quantization, and the quantization number of the added value at the time of the quantization, and calculates the final gain according to the selection result. The audio processing device according to claim 9.

11. The gain calculation unit selects the number of meshes used in the calculation of the gain, whether or not to perform the quantization, and the quantization number based on the number of the audio objects. The audio processing device according to claim 10.

12. The gain calculation unit selects the number of meshes used in the calculation of the gain, whether or not to perform the quantization, and the quantization number based on the importance of the audio object. The audio processing device according to claim 10.

13. The gain calculation unit selects the number of meshes used for the calculation of the gain such that the number of meshes used for the calculation of the gain increases for the audio object located closer to the audio object having a higher importance. The audio processing device according to claim 12.

14. The gain calculation unit selects the number of meshes used in the calculation of the gain, whether or not to perform the quantization, and the quantization number based on a sound pressure of the audio signal of the audio object. The audio processing device according to claim 10.

15. The gain calculation unit selects, according to a result of the selection of the number of meshes, three or more of the audio output units including the audio output units located at different heights from each other among the plurality of the audio output units, and calculates the gain based on one or more of the meshes formed from the selected audio output units. The audio processing device according to claim 9.

16. acquiring metadata including position information indicating a position of an audio object and sound image information, which is made up of a vector of at least two dimensions, indicating a spread of a sound image from the position; calculating a spread vector indicating a position within a region representing a spread of a sound image determined by the sound image information, based on a horizontal angle and a vertical angle of the region; Calculating gains of audio signals to be supplied to two or more audio output units located near the position indicated by the position information, based on the spread vector.

1. An audio processing method comprising the steps of:

17. acquiring metadata including position information indicating a position of an audio object and sound image information, which is made up of a vector of at least two dimensions, indicating a spread of a sound image from the position; calculating a spread vector indicating a position within a region representing a spread of a sound image determined by the sound image information, based on a horizontal angle and a vertical angle of the region; Calculating gains of audio signals to be supplied to two or more audio output units located near the position indicated by the position information, based on the spread vector. A program that causes a computer to execute a process including steps.

Images