Information processing device and method, reproduction device and method, and program

Abstract

The present technology relates to an information processing device and method, a reproduction device and method, and a program, which are configured to be able to perform gain correction more easily. The information processing device includes: a gain correction value determination unit that determines a correction value of a gain value of an audio signal of an audio object for gain correction according to the direction of the audio object viewed by a viewer. The present technology can be applied to a gain determination device and a reproduction device.

Description

Technical Field

[0001] The present technology relates to an information processing device and method, a reproducing device and method, and a program, and in particular, to an information processing device and method, a reproducing device and method, and a program that can more easily perform gain correction. Background Art

[0002] Conventionally, Moving Picture Experts Group (MPEG)-H 3D audio standard is known (for example, see Non-Patent Document 1 and Non-Patent Document 2).

[0003] 3D audio processed by the MPEG-H 3D audio standard or the like can reproduce three-dimensional sound direction, distance, extension, etc., and can perform audio reproduction with a higher sense of reality than conventional stereo reproduction.

[0004] Prior art literature

[0005] Non-patent literature

[0006] Non-patent document 1: ISO / IEC 23008-3, MPEG-H 3D Audio

[0007] Non-patent document 2: ISO / IEC 23008-3:2015 / AMENDMENT3, MPEG-H 3D Audio Phase 2 Summary of the invention

[0008] Technical issues

[0009] However, with 3D audio, the production time cost of content (3D audio content) increases.

[0010] For example, in 3D audio, the dimensionality of the position information of an object (i.e., the position information of a sound source) is higher than that of stereo (3D audio is three-dimensional, while stereo is two-dimensional). Therefore, with 3D audio, in particular, the time cost increases in the work of determining parameters constituting metadata of each object (such as the horizontal and vertical angles indicating the position of the object, the distance, and the gain of the object).

[0011] In addition, compared with stereo content, 3D audio content is much less in terms of content and creators. Therefore, the amount of high-quality 3D audio content is currently small.

[0012] On the other hand, as an auditory characteristic, the perception of the loudness of a sound depends on the direction of arrival of the sound. That is, even if it is the sound of the same object, the loudness of the sound in auditory perception changes when the object is in front of the listener and when the object is on the side of the listener, and when the object is above the listener and when the object is below the listener. Therefore, gain correction needs to be performed according to such auditory characteristics.

[0013] Thus, it is desirable to perform gain correction more easily and, therefore, be able to generate 3D audio content of sufficient quality in a short time.

[0014] The present technology has been proposed in consideration of such a situation, and can perform gain correction more easily.

[0015] Solution to the problem

[0016] An information processing device of a first aspect of the present technology includes a gain correction value decision unit that decides a correction value of a gain value for performing gain correction on an audio signal of an audio object, according to a direction from which the audio object is viewed from a listener.

[0017] An information processing method or program of a first aspect of the present technology includes a step of deciding a correction value of a gain value for performing gain correction on an audio signal of an audio object, according to a direction from which the audio object is viewed by a listener.

[0018] In the first aspect of the present technology, a correction value of a gain value for performing gain correction on an audio signal of an audio object is decided according to a direction from which the audio object is viewed by a listener.

[0019] A reproduction device of a second aspect of the present technology includes: a gain correction unit, which determines a correction value of a gain value for performing gain correction on an audio signal of an audio object based on position information indicating the position of the audio object, the correction value depending on the direction from which the audio object is viewed from a listener, and performs gain correction on the audio signal based on the gain value corrected by the correction value; and a renderer processing unit, which performs rendering processing based on the audio signal obtained by the gain correction, and generates a reproduction signal of multiple channels for reproducing the sound of the audio object.

[0020] The reproduction method or program of the second aspect of the present technology includes: a step of determining a correction value of a gain value for performing gain correction on an audio signal of an audio object based on position information indicating the position of an audio object, the correction value depending on the direction from which the audio object is viewed from a listener; a step of performing gain correction on the audio signal based on the gain value corrected by the correction value; a step of performing rendering processing based on the audio signal obtained by the gain correction; and a step of generating a reproduction signal of multiple channels for reproducing the sound of the audio object.

[0021] In a second aspect of the present technology, based on position information indicating the position of an audio object, a correction value of a gain value for performing gain correction on an audio signal of the audio object is determined, the correction value depending on the direction from which the audio object is viewed from a listener; gain correction is performed on the audio signal based on the gain value corrected by the correction value; rendering processing is performed based on the audio signal obtained by the gain correction, and a reproduction signal of multiple channels for reproducing the sound of the audio object is generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a diagram illustrating the auditory characteristics regarding the arrival direction of sound.

[0023] Figure 2 This is a diagram illustrating the auditory characteristics regarding the arrival direction of sound.

[0024] Figure 3 This is a diagram illustrating the auditory characteristics regarding the arrival direction of sound.

[0025] Figure 4 is a diagram showing a configuration example of an information processing apparatus.

[0026] Figure 5 is a view showing an example of an auditory characteristic table.

[0027] Figure 6 is a view showing an example of an auditory characteristic table.

[0028] Figure 7 : is a flowchart explaining the gain value determination process.

[0029] Figure 8 is a view showing an example of a display screen of a content creation tool.

[0030] Fig. 9 is a view showing an example of a display screen of a content creation tool.

[0031] Fig.10 is a view showing an example of a display screen of a content creation tool.

[0032] Fig.11 is a view showing an example of a display screen of a content creation tool.

[0033] Fig.12 is a diagram showing a configuration example of an information processing apparatus.

[0034] Fig.13 This is a flowchart explaining the table generation process.

[0035] Fig.14 is a diagram showing a configuration example of a speech processing device.

[0036] Fig.15 : is a flowchart illustrating the reproduction signal generation process.

[0037] Fig.16 is a view showing an example of an auditory characteristic table.

[0038] Fig.17 is a view showing a syntax example of gain auditory characteristic information.

[0039] Fig.18 is a diagram showing a configuration example of a speech processing device.

[0040] Fig.19 is a diagram showing a configuration example of a computer. DETAILED DESCRIPTION

[0041] Embodiments to which the present technology is applied will be described below with reference to the drawings.

[0042] <First Embodiment>

[0043] <About this technology>

[0044] The present technology enables gain correction to be performed more easily by determining a gain correction value according to the direction of an object viewed from a listener, and thus enables creation of sufficiently high-quality 3D audio content more easily, i.e., in a short time.

[0045] In particular, the present technology has the following features (F1) to (F5).

[0046] Feature (F1): The gain correction value of the object is determined based on the three-dimensional auditory characteristics regarding the localization position of the sound image.

[0047] Feature (F2): When auditory characteristics are given by a table or the like, a gain correction value for a localization position for which there is no data is calculated by interpolation processing or the like based on gain correction values ​​at adjacent positions.

[0048] Feature (F3): In automatic mixing, gain information is determined from position information determined separately.

[0049] Features (F4): Provides a user interface for setting and adjusting gain correction values ​​with respect to object positions.

[0050] Feature (F5): A gain correction value corresponding to the three-dimensional auditory characteristic is applied in association with a change in the position of the object relative to the listening position.

[0051] First, determination of gain parameters based on three-dimensional auditory characteristics of humans will be explained.

[0052] Figure 1The gain correction amount when performing gain correction of pink noise is shown so that when the same pink noise is reproduced from different directions, the listener feels that the loudness of the sound is the same auditorily, with reference to the loudness of the sound auditorily when a certain pink noise is reproduced right in front of the listener. In other words, Figure 1 The figure shows the auditory characteristics of humans in the horizontal direction.

[0053] Note that Figure 1 , the vertical axis represents the gain correction amount, and the horizontal axis represents the azimuth value (horizontal angle), which is an angle in the horizontal direction indicating the position of the sound source viewed from the listener.

[0054] For example, the azimuth value indicating the front direction viewed from the listener is 0 degrees, the azimuth value indicating the side, i.e., the side direction viewed from the listener is ±90 degrees, and the azimuth value indicating the rear, i.e., the back direction is 180 degrees. In particular, the left direction viewed from the listener is the positive direction of the azimuth value.

[0055] In addition, Figure 1 In the example, the vertical position of the sound source when the pink noise is reproduced is the same as the height of the listener. That is, the vertical angle representing the position of the sound source in the vertical direction (elevation direction) as viewed from the listener is set as the elevation value, Figure 1 is an example of a case where the elevation angle value is 0 degrees. Note that the upward direction viewed from the listener is the positive direction of the elevation angle value.

[0056] This example shows the average value of the gain correction amount for each azimuth angle value obtained from the results of experiments conducted on a plurality of listeners, and in particular, the range indicated by the dotted line in each azimuth angle value indicates a 95% confidence interval.

[0057] For example, when pink noise is reproduced in the lateral direction (azimuth value = ±90 degrees, elevation value = 0 degrees), it is known that by slightly reducing the gain, the listener perceives the sound he hears as loud as when the pink noise is reproduced in the front direction.

[0058] Furthermore, for example, when pink noise is reproduced in the rear (azimuth value = 180 degrees, elevation value = 0 degrees), it is known that by slightly increasing the gain, the listener feels that the sound he hears is the same loudness as when the pink noise is reproduced in the front direction.

[0059] That is, for a certain object sound source, when the localization position of the object sound source is at the side of the listener, by slightly reducing the sound gain of the object sound source, and when the localization position of the object sound source is behind the listener, by slightly increasing the sound gain of the object sound source, the listener can feel that he hears sounds with the same loudness.

[0060] In addition, for example, Figure 2 and Figure 3 As shown, even if the azimuth value is the same, if the elevation value changes, what the listener hears also changes.

[0061] Note that Figure 2 and Figure 3 , the vertical axis represents the gain correction amount, and the horizontal axis represents the azimuth value (horizontal angle) indicating the position of the sound source viewed from the listener. Figure 2 and Figure 3 The range represented by the dashed line in each azimuth value indicates a 95% confidence interval.

[0062] Figure 2 The gain correction amount at each azimuth angle value is shown for an elevation angle value of 30 degrees.

[0063] Figure 2 It is shown that, in the case where the sound source is located at a position higher than the listener, the sound is silent when the sound source is located in front of, behind, or diagonally behind the listener, and the sound is slightly louder when the sound source is located diagonally in front of the listener.

[0064] Similarly, Figure 3 The amount of gain correction at each azimuth angle value is shown for an elevation angle value of -30 degrees.

[0065] Figure 3 It means that, in the case where the sound source exists at a position lower than the listener, the sound is heard loudly when the sound source exists in front of or diagonally in front of the listener, and the sound is silent when the sound source exists behind or diagonally behind the listener.

[0066] It is known that appropriate gain correction can be performed more easily if the gain correction amount for the target sound source is determined based on position information indicating the position of the target sound source and the auditory characteristics of the listener with respect to the sound arrival direction as described above.

[0067] <Configuration Example of Information Processing Device>

[0068] Figure 4 is a diagram showing a configuration example of an embodiment of an information processing device to which the present technology is applied.

[0069] Figure 4 The illustrated information processing device 11 functions as a gain determination device that determines a gain value for gain correction of an audio signal for reproducing the sound of an audio object (hereinafter, simply referred to as an object) constituting 3D audio content.

[0070] Such an information processing device 11 is provided in, for example, an editing device or the like that performs mixing of audio signals constituting 3D audio content.

[0071] The information processing device 11 includes a gain correction value determination unit 21 and an auditory characteristics table storage unit 22 .

[0072] The position information and the gain initial value are supplied to the gain correction value decision unit 21 as metadata of an object constituting the 3D audio content.

[0073] Here, the position information of the object is information indicating the position of the object viewed from a reference position in a three-dimensional space, and here, the position information includes an azimuth value, an elevation value, and a radius value. Note that in this example, the position of the listener is the reference position.

[0074] The azimuth value and the elevation value are angles representing each position in the horizontal direction and the vertical direction of the object viewed from the listener (user) existing at the reference position, and the azimuth value and the elevation value are similar to Figures 1 to 3 Those in the case of.

[0075] Furthermore, the radius value is the distance (radius) from the listener at a reference position existing in the three-dimensional space to the object.

[0076] It can be said that the position information including the azimuth value, the elevation value, and the radius value indicates the localized position of the sound image of the sound of the object.

[0077] In addition, the gain initial value included in the metadata provided to the gain correction value decision unit 21 is a gain value for gain correction of the audio signal of the object, that is, an initial value of gain information, and is determined by, for example, a creator of 3D audio content, etc. Note that, for simplicity of explanation, the gain initial value is assumed to be 1.0.

[0078] The gain correction value decision unit 21 decides a gain correction value indicating a gain correction amount for correcting an initial value of a gain of a target based on the position information as the supplied metadata and the auditory characteristic table retained in the auditory characteristic table retention unit 22 .

[0079] Furthermore, the gain correction value decision unit 21 corrects the supplied gain initial value based on the decided gain correction value, and sets the resulting gain value as information indicating a final gain correction amount for performing gain correction on the audio signal of the object.

[0080] In other words, the gain correction value decision unit 21 determines the gain correction value according to the direction of the object viewed from the listener (the direction of sound arrival) indicated by the position information, thereby determining the gain value of the audio signal. The gain value thus determined and the provided position information are output to the subsequent stage as the final metadata of the object.

[0081] The auditory characteristic table holding unit 22 holds the auditory characteristic table, and supplies the gain correction value indicated by the auditory characteristic table to the gain correction value decision unit 21 as necessary.

[0082] Here, the auditory characteristic table is a table in which the arrival direction of sound from an object as a sound source to a listener, that is, the direction of the sound source viewed from the listener is associated with a gain correction value according to the direction.

[0083] That is, more specifically, the auditory sense characteristic table is a table in which the relative positional relationship between the sound source and the listener is associated with the gain correction value according to the positional relationship.

[0084] For example, Figures 1 to 3 As shown, the gain correction value indicated by the auditory characteristic table is determined according to the auditory characteristic of a person regarding the direction from which the sound comes, and is a gain correction amount so that the auditory loudness of the sound becomes constant regardless of the direction from which the sound comes.

[0085] That is, if gain correction is performed on the audio signal of the object using a gain value obtained by correcting the gain initial value to a gain correction value indicated by the auditory characteristic table, the sound of the same object is heard at the same loudness regardless of the position of the object.

[0086] here, Figure 5 An example of an auditory characteristic table is shown.

[0087] exist Figure 5 In the example shown, the gain correction value is associated with the position of the object (ie, the direction of the object) determined by an azimuth value, an elevation value, and a radius value.

[0088] In particular, this example assumes that all elevation and radius values ​​are 0 and 1.0, that objects are located at the same vertical height as the listener, and that the distance from the listener to the object is always constant.

[0089] exist Figure 5 In the example, for example, in a case where an object serving as a sound source exists behind the listener, such as in a case where the azimuth angle value is 180 degrees, the gain correction value becomes greater than the gain correction value in a case where the object exists in front of the listener, such as in a case where the azimuth angle value is 0 degrees or 30 degrees.

[0090] On the other hand, for example, in the case where an object as a sound source exists on the side of the listener, such as in the case where the azimuth angle value is 90 degrees, the gain correction value becomes smaller than the gain correction value in the case where the object exists in front of the listener.

[0091] In addition, the auditory characteristic table holding unit 22 holds Figure 5This is a specific example of correction of the initial gain value by the gain correction value determination unit 21 in the case of the auditory characteristics table shown in FIG.

[0092] For example, assuming that the azimuth value, elevation value, and radius value representing the position of the object are 90 degrees, 0 degrees, and 1.0 m, Figure 5 The gain correction value corresponding to the object position is -0.52dB.

[0093] Therefore, the gain correction value decision unit 21 performs calculation of the following expression (1) based on the gain correction value "-0.52 dB" read from the auditory characteristic table and the gain initial value "1.0", and gives a gain value of "0.94".

[0094] [Expression 1]

[0095]

[0096] Similarly, for example, assuming that the azimuth value, elevation value, and radius value indicating the position of the object are -150 degrees, 0 degrees, and 1.0 m, then Figure 5 The gain correction value corresponding to the object position is 0.51dB.

[0097] Therefore, the gain correction value decision unit 21 performs calculation of the following Expression (2) based on the gain correction value “0.51 dB” read from the auditory characteristic table and the gain initial value “1.0”, and gives a gain value “1.06”.

[0098] [Expression 2]

[0099]

[0100] Note that Figure 5 In the above, an example of using a gain correction value determined based on a two-dimensional auditory characteristic that considers only the horizontal direction among two-dimensional auditory characteristics has been described. That is, an example of using an auditory characteristic table generated based on two-dimensional auditory characteristics (hereinafter, also referred to as a two-dimensional auditory characteristic table) has been described.

[0101] However, the gain initial value may be corrected using a gain correction value determined based on three-dimensional auditory characteristics in which not only the characteristics in the horizontal direction but also the characteristics in the vertical direction are considered.

[0102] In this case, for example, one can use Figure 6 The auditory characteristics table shown.

[0103] exist Figure 6 In the example shown, the gain correction value is associated with the position of the object (ie, the direction of the object) determined by an azimuth value, an elevation value, and a radius value.

[0104] In particular, in this example, the radius value is 1.0 in all combinations of azimuth and elevation values.

[0105] In the following, according to Figure 6 The auditory characteristic table generated with respect to the three-dimensional auditory characteristic of the sound arrival direction is also specifically referred to as a three-dimensional auditory characteristic table.

[0106] Here, the description will be given of the Figure 6 This is a specific example of correction of the initial gain value by the gain correction value determination unit 21 in the case of the auditory characteristics table shown in FIG.

[0107] For example, assuming that the azimuth value, elevation value, and radius value representing the position of the object are 60 degrees, 30 degrees, and 1.0 m, then Figure 6 The gain correction value corresponding to the object position is -0.07dB.

[0108] Therefore, the gain correction value decision unit 21 performs calculation of the following Expression (3) based on the gain correction value "-0.07 dB" read from the auditory characteristic table and the gain initial value "1.0", and gives a gain value of "0.99".

[0109] [Expression 3]

[0110]

[0111] Note that in the above-described specific example of gain value calculation, gain correction values ​​based on auditory characteristics determined about the position (direction) of the object are prepared in advance. That is, an example is described in which gain correction values ​​corresponding to the position information of the object are stored in the auditory characteristics table.

[0112] However, the position of the object does not necessarily exist at a position where the corresponding gain correction value is stored in the auditory characteristic table.

[0113] Specifically, for example, assuming that the auditory characteristic table holding unit 22 holds Figure 6 The auditory characteristic table shown in FIG. 1 , and the azimuth angle value, elevation angle value, and radius value as the position information are -120 degrees, 15 degrees, and 1.0 m.

[0114] In this case, the gain correction value corresponding to the azimuth value "-120", the elevation value "15", and the radius value "1.0" is not stored in Figure 6 The auditory characteristics table.

[0115] Therefore, in a case where a gain correction value corresponding to a position indicated by the position information does not exist in the auditory characteristic table, the gain correction value determination unit 21 can use data (gain correction values) at multiple positions where corresponding gain correction values ​​adjacent to the position indicated by the position information exist, and calculate the gain correction value at the expected position through interpolation processing, etc.

[0116] In other words, in the case where the gain correction value corresponding to the direction (position) of the object viewed from the listener is not stored in the auditory characteristic table, the gain correction value can be obtained through interpolation processing, etc. based on the gain correction value corresponding to another direction of the object viewed from the listener.

[0117] For example, gain correction value interpolation methods include Vector Base Amplitude Translation (VBAP).

[0118] VBAP is used to obtain gain values ​​for multiple speakers in a reproduction environment from the metadata of the object for each object.

[0119] Here, the gain correction value at the desired position may be calculated by replacing a plurality of speakers in the reproduction environment with a plurality of gain correction values.

[0120] Specifically, meshes are divided at a plurality of positions in a three-dimensional space where gain correction values ​​are prepared. That is, for example, under the assumption that gain correction values ​​for each of three positions in a three-dimensional space are prepared, a triangular area having the three positions as vertices is a mesh.

[0121] When the three-dimensional space is divided into a plurality of meshes in this way, a mesh including a position of interest is specified with a desired position for obtaining a gain correction value as the position of interest.

[0122] Furthermore, when a position vector indicating a focus position is expressed by multiplication and addition of position vectors indicating three vertex positions constituting a specified mesh, a coefficient to be multiplied by the position vector indicating each of the three vertex positions is obtained.

[0123] Then, each of the three coefficients thus obtained is multiplied by the corresponding gain correction value of the three vertex positions of the mesh including the position of interest, and the sum of the gain correction values ​​multiplied by the coefficients is calculated as the gain correction value of the position of interest.

[0124] Specifically, it is assumed that position vectors representing the positions of three vertices of a mesh including a position of interest are P1 to P3, and gain correction values ​​of the vertex positions are G1 to G3.

[0125] At this time, it is assumed that the position vector indicating the position of interest is represented by g1P1+g2P2+g3P3. In this case, the gain correction value of the position of interest is g1G1+g2G2+g3G3.

[0126] Note that the interpolation method of the gain correction value is not limited to interpolation by VBAP, and any other method may be used.

[0127] For example, among the positions where gain correction values ​​exist in the auditory characteristic table, an average value of gain correction values ​​at N (eg, N=5) positions near the position of interest may be used as the gain correction value of the position of interest.

[0128] Furthermore, for example, the gain correction value at the position closest to the attention position among the positions where the gain correction value exists in the auditory sense characteristic table may be used as the gain correction value of the attention position.

[0129] In addition, although an example in which the gain correction value is obtained in a decibel value in the example has been described here, the gain correction value may be obtained in a linear value. In this case, for example, even when the gain correction value in a linear value is obtained by interpolation using VBAP, the gain correction value at an arbitrary position can be obtained by a calculation similar to the above-mentioned decibel value case.

[0130] Furthermore, the present technology can also be applied to a case where position information is used as metadata of an object, that is, the azimuth value, elevation value, and radius value are determined based on the type, priority, sound pressure, pitch, etc. of the object.

[0131] In this case, the gain correction value is determined based on, for example, position information determined based on the object type, priority, etc., and a three-dimensional auditory characteristic table prepared in advance.

[0132] <Gain value determination process description>

[0133] Subsequently, the operation of the information processing apparatus 11 will be described. Figure 7 The gain value determination process performed by the information processing device 11 is described with reference to a flowchart of FIG.

[0134] In step S11 , the gain correction value decision unit 21 acquires metadata from the outside.

[0135] That is, the gain correction value decision unit 21 acquires the position information including the azimuth value, the elevation value, and the radius value and the gain initial value as metadata.

[0136] In step S12 , the gain correction value determination unit 21 determines a gain correction value based on the position information acquired in step S11 and the auditory characteristic table held in the auditory characteristic table holding unit 22 .

[0137] That is, the gain correction value decision unit 21 reads the gain correction value associated with the azimuth value, elevation value, and radius value constituting the acquired position information from the auditory characteristics table, and sets the read gain correction value as the decided gain correction value.

[0138] In step S13 , the gain correction value determination unit 21 determines a gain value based on the gain initial value acquired in step S11 and the gain correction value determined in step S12 .

[0139] That is, the gain correction value decision unit 21 obtains the gain value by performing calculation similar to Expression (1) based on the gain initial value and the gain correction value and correcting the gain initial value with the gain correction value.

[0140] When the gain value is thus decided, the gain correction value decision unit 21 outputs the decided gain value to the subsequent stage, and the gain value decision process ends. The gain value that has been output is used to perform gain correction (gain adjustment) on the audio signal in the subsequent stage.

[0141] As described above, the information processing device 11 determines the gain correction value using the auditory characteristic table, and determines the gain value by correcting the initial gain value with the gain correction value.

[0142] This makes it easier to perform gain correction. Thus, for example, 3D audio content of sufficiently high quality can be created more easily, i.e. in a short time.

[0143] <Second Embodiment>

[0144] <About the User Interface>

[0145] Furthermore, according to the present technology, a user interface for setting and adjusting the above-mentioned gain correction value can be provided.

[0146] For example, the present technology can be applied to a 3D audio content creation tool that determines the position of an object, etc., by user input or automatically.

[0147] Specifically, in a 3D audio content creation tool, for example, by Figure 8 The user interface (display screen) shown can perform setting or adjustment of a gain correction value (gain value) based on the auditory characteristics with respect to the direction of an object viewed from the listener.

[0148] exist Figure 8 In the example shown, the display screen of the 3D audio content creation tool is provided with a pull-down box BX11 for selecting a desired auditory characteristic from a plurality of preset auditory characteristics that are different from each other.

[0149] In this example, a plurality of two-dimensional auditory characteristics, such as the auditory characteristics of men, the auditory characteristics of women, and the auditory characteristics of a single user, are prepared in advance, and the user can select a desired auditory characteristic by operating the pull-down box BX11.

[0150] When the user selects the hearing feature, Figure 8 The gain correction value display area R11 provided below the pull-down box BX11 in displays the gain correction value at each azimuth value corresponding to the auditory characteristic selected by the user.

[0151] Specifically, in the gain correction value display region R11 , the vertical axis represents the gain correction value, and the horizontal axis represents the azimuth angle value.

[0152] Furthermore, curve L11 represents the gain correction value at each azimuth value having a negative value, that is, the gain correction value at the right direction viewed from the listener, and curve L12 represents the gain correction value at each azimuth value at the left direction viewed from the listener.

[0153] By viewing such a gain correction value display area R11 , the user can intuitively and instantaneously grasp the gain correction value at each azimuth value.

[0154] Furthermore, a slider display region R12 for displaying a slider or the like for adjusting the gain correction value displayed in the gain correction value display region R11 is provided below the gain correction value display region R11 in the figure.

[0155] In the slider display region R12 , for each azimuth value at which the user can adjust the gain correction value, a number indicating the azimuth value, a scale indicating the gain correction value, and a slider for adjusting the gain correction value are displayed.

[0156] For example, the slider SD11 is used to adjust the gain correction value when the azimuth angle value is 30 degrees, and the user can designate a desired value as the adjusted gain correction value by moving the slider SD11 up and down.

[0157] When the gain correction value is adjusted using the slider SD11, the display of the gain correction value display area R11 is updated according to the adjustment. That is, here, the curve L12 changes according to the operation of the slider SD11.

[0158] Therefore, in Figure 8 In the example shown, the gain correction value in each direction on the right side viewed from the listener and the gain correction value in each direction on the left side viewed from the listener can be adjusted independently.

[0159] Specifically, in this example, by selecting any one of a plurality of auditory characteristics prepared in advance, the gain correction value can be specified according to the desired auditory characteristic, that is, the auditory characteristic table. Then, by operating the slider, the gain correction value can be further adjusted according to the selected auditory characteristic.

[0160] For example, since the auditory characteristics prepared in advance are average characteristics, by operating the slider, the user can adjust the gain correction value according to the auditory characteristics of a single user. In addition, by operating the slider to adjust the gain correction value, it is also possible to perform adjustments according to the user's intention, such as enhancing an object in the back by performing a large gain correction.

[0161] When the gain correction value in each azimuth angle value is thus set and adjusted, for example, by operating a save button not shown, etc., a two-dimensional auditory characteristic table is generated in which the gain correction value and each azimuth angle value are associated with each other and displayed on the gain correction value display area R11 in the two-dimensional auditory characteristic table.

[0162] Note that Figure 8 In the example, the gain correction value is different in each direction of the right side and the left side viewed from the listener, that is, the gain correction value is bilaterally asymmetric in this example. However, the gain correction value may be bilaterally symmetric.

[0163] In this case, for example, Fig. 9 Set and adjust the gain correction value as shown. Fig. 9 In, with Figure 8 The corresponding parts in the figure are given the same figure marks, and their descriptions are appropriately omitted.

[0164] Fig. 9 The display screen of the 3D audio content creation tool is shown, and in this example, a pull-down box BX11 , a gain correction value display area R21 , and a slider display area R22 are displayed on the display screen.

[0165] In the gain correction value display area R21, Figure 8 The gain correction value display area R11 in φ similarly displays the gain correction value at each azimuth value. However, here, since the gain correction values ​​in each direction on the left and right sides are common, only one curve indicating the gain correction value is displayed.

[0166] For example, the average value of the gain correction values ​​in each direction on the left and right sides may be a gain correction value common to the left and right sides. In this case, for example, Figure 8 The average of the gain correction values ​​for the example with 90 degree azimuth values ​​and -90 degree azimuth values ​​is set to Fig. 9 The example in FIG. 4 has a common gain correction value with an azimuth angle value of ±90 degrees.

[0167] Furthermore, a slider or the like for adjusting the gain correction value displayed in the gain correction value display region R21 is displayed in the slider display region R22 .

[0168] For example, in this example, by moving the slider SD21 up and down, the user can adjust the common gain correction value having an azimuth angle value of ±30 degrees.

[0169] In addition, for example, Fig.10 As shown, the gain correction value at each azimuth angle value can be adjusted for each elevation angle value. Fig.10 In, with Figure 8 The corresponding parts in the figure are given the same figure marks, and their descriptions are appropriately omitted.

[0170] Fig.10 The display screen of the 3D audio content creation tool is shown, and in this example, a pull-down box BX11 , gain correction value display areas R31 to R33 , and slider display areas R34 to R36 are displayed on the display screen.

[0171] exist Fig.10 In the example shown, the gain correction value is Fig. 9 The example shown is similarly bilaterally symmetrical.

[0172] The gain correction value at each azimuth angle value when the elevation angle value is 30 degrees is displayed in the gain correction value display area R31 , and the user can adjust the gain correction value by operating a slider or the like displayed in the slider display area R34 .

[0173] Similarly, the gain correction value at each azimuth angle value when the elevation angle value is 0 degree is displayed in the gain correction value display area R32 , and the user can adjust the gain correction value by operating a slider or the like displayed in the slider display area R35 .

[0174] Furthermore, the gain correction value at each azimuth angle value when the elevation angle value is -30 degrees is displayed in the gain correction value display area R33, and the user can adjust the gain correction value by operating a slider or the like displayed in the slider display area R36.

[0175] When the gain correction value in each azimuth value is thus set and adjusted, for example, by operating a save button not shown, a three-dimensional auditory characteristic table is generated in which the gain correction value, elevation value, and azimuth value are associated with each other.

[0176] In addition, if Fig.11As shown, as another example of the display screen of the 3D audio content creation tool, a gain correction value display area of ​​a radar chart type may be provided. Fig.11 In, with Fig.10 The corresponding parts in are given the same figure marks, and their descriptions are appropriately omitted.

[0177] exist Fig.11 In the example of FIG. 1 , a pull-down box BX11, gain correction value display areas R41 to R43, and slider display areas R34 to R36 are displayed on the display screen. In this example, the gain correction value is similar to Fig.10 The example shown is bilaterally symmetric.

[0178] The gain correction value display area R41 displays the gain correction values ​​at each azimuth angle when the elevation angle is 30 degrees, and the user can adjust the gain correction value by operating a slider displayed in the slider display area R34 or the like.

[0179] Specifically, in the gain correction value display area R41, since each item of the radar chart is an azimuth value, the user can not only instantly grasp each direction (azimuth value) and the gain correction value in these directions, but also grasp the relative difference in the gain correction value between the directions.

[0180] Similar to the gain correction value display area R41, the gain correction value at each azimuth value when the elevation angle value is 0 degrees is displayed in the gain correction value display area R42. Also, the gain correction value at each azimuth value when the elevation angle value is -30 degrees is displayed in the gain correction value display area R43.

[0181] <Configuration Example of Information Processing Device>

[0182] Next, the description will be made by referring to Figure 8 An information processing device that generates an auditory characteristic table using the 3D audio content creation tool described in the invention.

[0183] For example, such an information processing device is configured as follows Fig.12 shown.

[0184] Fig.12 The illustrated information processing device 51 realizes a content creation tool, and causes the display device 52 to display a display screen of the content creation tool.

[0185] The information processing device 51 has an input unit 61 , an auditory characteristic table generating unit 62 , an auditory characteristic table holding unit 63 , and a display control unit 64 .

[0186] The input unit 61 includes, for example, a mouse, a keyboard, a switch, a button, and a touch screen, and supplies an input signal corresponding to a user operation to the auditory characteristic table generating unit 62 .

[0187] The auditory characteristic table generation unit 62 generates a new auditory characteristic table based on the input signal supplied from the input unit 61 and the auditory characteristic table of preset auditory characteristics held in the auditory characteristic table holding unit 63 , and supplies the new auditory characteristic table to the auditory characteristic table holding unit 63 .

[0188] Furthermore, the auditory characteristics table generating unit 62 , for example, appropriately instructs the display control unit 64 to update the display of the display screen in the display device 52 when generating the auditory characteristics table.

[0189] The auditory characteristic table holding unit 63 holds an auditory characteristic table of auditory characteristics preset in advance, supplies the auditory characteristic table to the auditory characteristic table generating unit 62 as appropriate, and holds the auditory characteristic table supplied from the auditory characteristic table generating unit 62 .

[0190] The display control unit 64 controls the display of the display screen by the display device 52 according to the instruction from the auditory characteristics table generation unit 62 .

[0191] Notice, Fig.12 The input unit 61, the auditory characteristic table generating unit 62, and the display control unit 64 shown in FIG. Figure 4 In the information processing device 11 shown.

[0192]

[0193] Subsequently, the operation of the information processing device 51 will be described.

[0194] That is, the following will refer to Fig.13 The table generation process performed by the information processing device 51 is described with reference to the flowchart of FIG.

[0195] In step S41 , the display control unit 64 causes the display device 52 to display a display screen of the content creation tool in response to an instruction from the auditory characteristic table generation unit 62 .

[0196] Specifically, for example, the display control unit 64 causes the display device 52 to display Figure 8 , Fig. 9 , Fig.10 , Fig.11 The display screen shown in the figure is shown in the figure.

[0197] At this time, for example, when the user operates the input unit 61 and selects a preset auditory characteristic, the auditory characteristic table generation unit 62 reads the auditory characteristic table corresponding to the auditory characteristic selected by the user from the auditory characteristic table holding unit 63 in response to an input signal provided from the input unit 61.

[0198] The auditory characteristic table generation unit 62 instructs the display control unit 64 to display the gain correction value display area so as to display the gain correction value for each azimuth value indicated by the auditory characteristic table that has been read on the display device 52. In response to the instruction of the auditory characteristic table generation unit 62, the display control unit 64 causes the display screen of the display device 52 to display the gain correction value display area.

[0199] When the display screen of the content creation tool is displayed on the display device 52 , the user appropriately operates the input unit 61 , and operates a slider or the like displayed in the slider display area, thereby instructing a change (adjustment) of the gain correction value.

[0200] Then, in step S42 , the auditory characteristic table generation unit 62 generates an auditory characteristic table based on the input signal supplied from the input unit 61 .

[0201] That is, the auditory characteristic table generation unit 62 generates a new auditory characteristic table by changing the auditory characteristic table read from the auditory characteristic table holding unit 63 according to the input signal provided from the input unit 61. That is, the preset auditory characteristic table is changed (updated) according to the operation of the slider displayed in the slider display area.

[0202] Therefore, when the gain correction value for each azimuth value is adjusted (changed) according to the operation of a slider or the like and a new auditory characteristic table is generated, the auditory characteristic table generation unit 62 instructs the display control unit 64 to update the display of the gain correction value display area according to the new auditory characteristic table.

[0203] In step S43 , the display control unit 64 controls the display device 52 according to the instruction of the auditory characteristic table generating unit 62 , and performs display according to the newly generated auditory characteristic table.

[0204] Specifically, the display control unit 64 updates the display of the gain correction value display area on the display screen of the display device 52 according to the newly generated auditory characteristics table.

[0205] In step S44 , the auditory characteristic table generation unit 62 determines whether to end the process based on the input signal supplied from the input unit 61 .

[0206] For example, when the user operates the input unit 61 and operates a save button or the like displayed on the display device 52 , the auditory characteristic table generating unit 62 determines to end the processing if a signal indicating an instruction to save the auditory characteristic table is provided as an input signal.

[0207] In a case where it is determined in step S44 that the processing has not been ended, the processing returns to step S42, and the above-described processing is repeatedly executed.

[0208] On the other hand, in the case where it is determined in step S44 that the processing is to be ended, the processing proceeds to step S45.

[0209] In step S45 , the auditory characteristic table generating unit 62 supplies the auditory characteristic table obtained in the most recently executed step S42 as a newly generated auditory characteristic table to the auditory characteristic table holding unit 63 , and causes the auditory characteristic table holding unit 63 to hold the newly generated auditory characteristic table.

[0210] When the auditory characteristic table is retained in the auditory characteristic table holding unit 63, the table generation process ends.

[0211] As described above, the information processing device 51 causes the display device 52 to display the display screen of the content creation tool, and adjusts the gain correction value according to the user operation, thereby generating a new auditory characteristic table.

[0212] This allows the user to easily and intuitively obtain an auditory characteristic table corresponding to the desired auditory characteristic. Thus, the user can create 3D audio content of sufficiently high quality more easily, i.e. in a short time.

[0213] <Third Embodiment>

[0214] <Voice Processing Device Configuration Example>

[0215] Furthermore, for example, in free viewpoint content, since the position of the listener in the three-dimensional space can be freely moved, the relative positional relationship between the objects in the three-dimensional space and the listener also changes as the listener moves.

[0216] A technology in which, in a case where the position of a listener can be freely moved as described above, the position of a sound source is corrected according to a change in the position of the listener, and a rendering process is performed based on the obtained corrected position information (for example, see WO2015 / 107926).

[0217] The present technology is also applicable to a reproduction device that reproduces such free viewpoint content. In this case, gain correction is performed using not only correction position information but also the above-mentioned three-dimensional auditory characteristics.

[0218] Fig.14 is a diagram showing a configuration example of an embodiment of a speech processing device used as a reproduction device for reproducing content from a free viewpoint to which the present technology is applied. Fig.14 In, with Figure 4The corresponding parts in the figure are given the same figure marks, and their descriptions are appropriately omitted.

[0219] Fig.14 The illustrated speech processing device 91 has an input unit 121 , a position information correction unit 122 , a gain / frequency characteristic correction unit 123 , an auditory characteristic table holding unit 22 , a spatial acoustic characteristic adding unit 124 , a renderer processing unit 125 , and a convolution processing unit 126 .

[0220] The audio signal of the object and the metadata of the audio signal of each object are provided to the voice processing device 91 as the audio information of the content to be reproduced. Fig.14 , an example in which audio signals and metadata of two objects are provided to the information processing device 91 will be described, but the present invention is not limited to this, and the number of objects can be any number.

[0221] Here, the metadata provided to the voice processing device 91 is the position information and the gain initial value of the object.

[0222] In addition, the position information includes the above-mentioned azimuth value, elevation value, and radius value, and is information indicating the position of the object viewed from the reference position in the three-dimensional space, that is, the localization position of the object sound. Note that hereinafter, the reference position in the three-dimensional space is also particularly referred to as the standard listening position.

[0223] The input unit 121 includes a mouse, a button, a touch screen, etc., and when operated by a user, outputs a signal according to the operation. For example, the input unit 121 accepts an input of an assumed listening position of the user, and provides assumed listening position information indicating the assumed listening position input by the user to the position information correction unit 122 and the spatial acoustic characteristic adding unit 124.

[0224] Here, the assumed listening position is a listening position of the sound constituting the content in the virtual sound field that is intended to be reproduced. Therefore, it can be said that the assumed listening position indicates a position after the change when the predetermined standard listening position is changed (corrected).

[0225] Based on the assumed listening position information supplied from the input unit 121 and the direction information indicating the orientation of the listener supplied from the outside, the position information correction unit 122 corrects the position information of metadata which is an object supplied from the outside.

[0226] The position information correction unit 122 supplies the corrected position information obtained through the correction of the position information to the gain / frequency characteristic correction unit 123 and the renderer processing unit 125 .

[0227] Note that, for example, the direction information can be obtained from a gyro sensor or the like provided on the head of the user (listener). In addition, the corrected position information is information indicating the position of the object viewed from a listener appearing at the assumed listening position and facing the direction indicated by the direction information, i.e., the localized position of the object sound.

[0228] The gain / frequency characteristic correction unit 123 performs gain correction and frequency characteristic correction on the audio signal of the object supplied externally based on the corrected position information supplied from the position information correction unit 122, the auditory characteristic table held in the auditory characteristic table holding unit 22, and metadata supplied externally.

[0229] The gain / frequency characteristic correction unit 123 supplies the audio signal obtained through the gain correction and the frequency characteristic correction to the spatial acoustic characteristic adding unit 124 .

[0230] Based on the assumed listening position information supplied from the input unit 121 and the position information of the object supplied from the outside, the spatial acoustic characteristic adding unit 124 adds spatial acoustic characteristics to the audio signal supplied from the gain / frequency characteristic correcting unit 123 and supplies the audio signal to the renderer processing unit 125 .

[0231] Based on the corrected position information supplied from the position information correcting unit 122 , the renderer processing unit 125 performs rendering processing, that is, mapping processing, on the audio signal supplied from the spatial acoustic characteristic adding unit 124 , and generates reproduction signals of 2 or more M channels.

[0232] That is, an M-channel reproduction signal is generated from the audio signal of each object. The renderer processing unit 125 supplies the generated M-channel reproduction signal to the convolution processing unit 126.

[0233] The M-channel reproduction signal thus obtained is an audio signal that reproduces the sound output from each object, the sound being listed at an assumed listening position where it is desired to reproduce the virtual sound field required to be reproduced by reproducing with the virtual M speakers (M-channel speakers).

[0234] The convolution processing unit 126 performs convolution processing on the M-channel reproduction signal supplied from the renderer processing unit 125 , and generates and outputs a 2-channel reproduction signal.

[0235] That is, in this example, the device on the reproduction side of the content is a headphone, and the convolution processing unit 126 generates and outputs a reproduction signal to be reproduced by two speakers (drivers) provided in the headphone.

[0236] <Explanation of Reproduction Signal Generation Processing>

[0237] Subsequently, the operation of the voice processing device 91 will be described.

[0238] That is, the following will refer to Fig.15 The reproduction signal generation process performed by the voice processing device 91 is described with reference to a flowchart of FIG.

[0239] In step S71 , the input unit 121 accepts input of an assumed listening position.

[0240] When the user operates the input unit 121 and inputs an assumed listening position, the input unit 121 supplies the assumed listening position information indicating the assumed listening position to the position information correction unit 122 and the spatial acoustic characteristic addition unit 124 .

[0241] Based on the assumed listening position information supplied from the input unit 121 and the position information and direction information of the object supplied from the outside, the position information correction unit 122 calculates the corrected position information in step S72 .

[0242] The position information correction unit 122 supplies the corrected position information obtained for each object to the gain / frequency characteristic correction unit 123 and the renderer processing unit 125 .

[0243] Based on the corrected position information provided from the position information correction unit 122, the metadata provided externally, and the auditory characteristic table held in the auditory characteristic table holding unit 22, the gain / frequency characteristic correction unit 123 performs gain correction and frequency characteristic correction on the audio signal of the object provided externally in step S73.

[0244] Specifically, for example, the gain / frequency characteristic correction unit 123 reads the gain correction value associated with the azimuth value, the elevation value, and the radius value constituting the correction position information from the auditory characteristic table.

[0245] Furthermore, the gain / frequency characteristic correction unit 123 corrects the gain correction value by multiplying the gain correction value by the ratio between the radius value of the position information provided as metadata and the radius value of the corrected position information, and obtains the gain value by correcting the gain initial value by the obtained gain value.

[0246] Therefore, gain correction according to the direction of the object viewed from the assumed listening position and gain correction according to the distance from the assumed listening position to the object are achieved by gain correction having a gain value.

[0247] Furthermore, the gain / frequency characteristic correction unit 123 selects a filter coefficient based on the radius value of the position information provided as metadata and the radius value of the corrected position information.

[0248] The filter coefficients thus selected are used for filter processing to achieve the desired frequency characteristic correction. More specifically, for example, the filter coefficients are used to reproduce a property in which the high-frequency components of the sound from the object are attenuated by the walls or ceiling of the virtual sound field desired to be reproduced, depending on the distance from the assumed listening position to the object.

[0249] The gain / frequency characteristic correction unit 123 realizes gain correction and frequency characteristic correction by performing gain correction and filter processing on the audio signal of the object based on the filter coefficient and gain value obtained as described above.

[0250] The gain / frequency characteristic correction unit 123 supplies the audio signal of each object obtained through the gain correction and the frequency characteristic correction to the spatial acoustic characteristic adding unit 124 .

[0251] Based on the assumed listening position information supplied from the input unit 121 and the position information of the object supplied from the outside, the spatial acoustic characteristic adding unit 124 adds spatial acoustic characteristics to the audio signal supplied from the gain / frequency characteristic correcting unit 123 in step S74 and supplies the audio signal to the renderer processing unit 125 .

[0252] For example, the spatial acoustic characteristic adding unit 124 adds spatial acoustic characteristics by performing multi-tap delay processing, comb filter processing, and all-pass filter processing on the audio signal based on the delay amount and gain amount determined from the position information of the object and the assumed listening position information. Therefore, for example, initial reflection, reverberation characteristics, etc. are added to the audio signal as spatial acoustic characteristics.

[0253] In step S75 , the renderer processing unit 125 performs mapping processing on the audio signal supplied from the spatial acoustic characteristic adding unit 124 based on the corrected position information supplied from the position information correcting unit 122 , thereby generating an M-channel reproduction signal and supplies the reproduction signal to the convolution processing unit 126 .

[0254] For example, in the process of step S75, the reproduction signal is generated by VBAP, but the M channel reproduction signal may be generated by any method.

[0255] In step S76, the convolution processing unit 126 performs convolution processing on the M channel reproduction signal supplied from the renderer processing unit 125, thereby generating and outputting a 2 channel reproduction signal. For example, binaural room impulse response (BRIR) processing is performed as the convolution processing.

[0256] When the 2-channel reproduction signal is generated and output, the reproduction signal generation process ends.

[0257] As described above, the voice processing device 91 calculates correction position information based on the assumed listening position information, performs gain correction and frequency characteristic correction on the audio signal of each object based on the obtained correction position information and the assumed listening position information, and increases the spatial acoustic characteristics.

[0258] This makes it easier to perform appropriate gain correction and frequency characteristic correction. In addition, it is possible to realistically reproduce how a listener hears the sound output from each object at an arbitrarily assumed listening position. Therefore, the user can freely specify the listening position according to his or her preferences when reproducing content, and can achieve audio reproduction with a higher degree of freedom.

[0259] Note that in step S73 , in addition to performing gain correction and frequency characteristic correction according to the distance from the assumed listening position to the object based on the correction position information, gain correction based on three-dimensional auditory characteristics is performed using the auditory characteristic table.

[0260] At this time, the auditory characteristic table used in step S73 is, for example, Fig.16 The auditory characteristics table shown.

[0261] Fig.16 The auditory characteristics table shown is obtained by Figure 6 The gain correction values ​​shown in the auditory characteristics table are obtained by inverting the signs of the gain correction values.

[0262] By correcting the initial gain value using such an auditory characteristic table, it is possible to reproduce, through gain correction, a phenomenon in which the auditory loudness of a sound changes depending on the direction from which the sound comes, even if the sound comes from the same object (sound source). This makes it possible to achieve sound field reproduction with a higher sense of reality.

[0263] On the other hand, depending on the reproduction conditions, sometimes by using Figure 6 The auditory characteristics shown in the table are not Fig.16 The auditory characteristics table shown can be used to achieve more appropriate gain correction.

[0264] That is, for example, a case where speaker reproduction is performed using actual speakers arranged in a three-dimensional space instead of using headphones for content reproduction will be considered.

[0265] In this case, in the voice processing device 91, the M-channel reproduction signal obtained by the renderer processing unit 125 is supplied to the speaker corresponding to each of the M channels, and the sound of the content is reproduced.

[0266] In content reproduction using such actual speakers, the sound source, that is, the sound of the object is actually reproduced at the position of the object viewed from the assumed listening position.

[0267] Therefore, there is no need to perform gain correction to reproduce the phenomenon that the loudness of the acoustic sound changes depending on the arrival direction of the sound, but sometimes it is undesirable to change the loudness of the acoustic sound so as not to change the volume balance.

[0268] In this case, in step S73, only the Figure 6 The gain correction value is determined by using the auditory characteristic table shown, and it is only required to correct the gain initial value using the gain correction value. Therefore, gain correction is performed so that the loudness of the sound becomes constant auditorily regardless of the direction in which the object exists.

[0269] <Variation 1 of the Third Embodiment>

[0270] <Encoding and transmission of gain auditory characteristic information>

[0271] Incidentally, audio signals, metadata, and the like are sometimes encoded and transmitted via an encoded bit stream.

[0272] In this case, for example, the gain / frequency characteristic correction unit 123 may transmit gain auditory characteristic information including flag information on whether to perform gain correction using an auditory characteristic table and the like through an encoded bit stream.

[0273] At this time, the gain auditory characteristic information includes not only flag information but also an auditory characteristic table, index information indicating an auditory characteristic table used for gain correction among a plurality of auditory characteristic tables, and the like.

[0274] The syntax for such gain auditory characteristic information may be, for example, Fig.17 The syntax is shown.

[0275] exist Fig.17 In the example of , characters “numGainAuditoryPropertyTables” represent the number of auditory characteristic tables transmitted by the encoded bit stream, that is, the number of auditory characteristic tables included in the gain auditory characteristic information.

[0276] Furthermore, characters “numElements[i]” represent the number of elements constituting the i-th auditory characteristic table included in the gain auditory characteristic information.

[0277] The elements mentioned here are the azimuth value, elevation value, radius value, and gain correction value which are related to each other.

[0278] Furthermore, the characters "azimuth[i][n]", "elevation[i][n]", and "radius[i][n]" represent the azimuth value, elevation value, and radius value of the nth element constituting the i-th auditory characteristic table.

[0279] In other words, azimuth[i][n], elevation[i][n], and radius[i][n] represent the direction from which the sound of the object serving as the sound source arrives, namely, the horizontal angle, the vertical angle, and the distance (radius) representing the position of the object.

[0280] Furthermore, characters "gainCompensValue[i][n]" represent the gain correction value of the nth element constituting the i-th auditory characteristic table, that is, the gain correction value with respect to the position (direction) represented by azimuth[i][n], elevation[i][n], and radius[i][n].

[0281] Furthermore, the characters "hasGainCompensObjects" are flag information indicating whether there is an object on which gain correction using the sense of hearing characteristic table is to be performed.

[0282] Furthermore, characters "num_objects" represent the number of objects constituting the content (the number of objects), and this number of objects num_objects is transmitted to the device on the reproduction side of the content, that is, the voice processing device, separately from the gain auditory characteristic information.

[0283] In the case where the value of the flag information hasGainCompensObjects is a value indicating the presence of an object for which gain correction using the auditory characteristic table is to be performed, the gain auditory characteristic information includes flag information indicated by the number of objects num_objects by characters “isGainCompensObject[o]”.

[0284] The flag information isGainCompensObject[o] indicates whether gain correction is performed using the auditory characteristic table with respect to the o-th object.

[0285] Furthermore, in the case where the value isGainCompensObject[o] of the flag information is a value indicating that gain correction is performed using the auditory characteristic table, the gain auditory characteristic information includes an index indicated by characters “applyTableIndex[o]”.

[0286] This index applyTableIndex[o] is information indicating the auditory characteristic table used when gain correction is performed with respect to the o-th object.

[0287] For example, when the number numGainAuditoryPropertyTables of the auditory characteristic table is 0, the auditory characteristic table is not transmitted, and the gain auditory characteristic information does not include the index applyTableIndex[o]. That is, the index applyTableIndex[o] is not transmitted.

[0288] In this case, for example, gain correction may be performed using the auditory characteristic table retained in the auditory characteristic table holding unit 22, or gain correction may not be performed.

[0289] <Voice Processing Device Configuration Example>

[0290] In the case where the gain auditory characteristic information as described above is transmitted through a coded bit stream, for example, Fig.18 The voice processing device is configured as shown. Fig.18 In, with Fig.14 The corresponding parts in the figure are given the same figure marks, and their descriptions are appropriately omitted.

[0291] Fig.18 The illustrated speech processing device 151 has an input unit 121 , a position information correction unit 122 , a gain / frequency characteristic correction unit 123 , an auditory characteristic table holding unit 22 , a spatial acoustic characteristic adding unit 124 , a renderer processing unit 125 , and a convolution processing unit 126 .

[0292] The configuration of the speech processing device 151 is Fig.14 The configuration of the illustrated speech processing device 91 is the same, but differs from the speech processing device 91 in that an auditory characteristic table or the like read from gain auditory characteristic information extracted from an encoded bit stream is supplied to the gain / frequency characteristic correction unit 123 .

[0293] That is, in the speech processing device 151 , the auditory characteristic table read from the gain auditory characteristic information, flag information hasGainCompensObjects, flag information isGainCompensObject[o], index applytableindex[o], and the like are supplied to the gain / frequency characteristic correction unit 123 .

[0294] The speech processing device 151 basically performs reference Fig.15 Described reproduction signal generation processing.

[0295] However, when the number of auditory characteristic tables numGainAuditoryPropertyTables is 0, that is, when no auditory characteristic table is provided externally, the gain / frequency characteristic correction unit 123 performs gain correction using the auditory characteristic table retained in the auditory characteristic table holding unit 22 in step S73.

[0296] On the other hand, in the case where the auditory characteristic table is supplied from the outside, the gain / frequency characteristic correction unit 123 performs gain correction using the supplied auditory characteristic table.

[0297] Specifically, the gain / frequency characteristic correction unit 123 performs gain correction with respect to the o-th object using the auditory characteristic table indicated by the index applyTableIndex[o] among the plurality of auditory characteristic tables supplied from the outside.

[0298] However, for an object whose value of the flag information isGainCompensObject[o] is a value indicating that gain correction is not performed using the auditory characteristic table, the gain / frequency characteristic correction unit 123 does not perform gain correction using the auditory characteristic table.

[0299] That is, in the case where flag information to which a value indicating that gain correction is performed using the auditory characteristic table is provided isGainCompensObject[o], the gain / frequency characteristic correction unit 123 performs gain correction using the auditory characteristic table indicated by the index applyTableIndex[o].

[0300] Also, for example, when the value of the flag information hasGainCompensObjects is a value indicating that there is no object for which gain correction using the auditory characteristic table is to be performed, the gain / frequency characteristic correction unit 123 does not perform gain correction using the auditory characteristic table on the object.

[0301] As described above, according to the present technology, it is possible to easily decide gain information for each object, that is, a gain value in 3D mixing of object audio, reproduction of content of a free viewpoint, etc. Therefore, gain correction can be performed more easily.

[0302] Furthermore, according to the present technology, when the listening position changes, it is possible to appropriately correct the change in volume in the auditory sense accompanying the change in the relative positional relationship between the listener and the object (sound source).

[0303] <Computer Configuration Example>

[0304] Incidentally, the series of processing described above may be performed by hardware, or may be performed by software. In the case where the series of processing is performed by software, the program constituting the software is installed in a computer. Here, the computer includes a computer integrated in dedicated hardware, and, for example, a general-purpose personal computer capable of performing various functions by installing various programs.

[0305] Fig.19 : is a block diagram showing a configuration example of hardware of a computer that executes the above-described series of processes by a program.

[0306] In the computer, a central processing unit (CPU) 501 , a read only memory (ROM) 502 , and a random access memory (RAM) 503 are interconnected by a bus 504 .

[0307] An input / output interface 505 is further connected to the bus 504 . An input unit 506 , an output unit 507 , a recording unit 508 , a communication unit 509 , and a drive 510 are connected to the input / output interface 505 .

[0308] The input unit 506 includes a keyboard, a mouse, a microphone, an imaging element, etc. The output unit 507 includes a display, a speaker, etc. The recording unit 508 includes a hard disk, a nonvolatile memory, etc. The communication unit 509 includes a network interface, etc. The drive 510 drives a removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0309] The computer configured as described above performs a series of processes as described above when the CPU 501 loads the program recorded in the recording unit 508 , for example, to the RAM 503 via the input / output interface 505 and the bus 504 and executes the program.

[0310] For example, the program executed by the computer (CPU 501) may be recorded as a package medium and provided in the removable recording medium 511. Furthermore, the program may be provided via a wired or wireless transmission medium such as a local area network, the Internet, and digital satellite broadcasting.

[0311] By mounting the removable recording medium 511 to the drive 510, the computer can install the program into the recording unit 508 through the input / output interface 505. In addition, the program can be received by the communication unit 509 via a wired or wireless transmission medium and installed in the recording unit 508. In addition to this, the program can be pre-installed in the ROM 502 or the recording unit 508.

[0312] Note that the program executed by the computer may be a program that executes processing in time series along the order described in this specification, or may be a program that executes processing in parallel or at necessary timing such as when a call is made.

[0313] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made within a scope that does not depart from the gist of the present technology.

[0314] For example, the present technology may have a configuration of cloud computing in which one function is shared by a plurality of devices through a network and processed in coordination.

[0315] Furthermore, each step described in the above flowchart may be performed by one device or by a plurality of devices in a shared manner.

[0316] Furthermore, in the case where one step includes a plurality of processes, the plurality of processes included in one step may be executed by one device, or may be executed by a plurality of devices in a shared manner.

[0317] Furthermore, the present technology can have the following configurations.

[0318] (1) An information processing device comprising:

[0319] The gain correction value determination unit determines a correction value of a gain value for performing gain correction on an audio signal of the audio object according to a direction from which the audio object is viewed from a listener.

[0320] (2) The information processing device according to (1), wherein

[0321] The gain correction value determination unit determines the correction value based on the three-dimensional auditory characteristics of the listener with respect to the sound arrival direction.

[0322] (3) The information processing device according to (1) or (2), wherein

[0323] The gain correction value determination unit determines the correction value based on the direction of the listener.

[0324] (4) The information processing device according to any one of (1) to (3), wherein

[0325] The gain correction value decision unit decides the correction value so that the correction value becomes larger in a case where the audio object exists behind the listener than in a case where the audio object exists in front of the listener.

[0326] (5) The information processing device according to any one of (1) to (4), wherein

[0327] The gain correction value decision unit decides the correction value so that the correction value becomes smaller when the audio object exists at the side of the listener than when the audio object exists in front of the listener.

[0328] (6) The information processing device according to any one of (1) to (5), wherein

[0329] The gain correction value decision unit decides the correction value corresponding to the predetermined direction by obtaining the correction value corresponding to the predetermined direction through interpolation processing based on the correction value corresponding to another direction.

[0330] (7) The information processing device according to (6), wherein

[0331] The gain correction value decision unit performs vector base amplitude shift VBAP as an interpolation process.

[0332] (8) The information processing device according to (7), wherein

[0333] The gain correction value determination unit obtains the correction value in a linear value or a decibel value.

[0334] (9) An information processing method, wherein

[0335] An information processing apparatus determines a correction value of a gain value for performing gain correction on an audio signal of an audio object according to a direction from which the audio object is viewed from a listener.

[0336] (10) A program that causes a computer to execute a process comprising the following steps:

[0337] A correction value of a gain value for performing gain correction on an audio signal of the audio object is decided according to a direction from which the audio object is viewed by a listener.

[0338] (11) A reproduction device comprising:

[0339] a gain correction unit that decides a correction value of a gain value for performing gain correction on an audio signal of the audio object based on position information indicating the position of the audio object, the correction value depending on a direction from which the audio object is viewed by a listener, and performs gain correction on the audio signal based on the gain value corrected by the correction value, and

[0340] The renderer processing unit performs rendering processing based on the audio signal obtained by the gain correction, and generates reproduction signals of a plurality of channels for reproducing the sound of the audio object.

[0341] (12) The reproduction device according to (11), wherein

[0342] The gain correction unit corrects the gain value included in the metadata of the audio signal by the correction value.

[0343] (13) The reproduction device according to (11) or (12), wherein

[0344] In the case where a flag indicating that correction of the gain value is performed is provided, the gain correction unit corrects the gain value by the correction value.

[0345] (14) The reproduction device according to (13), wherein

[0346] The gain correction unit decides the correction value by using a table indicated by the provided index among a plurality of tables in which directions from which the listener views the audio object are associated with the correction values.

[0347] (15) The reproduction device according to any one of (11) to (14), further comprising:

[0348] a position information correction unit that corrects the position information included in the metadata of the audio signal based on the information indicating the position of the listener, wherein

[0349] The gain correction unit decides a correction value based on the position information that has been corrected.

[0350] (16) The reproduction device according to (15), wherein

[0351] The position information correction unit corrects the position information based on information indicating the position of the listener and direction information indicating the orientation of the listener.

[0352] (17) A reproduction method, wherein:

[0353] Reproduction device:

[0354] determining a correction value of a gain value for performing gain correction on an audio signal of the audio object based on position information indicating the position of the audio object, the correction value depending on a direction from which the audio object is viewed by a listener,

[0355] performing gain correction on the audio signal based on the gain value corrected by the correction value,

[0356] performing rendering processing based on the audio signal obtained by the gain correction, and

[0357] A reproduction signal of multiple channels is generated for reproducing the sound of the audio object.

[0358] (18) A program that causes a computer to execute a process comprising the following steps:

[0359] determining a correction value of a gain value for performing gain correction on an audio signal of the audio object based on position information indicating the position of the audio object, the correction value depending on a direction from which the audio object is viewed by a listener,

[0360] performing gain correction on the audio signal based on the gain value corrected by the correction value,

[0361] performing rendering processing based on the audio signal obtained by the gain correction, and

[0362] A reproduction signal of multiple channels is generated for reproducing the sound of the audio object.

[0363] Reference numerals list

[0364] 11 Information processing device

[0365] 21 Gain correction value determination unit

[0366] 22. Auditory characteristics table holding unit

[0367] 62 Auditory characteristic table generation unit

[0368] 64 Display control unit

[0369] 122 Position information correction unit

[0370] 123 Gain / frequency characteristic correction unit.

Claims

1. An information processing device, comprising: a gain correction value determination unit that determines a correction value of a gain value for performing gain correction on an initial gain value of an audio signal of the audio object according to a direction from which the audio object is viewed by a listener; wherein the gain correction value determination unit determines the correction value based on the provided position information and the auditory characteristic table; The position information includes: an azimuth value indicating the angle of the position of the audio object in the horizontal direction as viewed from the listener; an elevation value indicating the angle of the position of the audio object in the vertical direction as viewed from the listener; and a radius value indicating the distance from the listener to the audio object; and In the auditory characteristic table, the correction value is associated with the azimuth value, the elevation value, and the radius value, The gain correction value determination unit determines the correction value so that the correction value becomes larger when the audio object exists behind the listener than when the audio object exists in front of the listener.

2. The information processing device according to claim 1, wherein: The gain correction value determination unit determines the correction value based on the three-dimensional auditory characteristics of the listener with respect to the sound arrival direction.

3. The information processing device according to claim 1, wherein: The gain correction value determination unit determines the correction value based on the direction of the listener.

4. The information processing device according to claim 1, wherein: The gain correction value determination unit determines the correction value so that the correction value becomes smaller when the audio object exists at the side of the listener than when the audio object exists in front of the listener.

5. The information processing device according to claim 1, wherein: The gain correction value decision unit decides the correction value corresponding to the predetermined direction by obtaining the correction value corresponding to the predetermined direction through interpolation processing based on the correction value corresponding to another direction.

6. The information processing device according to claim 5, wherein: The gain correction value decision unit performs vector base amplitude shift VBAP as the interpolation process.

7. The information processing device according to claim 6, wherein: The gain correction value determination unit obtains the correction value through a linear value or a decibel value.

8. An information processing method, wherein: An information processing device that determines a correction value of a gain value for performing gain correction on an initial gain value of an audio signal of the audio object according to a direction from which a listener views the audio object; wherein the information processing device determines the correction value based on the provided position information and the auditory characteristic table; The position information includes: an azimuth value indicating the angle of the position of the audio object in the horizontal direction as viewed from the listener; an elevation value indicating the angle of the position of the audio object in the vertical direction as viewed from the listener; and a radius value indicating the distance from the listener to the audio object; and In the auditory characteristic table, the correction value is associated with the azimuth value, the elevation value, and the radius value, The information processing device determines the correction value so that the correction value becomes larger when the audio object exists behind the listener than when the audio object exists in front of the listener.

9. A computer-readable storage medium storing a program which, when executed by a computer, causes the computer to perform a process comprising the following steps: determining a correction value of a gain value for performing gain correction on an initial gain value of an audio signal of the audio object according to a direction from which the audio object is viewed by a listener; in, The correction value is determined based on the provided position information and the auditory characteristic table; The position information includes: an azimuth value indicating the angle of the position of the audio object in the horizontal direction as viewed from the listener; an elevation value indicating the angle of the position of the audio object in the vertical direction as viewed from the listener; and a radius value indicating the distance from the listener to the audio object; and In the auditory characteristic table, the correction value is associated with the azimuth value, the elevation value, and the radius value, The correction value is determined such that the correction value becomes larger when the audio object is behind the listener than when the audio object is in front of the listener.

10. A reproduction device, comprising: a gain correction unit that decides, based on position information indicating a position of an audio object, a correction value of a gain value for performing gain correction on an initial value of a gain of an audio signal of the audio object, the correction value depending on a direction from which the audio object is viewed by a listener, and performs the gain correction on the audio signal based on the gain value corrected by the correction value; as well as a renderer processing unit that performs rendering processing based on the audio signal obtained by the gain correction, and generates a reproduction signal of a plurality of channels for reproducing the sound of the audio object; wherein the gain correction unit determines the correction value based on the position information and the auditory characteristic table; The position information includes: an azimuth value indicating the angle of the position of the audio object in the horizontal direction as viewed from the listener; an elevation value indicating the angle of the position of the audio object in the vertical direction as viewed from the listener; and a radius value indicating the distance from the listener to the audio object; and In the auditory characteristic table, the correction value is associated with the azimuth value, the elevation value, and the radius value, The gain correction value determination unit determines the correction value so that the correction value becomes larger when the audio object exists behind the listener than when the audio object exists in front of the listener.

11. The reproduction device according to claim 10, wherein: The gain correction unit corrects the gain initial value included in the metadata of the audio signal by the correction value.

12. The reproduction device according to claim 10, wherein: In a case where a flag indicating execution of correction of the gain value is provided, the gain correction unit corrects the gain initial value by the correction value.

13. The reproduction device according to claim 12, wherein: The gain correction unit decides the correction value by using a table indicated by a provided index among a plurality of tables in which directions of the audio object viewed from the listener are associated with the correction values.

14. The reproduction device according to claim 10, further comprising: a position information correction unit that corrects the position information included in the metadata of the audio signal based on information indicating the position of the listener, wherein The gain correction unit determines the correction value based on the position information that has been corrected.

15. The reproduction device according to claim 14, wherein: The position information correction unit corrects the position information based on information indicating the position of the listener and direction information indicating the orientation of the listener.

16. A reproduction method, wherein: Reproduction device: determining a correction value of a gain value for performing gain correction on an initial value of a gain of an audio signal of the audio object based on position information indicating the position of the audio object, the correction value depending on a direction from which the audio object is viewed by a listener, performing the gain correction on the audio signal based on the gain value corrected by the correction value, performing rendering processing based on the audio signal obtained by the gain correction, and generating a reproduction signal of a plurality of channels for reproducing the sound of the audio object; wherein the reproduction device determines the correction value based on the position information and the auditory characteristic table; The position information includes: an azimuth value indicating the angle of the position of the audio object in the horizontal direction as viewed from the listener; an elevation value indicating the angle of the position of the audio object in the vertical direction as viewed from the listener; and a radius value indicating the distance from the listener to the audio object; and In the auditory characteristic table, the correction value is associated with the azimuth value, the elevation value, and the radius value, The reproduction device determines the correction value so that the correction value becomes larger when the audio object exists behind the listener than when the audio object exists in front of the listener.

17. A computer-readable storage medium storing a program, which, when executed by a computer, causes the computer to perform a process comprising the following steps: determining a correction value of a gain value for performing gain correction on an initial value of a gain of an audio signal of the audio object based on position information indicating the position of the audio object, the correction value depending on a direction from which the audio object is viewed by a listener, performing the gain correction on the audio signal based on the gain value corrected by the correction value, performing rendering processing based on the audio signal obtained by the gain correction, and generating a reproduction signal of a plurality of channels for reproducing the sound of the audio object; in, The correction value is determined based on the position information and the auditory characteristics table; The position information includes: an azimuth value indicating the angle of the position of the audio object in the horizontal direction as viewed from the listener; an elevation value indicating the angle of the position of the audio object in the vertical direction as viewed from the listener; and a radius value indicating the distance from the listener to the audio object; and In the auditory characteristic table, the correction value is associated with the azimuth value, the elevation value, and the radius value, The correction value is determined such that the correction value becomes larger when the audio object is behind the listener than when the audio object is in front of the listener.

Images