Sound processing device and method, and program

Abstract

To achieve audio reproduction with higher degree of freedom.SOLUTION: An input unit receives input of an assumed listening position of sound of an object, which is a sound source, and outputs assumed listening position information indicating the assumed listening position. A position information correction unit corrects position information of each object on the basis of the assumed listening position information to obtain corrected position information. A gain / frequency characteristic correction unit performs gain correction and frequency characteristic correction on a waveform signal of an object on the basis of the position information and the corrected position information. A spatial acoustic characteristic addition unit further adds a spatial acoustic characteristic to the waveform signal resulting from the gain correction and the frequency characteristic correction on the basis of the position information of the object and the assumed listening position information. The present technology is applicable to an audio processing device.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present technology relates to an audio processing device, method, and program, and more particularly to an audio processing device, method, and program that enable audio reproduction with a higher degree of freedom to be realized. [Background technology]

[0002] Generally, audio contents such as CDs (Compact Discs), DVDs (Digital Versatile Discs), and network-distributed audio are realized using channel-based audio.

[0003] Channel-based audio content is created by content creators by appropriately mixing multiple sound sources, such as vocals and musical instrument sounds, into 2-channel or 5.1-channel (hereafter, "channel" will also be referred to as "ch"). Users play it back through a 2-channel or 5.1-channel speaker system or headphones.

[0004] However, speaker placement and other aspects of users vary widely, and sound localization may not always be as intended by the content creator.

[0005] Meanwhile, object-based audio technology has been attracting attention in recent years. In object-based audio, a signal is rendered according to the playback system based on the waveform signal of the object's sound and metadata indicating the object's localization information, etc., indicated by its relative position from a reference listening point. Therefore, object-based audio has the advantage that the localization of sound is reproduced relatively as intended by the content creator.

[0006] For example, in object-based audio, a technology such as VBAP (Vector Base Amplitude Panning) is used to generate playback signals for channels corresponding to each speaker on the playback side from the waveform signal of each object (see, for example, Non-Patent Document 1).

[0007] In VBAP, the position of the target sound image is expressed as a linear sum of vectors pointing in the direction of two or three speakers around the position. The coefficients multiplied by each vector in the linear sum are used as the gain of the waveform signal output from each speaker to adjust the gain so that the sound image is localized at the target position. [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] Ville Pulkki, “Virtual Sound Source Positioning Using Vector Base Amplitude Panning”, Journal of AES, vol.45, no.6, pp.456-466, 1997 Summary of the Invention [Problem to be solved by the invention]

[0009] However, in both the channel-based audio and object-based audio described above, the sound localization is determined by the content creator, and the user can only listen to the audio of the provided content as is. For example, on the content playback side, it was not possible to reproduce how the sound would be heard if the listening point were changed as if moving from the back seat to the front seat in a live music venue.

[0010] As described above, the above-mentioned techniques cannot be said to realize audio reproduction with a sufficiently high degree of freedom.

[0011] The present technology has been made in view of such circumstances, and is intended to enable audio playback with a higher degree of freedom to be realized. [Means for solving the problem]

[0012] An audio processing device according to one aspect of the present technology includes a position information correction unit that calculates corrected position information indicating the position of the sound source relative to a standard listening position at which audio from the sound source is heard, based on position information indicating the position of the sound source relative to a standard listening position at which audio from the sound source is heard, and listening position information indicating a listening position at which audio from the sound source is heard that is different from the standard listening position; a generation unit that generates a playback signal using 3D VBAP based on a waveform signal of the sound source and the corrected position information, which reproduces audio from the sound source heard at the listening position; and a processing unit that performs convolution processing using BRIR on three or more playback signals generated by the generation unit to convert the three or more playback signals into two-channel signals, and the two-channel signals are output to headphones.

[0013] An audio processing method or program of one aspect of the present technology includes the steps of: calculating corrected position information indicating the position of the sound source relative to a standard listening position at which audio from the sound source is heard, based on position information indicating the position of the sound source relative to the listening position, and listening position information indicating a listening position at which the audio from the sound source is heard, which is different from the standard listening position; generating a playback signal that reproduces the audio from the sound source heard at the listening position using 3D VBAP, based on the waveform signal of the sound source and the corrected position information; and performing convolution processing using BRIR on the generated three or more playback signals to convert the three or more playback signals into two-channel signals, and the two-channel signals are output to headphones.

[0014] In one aspect of the present technology, based on position information indicating the position of the sound source based on a standard listening position where sound from the sound source is heard, and listening position information indicating a listening position where the sound from the sound source is heard, which is different from the standard listening position, corrected position information indicating the position of the sound source based on the listening position is calculated, a playback signal that reproduces the sound from the sound source heard at the listening position is generated using 3D VBAP based on the waveform signal of the sound source and the corrected position information, and a convolution process using BRIR is performed on the generated three or more playback signals, and the three or more playback signals are converted into two-channel signals. Also, the output destination of the two-channel signals is headphones. Effect of the Invention

[0015] According to one aspect of the present technology, audio reproduction with a higher degree of freedom can be realized.

[0016] Note that the effects described herein are not necessarily limited to those described herein, and may be any of the effects described in this disclosure. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a voice processing device. [Diagram 2] 10 is a diagram illustrating an assumed listening position and corrected position information. FIG. [Diagram 3] FIG. 13 is a diagram showing frequency characteristics when frequency characteristics are corrected. [Figure 4] FIG. 1 is a diagram illustrating VBAP. [Diagram 5] 11 is a flowchart illustrating a reproduction signal generation process. [Figure 6] FIG. 1 is a diagram illustrating a configuration of a voice processing device. [Figure 7] 11 is a flowchart illustrating a reproduction signal generation process. [Figure 8] FIG. 1 illustrates an example of the configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, embodiments to which the present technology is applied will be described with reference to the drawings.

[0019] First embodiment <Configuration example of voice processing device> This technology relates to a technology for reproducing, on the playback side, a sound to be heard at an arbitrary listening position from a waveform signal of the sound of an object that is a sound source.

[0020] FIG. 1 is a diagram showing an example of the configuration of an embodiment of a voice processing device to which the present technology is applied.

[0021] The sound processing device 11 includes an input unit 21, a position information correction unit 22, a gain / frequency characteristic correction unit 23, a spatial acoustic characteristic addition unit 24, a rendering processing unit 25, and a convolution processing unit .

[0022] The audio processing device 11 is supplied with waveform signals of a plurality of objects and metadata of these waveform signals as audio information of the content to be reproduced.

[0023] Here, the waveform signal of an object is an audio signal for reproducing the sound emitted from the object, which is the sound source.

[0024] Moreover, the metadata of the waveform signal of an object is defined as position information indicating the position of the object, i.e., the localization position of the sound of the object. This position information is information indicating the relative position of the object from a standard listening position, which is a predetermined reference point.

[0025] The object's position information may be expressed, for example, in spherical coordinates, i.e., azimuth, elevation, and radius relative to a position on a sphere centered on the standard listening position, or it may be expressed in coordinates of a Cartesian coordinate system with the standard listening position as the origin.

[0026] In the following, the position information of each object is expressed in spherical coordinates. Specifically, the position information of the nth object OB n The position information of the object OB on a sphere centered on the standard listening position n Azimuth angle A to n , elevation angle E n , and radius R n The azimuth angle A n and the elevation angle E n The unit of is, for example, degrees, and the radius R n The unit of is, for example, meters.

[0027] In addition, in the following, object OB n The location information of (A n ,E n ,R n ) will also be written as the nth object OB n Waveform signal of W n It will also be written as [t].

[0028] Therefore, for example, the waveform signal and position information of the first object OB1 are expressed as W1[t] and (A1, E1, R1), and the waveform signal and position information of the second object OB2 are expressed as W2[t] and (A2, E2, R2). In the following, for the sake of simplicity, it is assumed that the waveform signals and position information of the two objects OB1 and OB2 are supplied to the audio processing device 11.

[0029] The input unit 21 includes a mouse, a button, a touch panel, etc., and when operated by a user, outputs a signal according to the operation. For example, the input unit 21 accepts an input of an expected listening position by the user, and supplies expected listening position information indicating the expected listening position input by the user to the position information correction unit 22 and the spatial acoustic characteristic adding unit 24.

[0030] Here, the assumed listening position is the listening position of the sound that constitutes the content in the virtual sound field to be reproduced. Therefore, it can be said that the assumed listening position indicates the position after the change (correction) of the predetermined standard listening position.

[0031] The position information correction unit 22 corrects the position information of each object supplied from the outside based on the assumed listening position information supplied from the input unit 21, and supplies the corrected position information obtained as a result to the gain / frequency characteristic correction unit 23 and the rendering processing unit 25. The corrected position information is information that indicates the position of the object as viewed from the assumed listening position, that is, the localization position of the sound of the object.

[0032] The gain / frequency characteristic correction unit 23 performs gain correction and frequency characteristic correction of the waveform signal of the object supplied from the outside, based on the corrected position information supplied from the position information correction unit 22 and the position information supplied from the outside, and supplies the resulting waveform signal to the spatial acoustic characteristic addition unit 24.

[0033] The spatial acoustic characteristic adding unit 24 adds spatial acoustic characteristics to the waveform signal supplied from the gain / frequency characteristic correction unit 23 based on the assumed listening position information supplied from the input unit 21 and the position information of the object supplied from outside, and supplies the result to the rendering processing unit 25.

[0034] The renderer processing unit 25 performs mapping processing on the waveform signal supplied from the spatial acoustic characteristic adding unit 24 based on the corrected position information supplied from the position information correcting unit 22, and generates playback signals of M channels (where M is two or more). That is, playback signals of M channels are generated from the waveform signals of each object. The renderer processing unit 25 supplies the generated playback signals of M channels to the convolution processing unit 26.

[0035] The M-channel playback signal obtained in this manner is an audio signal that reproduces the sound output from each object when played back through M virtual speakers (M-channel speakers) and heard at the assumed listening position in the virtual sound field to be reproduced.

[0036] The convolution processing unit 26 performs convolution processing on the M-channel playback signals supplied from the renderer processing unit 25, and generates and outputs two-channel playback signals. That is, in this example, there are two speakers on the content playback side, and the convolution processing unit 26 generates and outputs playback signals to be played on those speakers.

[0037] <Generation of playback signals> Next, the playback signal generated by the audio processing device 11 shown in FIG. 1 will be described in more detail.

[0038] As described above, an example will be described here in which waveform signals and position information for two objects OB1 and OB2 are supplied to the audio processing device 11.

[0039] When playing back content, the user operates the input unit 21 to input an assumed listening position that will be the reference point for localizing the sound of each object during rendering.

[0040] Here, the assumed listening position is input as a horizontal movement distance X and a front-back movement distance Y from the standard listening position, and the assumed listening position information is represented as (X, Y). Note that the units of the movement distance X and the movement distance Y are meters, for example.

[0041] Specifically, in an xyz coordinate system with the standard listening position as the origin O, the horizontal direction as the x-axis and y-axis directions, and the height direction as the z-axis direction, the user inputs the distance X in the x-axis direction from the standard listening position to the assumed listening position, and the distance Y in the y-axis direction from the standard listening position to the assumed listening position. Then, information indicating the relative position from the standard listening position indicated by the input distance X and distance Y is set as assumed listening position information (X, Y). Note that the xyz coordinate system is an orthogonal coordinate system.

[0042] For simplicity, the assumed listening position is assumed to be on the xy plane, but the user may specify the height of the assumed listening position in the z-axis direction. In such a case, the user specifies the distance X in the x-axis direction, the distance Y in the y-axis direction, and the distance Z in the z-axis direction from the standard listening position to the assumed listening position, which are used as assumed listening position information (X, Y, Z). In the above description, the assumed listening position is input by the user, but the assumed listening position information may be acquired from an external source, or may be set in advance by the user, etc.

[0043] Once the assumed listening position information (X, Y) is obtained in this manner, the position information correcting unit 22 then calculates corrected position information indicating the position of each object with respect to the assumed listening position.

[0044] For example, assume that a waveform signal and position information are supplied for a given object OB11, and an assumed listening position LP11 is specified by the user, as shown in Fig. 2. Note that in Fig. 2, the horizontal direction, the depth direction, and the vertical direction indicate the x-axis direction, the y-axis direction, and the z-axis direction, respectively.

[0045] In this example, the origin O of the xyz coordinate system is set as the standard listening position. If the object OB11 is the n-th object, the position information indicating the position of the object OB11 from the standard listening position is (A n ,E n ,R n ) is said to be

[0046] That is, location information (A n ,E n ,R n ) Azimuth angle A n indicates the angle between the line connecting the origin O and the object OB11 and the y-axis on the xy plane. n ,E n ,R n ) Elevation angle E n indicates the angle between the line connecting the origin O and the object OB11 and the xy plane, and the position information (A n ,E n ,R n ) radius R n indicates the distance from the origin O to the object OB11.

[0047] Now, it is assumed that the distance X in the x-axis direction and the distance Y in the y-axis direction from the origin O to the assumed listening position LP11 have been input as assumed listening position information indicating the assumed listening position LP11.

[0048] In such a case, the position information correction unit 22 calculates the expected listening position information (X, Y) and the position information (A n ,E n ,R n ), corrected position information (A) indicating the position of the object OB11 as viewed from the assumed listening position LP11, that is, the position of the object OB11 with respect to the assumed listening position LP11 as a reference is obtained. n ',E n ',R n ') is calculated.

[0049] In addition, the correction position information (A n ',E n ',R n A in ') n ', E n ', and R n ' are the location information (A n ,E n ,R n )A n , E n , and R n 4 shows the azimuth, elevation and radius corresponding to

[0050] Specifically, for example, for the first object OB1, the position information correction unit 22 calculates corrected position information (A1', E1', R1') based on the position information (A1, E1, R1) of that object OB1 and the expected listening position information (X, Y) by calculating the following equations (1) to (3).

[0051]

number

number

number

[0052] That is, the azimuth angle A1' is calculated by the formula (1), the elevation angle E1' is calculated by the formula (2), and the radius R1' is calculated by the formula (3).

[0053] Similarly, the position information correction unit 22 calculates corrected position information (A2', E2', R2') for the second object OB2 based on the position information (A2, E2, R2) of the object OB2 and the expected listening position information (X, Y) of the object OB2, by calculating the following equations (4) to (6).

[0054]

number

number

number

[0055] That is, the azimuth angle A2' is calculated by equation (4), the elevation angle E2' is calculated by equation (5), and the radius R2' is calculated by equation (6).

[0056] Next, the gain / frequency characteristic correction unit 23 performs gain correction and frequency characteristic correction on the waveform signal of the object based on correction position information indicating the position of each object relative to the expected listening position and position information indicating the position of each object relative to the standard listening position.

[0057] For example, the gain / frequency characteristic correction unit 23 calculates the following equations (7) and (8) for objects OB1 and OB2 using the radii R1' and R2' of the corrected position information and the radii R1 and R2 of the position information, and determines the gain correction amount G1 and the gain correction amount G2 for each object.

[0058]

number

number

[0059] That is, the gain correction amount G1 of the waveform signal W1[t] of the object OB1 is calculated by equation (7), and the gain correction amount G2 of the waveform signal W2[t] of the object OB2 is calculated by equation (8). In this example, the ratio of the radius indicated by the correction position information to the radius indicated by the position information is set as the gain correction amount, and volume correction is performed according to the distance from the object to the assumed listening position using this gain correction amount.

[0060] Furthermore, the gain / frequency characteristic correction unit 23 performs frequency characteristic correction according to the radius indicated by the correction position information and gain correction according to the gain correction amount on the waveform signal of each object by calculating the following equations (9) and (10).

[0061]

number

number

[0062] That is, the calculation of equation (9) performs frequency characteristic correction and gain correction on the waveform signal W1[t] of object OB1, and waveform signal W1'[t] is obtained. Similarly, the calculation of equation (10) performs frequency characteristic correction and gain correction on the waveform signal W2[t] of object OB2, and waveform signal W2'[t] is obtained. In this example, the correction of the frequency characteristic of the waveform signal is realized by filtering.

[0063] In addition, in formula (9) and formula (10), h l (where l=0,1,…,L) is the waveform signal W n This indicates the coefficient by which [tl] (n=1,2) is multiplied.

[0064] Here, for example, if L=2 and the coefficients h0, h1, and h2 are as shown in the following equations (11) to (13), it is possible to reproduce the characteristic in which the high-frequency components of the sound from the object are attenuated by the walls and ceiling of the virtual sound field (virtual audio reproduction space) to be reproduced depending on the distance from the object to the expected listening position.

[0065]

number

number

number

[0066] In addition, in formula (12), R n is object OB n (However, n=1,2) location information (A n ,E n ,R n ) the radius R n R n ' is object OB n (However, n=1,2) Corrected position information (A n ',E n ',Rn The radius R indicated by n ' is displayed.

[0067] In this way, by calculating equations (9) and (10) using the coefficients shown in equations (11) to (13), filter processing with the frequency characteristics shown in Fig. 3 is performed. In Fig. 3, the horizontal axis represents normalized frequency, and the vertical axis represents amplitude, that is, the amount of attenuation of the waveform signal.

[0068] In Figure 3, the line C11 is R n '≦R n In this case, the distance from the object to the assumed listening position is equal to or less than the distance from the object to the standard listening position. In other words, the assumed listening position is closer to the object than the standard listening position, or the standard listening position and the assumed listening position are at the same distance from the object. Therefore, in such a case, each frequency component of the waveform signal is not particularly attenuated.

[0069] Also, curve C12 is R n '=R n This shows the frequency response when the listening position is set to +5. In this case, the assumed listening position is slightly farther from the object than the standard listening position, so the high-frequency components of the waveform signal are slightly attenuated.

[0070] Furthermore, curve C13 is R n '≧R n The figure shows the frequency response when the listening position is set to +10. In this case, the assumed listening position is farther away from the object than the standard listening position, so the high-frequency components of the waveform signal are significantly attenuated.

[0071] In this way, by performing gain correction and frequency characteristic correction according to the distance from the object to the expected listening position and attenuating the high-frequency components of the object's waveform signal, it is possible to reproduce changes in frequency characteristics and volume that occur when the user changes their listening position.

[0072] The gain / frequency characteristic correction unit 23 performs gain correction and frequency characteristic correction to generate the waveform signal W n When '[t] is obtained, the spatial acoustic characteristic adding unit 24 adds a waveform signal W n 'Spatial acoustic characteristics are added to [t]. For example, early reflections and reverberation characteristics are added to the waveform signal as spatial acoustic characteristics.

[0073] Specifically, when adding early reflections and reverberation characteristics to a waveform signal, the addition of these early reflections and reverberation characteristics can be achieved by combining multi-tap delay processing, comb filter processing, and all-pass filter processing.

[0074] In other words, the spatial acoustic characteristic adding unit 24 applies multi-tap delay processing to the waveform signal based on the delay amount and gain amount determined from the object position information and the expected listening position information, and adds the resulting signal to the original waveform signal, thereby adding an early reflection to the waveform signal.

[0075] The spatial acoustic characteristic adding unit 24 also applies comb filtering to the waveform signal based on the delay and gain amounts determined from the object position information and the assumed listening position information.The spatial acoustic characteristic adding unit 24 then applies all-pass filtering to the comb filtered waveform signal based on the delay and gain amounts determined from the object position information and the assumed listening position information, thereby obtaining a signal for adding reverberation characteristics.

[0076] Finally, the spatial acoustic characteristic adding unit 24 adds the waveform signal to which the early reflections have been added and a signal for adding reverberation characteristics to obtain a waveform signal to which the early reflections and reverberation characteristics have been added, and outputs the resulting signal to the rendering processing unit 25.

[0077] In this way, by adding spatial acoustic characteristics to the waveform signal using parameters determined for the object position information and the assumed listening position information, it is possible to reproduce changes in spatial acoustics that accompany changes in the user's listening position.

[0078] In addition, parameters such as delay amounts and gain amounts used in these multi-tap delay processes, comb filter processes, all-pass filter processes, etc. may be stored in advance in a table for each combination of object position information and expected listening position information.

[0079] In such a case, for example, the spatial acoustic characteristic adding unit 24 holds in advance a table in which a parameter set such as a delay amount is associated with each position indicated by the position information for each assumed listening position.The spatial acoustic characteristic adding unit 24 then reads out a parameter set determined from the object position information and the assumed listening position information from the table, and adds the spatial acoustic characteristic to the waveform signal using those parameters.

[0080] The parameter set used to add the spatial acoustic characteristics may be stored as a table, or may be stored as a function, etc. For example, when the parameters are calculated by a function, the spatial acoustic characteristics adding unit 24 substitutes the position information and the assumed listening position information into a function stored in advance, and calculates each parameter used to add the spatial acoustic characteristics.

[0081] When the waveform signals to which the spatial acoustic characteristics are added are obtained for each object in the above manner, the rendering processing unit 25 performs mapping processing for those waveform signals to each of the M channels, and generates playback signals for M channels. In other words, rendering is performed.

[0082] Specifically, for example, the renderer 25 determines the gain amount of the waveform signal of the object for each of the M channels by VBAP based on the correction position information for each object. Then, the renderer 25 performs a process of adding up the waveform signals of each object multiplied by the gain amount determined by VBAP for each channel, thereby generating a playback signal for each channel.

[0083] Now, VBAP will be described with reference to FIG.

[0084] For example, assume that a user U11 is listening to three-channel sounds output from three speakers SP1 to SP3 as shown in Fig. 4. In this example, the position of the head of the user U11 is a position LP21 that corresponds to the assumed listening position.

[0085] Moreover, a triangle TR11 on the spherical surface surrounded by the speakers SP1 to SP3 is called a mesh, and in VBAP, a sound image can be localized at any position within this mesh.

[0086] Now, let us consider localizing a sound image at a sound image position VSP1 using information indicating the positions of three speakers SP1 to SP3 that output the sound of each channel. Here, the sound image position VSP1 is located at one object OB n More specifically, the position of the corrected position information (A n ',E n ',R n Object OB indicated by ') n corresponds to the position of

[0087] For example, in a three-dimensional coordinate system with the position of the head of the user U11, that is, the position LP21 as the origin, the sound image position VSP1 is represented by a three-dimensional vector p with the position LP21 (origin) as the starting point.

[0088] Furthermore, if the three-dimensional vectors starting from position LP21 (origin) and pointing in the direction of the positions of each speaker SP1 to SP3 are vectors l1 to l3, then vector p can be expressed as a linear sum of vectors l1 to l3, as shown in the following equation (14).

[0089]

number

[0090] By calculating the coefficients g1 to g3 by which vectors l1 to l3 are multiplied in equation (14) and setting these coefficients g1 to g3 as the gain amounts of the sounds output from speakers SP1 to SP3, respectively, i.e., the gain amounts of the waveform signals, the sound image can be localized at sound image position VSP1.

[0091] Specifically, the inverse matrix L of a triangular mesh consisting of three speakers SP1 to SP3 is 123 -1 and object OB n By calculating the following equation (15) based on vector p indicating the position of the vector g1, coefficients g1 to g3 that are the gain amounts can be obtained.

[0092]

number

[0093] In addition, in equation (15), R, which is an element of vector p, n 'sinA n ' cosE n ', R n 'cosA n ' cosE n ', and R n 'sinE n ' is the sound image position VSP1, i.e., the object OB n The figures show the x', y', and z' coordinates on the x'y'z' coordinate system, indicating the position of

[0094] This x'y'z' coordinate system is an orthogonal coordinate system in which the x', y', and z' axes are parallel to the x, y, and z axes of the xyz coordinate system shown in Figure 2, and the origin is a position corresponding to the assumed listening position. n Corrected position information (A n ',E n ',R n ') can be found.

[0095] Also, in equation (15), l11 , l 12 , and l 13 are the values ​​of the x', y', and z' components when vector l1 pointing to the first speaker that makes up the mesh is decomposed into x'-, y'-, and z'-axis components, and correspond to the x', y', and z' coordinates of the first speaker.

[0096] Similarly, 21 , l 22 , and l 23 are the values ​​of the x', y', and z' components when the vector l2 toward the second speaker that constitutes the mesh is decomposed into the x', y', and z' components. Also, l 31 , l 32 , and l 33 are the values ​​of the x', y', and z' components when vector l3 directed toward the third speaker that constitutes the mesh is decomposed into x'-axis, y'-axis, and z'-axis components.

[0097] In this way, the method of determining the coefficients g1 to g3 using the positional relationship of the three speakers SP1 to SP3 and controlling the localization position of the sound image is particularly called three-dimensional VBAP. In this case, the number of channels M of the playback signal is 3 or more.

[0098] In addition, since the rendering processing unit 25 generates M-channel playback signals, the number of virtual speakers corresponding to each channel is M. In this case, each object OB n The gain amount of the waveform signal is calculated for each of the M channels corresponding to the M speakers.

[0099] In this example, multiple meshes consisting of M virtual speakers are arranged in a virtual audio playback space. nThe gain amounts of the three channels corresponding to the three speakers constituting the mesh containing are set to values ​​calculated by the above-mentioned formula (15). On the other hand, the gain amounts of the M-3 channels corresponding to the remaining M-3 speakers are set to 0.

[0100] After generating the M-channel reproduction signals in the above manner, the rendering processor 25 supplies the obtained reproduction signals to the convolution processor 26.

[0101] The M-channel playback signal obtained in this manner can more realistically reproduce how the sound of each object is heard at a desired assumed listening position. Note that, although an example of generating the M-channel playback signal by VBAP has been described here, the M-channel playback signal may be generated by any other method.

[0102] The M-channel playback signal is a signal for playing back audio through an M-channel speaker system, and in the audio processing device 11, this M-channel playback signal is further converted into a 2-channel playback signal and output. In other words, the M-channel playback signal is downmixed into a 2-channel playback signal.

[0103] For example, the convolution processing unit 26 performs BRIR (Binaural Room Impulse Response) processing as a convolution process on the M-channel playback signal supplied from the rendering processing unit 25, thereby generating and outputting a two-channel playback signal.

[0104] The convolution process for the reproduced signal is not limited to the BRIR process, but may be any process capable of obtaining a two-channel reproduced signal.

[0105] In addition, if the output destination of the two-channel playback signal is headphones, it is possible to prepare a table of impulse responses from various object positions to the expected listening position in advance. In such a case, by synthesizing the waveform signals of each object through BRIR processing using the impulse responses corresponding to the expected listening position from the object position, it is possible to reproduce how the sound output from each object sounds at the desired expected listening position.

[0106] However, this method requires impulse responses corresponding to a large number of points (positions), and as the number of objects increases, the number of BRIR processes must be increased accordingly, resulting in a large processing load.

[0107] Therefore, in the audio processing device 11, the playback signal (waveform signal) mapped to the virtual M-channel speakers by the rendering processing unit 25 is downmixed to a two-channel playback signal by BRIR processing using the impulse responses from the virtual M-channel speakers to both ears of the user (listener). In this case, it is only necessary to have the impulse responses from each of the M-channel speakers to both ears of the listener, and even when there are many objects, the BRIR processing is performed for M channels, so the processing load can be reduced.

[0108] <Description of Reproduction Signal Generation Process> Next, a description will be given of the flow of the above-described processing by the audio processing device 11. That is, hereinafter, a playback signal generation process by the audio processing device 11 will be described with reference to the flowchart of FIG.

[0109] In step S11, the input unit 21 accepts an input of an assumed listening position. When a user operates the input unit 21 to input an assumed listening position, the input unit 21 supplies assumed listening position information indicating the assumed listening position to the position information correction unit 22 and the spatial acoustic characteristic adding unit 24.

[0110] In step S12, the position information correction unit 22 generates corrected position information (A n ',E n ',R n The gain / frequency characteristic correction unit 23 calculates the corrected position information of each object, and supplies the calculated position information to the gain / frequency characteristic correction unit 23 and the rendering unit 25. For example, the above-mentioned formulas (1) to (3) and formulas (4) to (6) are calculated, and the corrected position information of each object is calculated.

[0111] In step S13, the gain / frequency characteristic correction unit 23 performs gain correction and frequency characteristic correction of the waveform signal of the object supplied from the outside, based on the corrected position information supplied from the position information correction unit 22 and the position information supplied from the outside.

[0112] For example, the above-mentioned formula (9) and formula (10) are calculated to obtain the waveform signal W n The gain / frequency characteristic correction unit 23 calculates the waveform signal W′[t] of each object. n '[t] is supplied to the spatial acoustic characteristic adding unit 24.

[0113] In step S14, the spatial acoustic characteristic adding unit 24 adds spatial acoustic characteristics to the waveform signal supplied from the gain / frequency characteristic correcting unit 23 based on the assumed listening position information supplied from the input unit 21 and the position information of the object supplied from the outside, and supplies the waveform signal to the rendering processing unit 25. For example, early reflections, reverberation characteristics, etc. are added to the waveform signal as spatial acoustic characteristics.

[0114] In step S15, the rendering processing unit 25 generates M channel playback signals by performing mapping processing on the waveform signal supplied from the spatial acoustic characteristic adding unit 24 based on the corrected position information supplied from the position information correcting unit 22, and supplies the M channel playback signals to the convolution processing unit 26. For example, in the processing of step S15, the playback signal is generated by VBAP, but the M channel playback signals may be generated by any other method.

[0115] In step S16, the convolution processing unit 26 generates and outputs a two-channel reproduction signal by performing a convolution process on the M-channel reproduction signal supplied from the rendering processing unit 25. For example, the above-mentioned BRIR process is performed as the convolution process.

[0116] When the two-channel playback signals are generated and output, the playback signal generation process ends.

[0117] In this manner, the audio processing device 11 calculates correction position information based on the expected listening position information, and performs gain correction and frequency characteristic correction on the waveform signal of each object, or adds spatial acoustic characteristics, based on the obtained correction position information and the expected listening position information.

[0118] This makes it possible to realistically reproduce how the sound output from each object position is heard at any assumed listening position. This allows users to freely specify the listening position for the sound according to their preferences when playing back content, realizing audio playback with a higher degree of freedom.

[0119] Second embodiment <Configuration example of voice processing device> In the above, an example has been described in which the user can specify any assumed listening position, but it is also possible to change (modify) not only the listening position but also the position of each object to any position.

[0120] In such a case, the voice processing device 11 is configured, for example, as shown in Fig. 6. In Fig. 6, the parts corresponding to those in Fig. 1 are given the same reference numerals, and the description thereof will be omitted as appropriate.

[0121] The audio processing device 11 shown in FIG. 6 includes an input unit 21, a position information correction unit 22, a gain / frequency characteristic correction unit 23, a spatial acoustic characteristic adding unit 24, a rendering processing unit 25, and a convolution processing unit 26, similar to the case in FIG.

[0122] 6, the user operates the input unit 21 to input, in addition to the assumed listening position, a correction position indicating the corrected (changed) position of each object. The input unit 21 supplies the correction position information indicating the correction position of each object input by the user to the position information correction unit 22 and the spatial acoustic characteristic adding unit 24.

[0123] For example, the corrected position information is the corrected object OB from the standard listening position, just like the position information. n Azimuth angle A n , elevation angle E n , and radius R n The modified position information may be information indicating the relative position of the modified (changed) object with respect to the position of the object before the modification (change).

[0124] Furthermore, the position information correction unit 22 calculates corrected position information based on the assumed listening position information and corrected position information supplied from the input unit 21, and supplies the corrected position information to the gain / frequency characteristic correction unit 23 and the rendering processing unit 25. Note that, for example, when the corrected position information is information indicating a relative position seen from the original object position, the corrected position information is calculated based on the assumed listening position information, position information, and corrected position information.

[0125] The spatial acoustic characteristic adding unit 24 adds spatial acoustic characteristics to the waveform signal supplied from the gain / frequency characteristic correction unit 23 based on the assumed listening position information and corrected position information supplied from the input unit 21, and supplies the result to the rendering processing unit 25.

[0126] For example, it has been explained that the spatial acoustic characteristic adding unit 24 of the sound processing device 11 shown in FIG. 1 holds in advance a table in which a parameter set is associated with each position indicated by the position information for each piece of assumed listening position information.

[0127] In response to this, the spatial acoustic characteristic adding unit 24 of the audio processing device 11 shown in Fig. 6 holds in advance a table in which, for example, for each assumed listening position information, a parameter set is associated with each position indicated by the corrected position information. Then, for each object, the spatial acoustic characteristic adding unit 24 reads out from the table a parameter set determined from the assumed listening position information and corrected position information supplied from the input unit 21, and performs multi-tap delay processing, comb filter processing, all-pass filter processing, etc. using these parameters to add spatial acoustic characteristics to the waveform signal.

[0128] <Description of Reproduction Signal Generation Process> Next, the playback signal generation process by the audio processing device 11 shown in Fig. 6 will be described with reference to the flowchart of Fig. 7. Note that the process of step S41 is similar to the process of step S11 in Fig. 5, and therefore the description thereof will be omitted.

[0129] In step S42, the input unit 21 accepts input of the correction position of each object. When the user operates the input unit 21 to input the correction position for each object, the input unit 21 supplies correction position information indicating the correction positions to the position information correction unit 22 and the spatial acoustic characteristic adding unit 24.

[0130] In step S43, the position information correction unit 22 calculates corrected position information (A n ',E n ',R n ') and supplies it to the gain / frequency characteristic correction unit 23 and the rendering processing unit 25.

[0131] In this case, for example, in the above-mentioned formulas (1) to (3), the azimuth angle, elevation angle, and radius of the position information are replaced with the azimuth angle, elevation angle, and radius of the corrected position information, and the corrected position information is calculated. Also, in the formulas (4) to (6), the position information is replaced with the corrected position information, and the calculation is performed.

[0132] After the corrected position information is calculated, the process of step S44 is performed. However, since the process of step S44 is similar to the process of step S13 in FIG. 5, the description thereof will be omitted.

[0133] In step S45, the spatial acoustic characteristic adding unit 24 adds spatial acoustic characteristics to the waveform signal supplied from the gain / frequency characteristic correction unit 23 based on the assumed listening position information and corrected position information supplied from the input unit 21, and supplies the waveform signal to the rendering processing unit 25.

[0134] After the spatial acoustic characteristics are added to the waveform signal, the processes of steps S46 and S47 are performed and the playback signal generation process is terminated. However, since these processes are similar to the processes of steps S15 and S16 in FIG. 5, their description will be omitted.

[0135] In this manner, the audio processing device 11 calculates correction position information based on the expected listening position information and the corrected position information, and performs gain correction and frequency characteristic correction on the waveform signal of each object, or adds spatial acoustic characteristics, based on the obtained correction position information, expected listening position information, and corrected position information.

[0136] This makes it possible to realistically reproduce how sound output from an arbitrary object position is heard at an arbitrary assumed listening position. Therefore, not only can users freely specify the listening position of the sound according to their preferences when playing back content, but they can also freely specify the position of each object, realizing audio playback with a higher degree of freedom.

[0137] For example, the audio processing device 11 can reproduce how sounds sound when the user changes the configuration and arrangement of singing voices, musical instrument sounds, etc. Therefore, the user can freely move the configuration and arrangement of musical instruments, singing voices, etc. corresponding to objects, and enjoy music and sounds with sound source configuration and arrangement that suit the user's taste.

[0138] Also, in the audio processing device 11 shown in FIG. 6, as in the case of the audio processing device 11 shown in FIG. 1, the processing load can be reduced by first generating an M-channel playback signal and then converting (downmixing) the playback signal into a 2-channel playback signal.

[0139] The above-mentioned series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed in a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose computer, for example, capable of executing various functions by installing various programs.

[0140] FIG. 8 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes according to a program.

[0141] In the computer, a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, and a RAM (Random Access Memory) 503 are interconnected via a bus 504.

[0142] An input / output interface 505 is further connected to the bus 504. An input unit 506, an output unit 507, a recording unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.

[0143] The input unit 506 includes a keyboard, a mouse, a microphone, an image sensor, etc. The output unit 507 includes a display, a speaker, etc. The recording unit 508 includes a hard disk, a non-volatile memory, etc. The communication unit 509 includes a network interface, etc. The drive 510 drives removable media 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0144] In a computer configured as described above, the CPU 501 loads, for example, a program recorded in the recording unit 508 into the RAM 503 via the input / output interface 505 and the bus 504, and executes the program, thereby performing the series of processes described above.

[0145] The program executed by the computer (CPU 501) can be provided by being recorded on a removable medium 511 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0146] In the computer, the program can be installed in the recording unit 508 via the input / output interface 505 by mounting the removable medium 511 in the drive 510. The program can also be received by the communication unit 509 via a wired or wireless transmission medium and installed in the recording unit 508. Alternatively, the program can be installed in the ROM 502 or the recording unit 508 in advance.

[0147] In addition, the program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or it may be a program in which processing is performed in parallel or at the required timing, such as when called.

[0148] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0149] For example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0150] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.

[0151] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices.

[0152] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0153] Furthermore, the present technology can also be configured as follows.

[0154] (1) a position information correction unit that calculates corrected position information indicating the position of the sound source based on position information indicating the position of the sound source and listening position information indicating a listening position at which a sound from the sound source is listened to; a generator that generates a reproduction signal that reproduces a sound from the sound source heard at the listening position based on a waveform signal of the sound source and the corrected position information; An audio processing device comprising: (2) The position information correction unit calculates the corrected position information based on corrected position information indicating the position of the sound source after the correction and the listening position information. An audio processing device as described in (1). (3) The apparatus further includes a correction unit that performs at least one of gain correction and frequency characteristic correction on the waveform signal according to the distance from the sound source to the listening position. An audio processing device according to (1) or (2). (4) The apparatus further includes a spatial acoustic characteristic adding unit that adds a spatial acoustic characteristic to the waveform signal based on the listening position information and the correction position information. An audio processing device as described in (2). (5) The spatial acoustic characteristic adding unit adds at least one of an early reflection characteristic and a reverberation characteristic to the waveform signal as the spatial acoustic characteristic. The audio processing device according to (4). (6) The apparatus further includes a spatial acoustic characteristic adding unit that adds a spatial acoustic characteristic to the waveform signal based on the listening position information and the position information. An audio processing device as described in (1). (7) a convolution processing unit that performs convolution processing on the reproduction signals of two or more channels generated by the generation unit to generate the reproduction signals of two channels. A voice processing device according to any one of (1) to (6). (8) Calculating corrected position information indicating the position of the sound source based on position information indicating the position of a sound source and listening position information indicating a listening position at which a sound from the sound source is listened to; A reproduction signal is generated based on the waveform signal of the sound source and the corrected position information, the reproduction signal reproducing the sound from the sound source heard at the listening position. 1. An audio processing method comprising the steps of: (9) Calculating corrected position information indicating the position of the sound source based on position information indicating the position of a sound source and listening position information indicating a listening position at which a sound from the sound source is listened to; A reproduction signal is generated based on the waveform signal of the sound source and the corrected position information, the reproduction signal reproducing the sound from the sound source heard at the listening position. A program that causes a computer to execute a process including steps. [Explanation of symbols]

[0155] 11 audio processing device, 21 input section, 22 position information correction section, 23 gain / frequency characteristic correction section, 24 spatial acoustic characteristic addition section, 25 rendering processing section, 26 convolution processing section

Claims

1. an acquisition unit for acquiring data of an audio object and metadata of the audio object; a position information correction unit that calculates corrected position information indicating a position of the audio object based on a standard listening position for listening to the sound of the audio object and listening position information indicating a listening position for listening to the sound of the audio object, the listening position being different from the standard listening position; and a generation unit that generates a playback signal that reproduces the sound from the audio object listened to at the listening position based on a waveform signal of the audio object and the correction position information, using VBAP; a processing unit that performs a convolution process using a BRIR on the three or more reproduction signals generated by the generation unit to convert the three or more reproduction signals into two-channel signals; An audio processing device comprising:

2. The audio processing device Obtaining audio object data and metadata for said audio object; Calculating corrected position information indicating the position of the audio object based on a standard listening position for listening to the sound of the audio object and listening position information indicating a listening position for listening to the sound of the audio object, the listening position being different from the standard listening position; generating a playback signal using VBAP, the playback signal reproducing the sound from the audio object listened to at the listening position, based on the waveform signal of the audio object and the correction position information; The three or more generated reproduction signals are subjected to a convolution process using BRIR, and the three or more reproduction signals are converted into two-channel signals. Audio processing methods.

3. Obtaining audio object data and metadata for said audio object; Calculating corrected position information indicating the position of the audio object based on a standard listening position for listening to the sound of the audio object and listening position information indicating a listening position for listening to the sound of the audio object, the listening position being different from the standard listening position; generating a playback signal using VBAP, the playback signal reproducing the sound from the audio object listened to at the listening position, based on the waveform signal of the audio object and the correction position information; The three or more generated reproduction signals are subjected to a convolution process using BRIR, and the three or more reproduction signals are converted into two-channel signals. A program that causes a computer to execute a process including steps.

Images