Signal processing device, method, and program

Abstract

The present technology relates to a signal processing device and method, and a program, which enable a greater sense of realism to be obtained. This signal processing device is provided with: an acquiring unit for acquiring metadata including position information indicating the position of an audio object, and azimuth information indicating the orientation of the audio object, and audio data relating to the audio object; and a signal generating unit for generating a reproduction signal for reproducing the sound of the audio object at a listening position on the basis of listening position information indicating the listening position, listener azimuth information indicating the orientation of a listener at the listening position, position information, azimuth information, and the audio data. The present technology can be applied to transmission / reproduction systems.

Description

[Technical Field]

[0001] The present technology relates to a signal processing device, method, and program, and in particular to a signal processing device, method, and program that enable a higher sense of realism to be obtained. [Background technology]

[0002] For example, when reproducing sound fields from any viewpoint, such as bird's-eye views or walk-throughs, it is important to record target sounds, such as human voices, the sounds of athletes in action, such as the kicking of a ball in sports, or the sounds of musical instruments, with as high a signal-to-noise ratio as possible.

[0003] At the same time, it is necessary to reproduce the sound with accurate localization for each target sound source, and to follow the sound image localization that accompanies the movement of the viewpoint or sound source.

[0004] Meanwhile, in the case of free viewpoint and fixed viewpoint content, there is a demand for technology that allows a higher sense of realism to be achieved, and many such technologies have been proposed.

[0005] For example, as a technology related to free viewpoint sound field reproduction, a technology has been proposed in which, when a user can freely specify a listening position, gain correction and frequency characteristic correction are performed according to the distance from the changed listening position to an audio object (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2015 / 107926 Summary of the Invention [Problem to be solved by the invention]

[0007] However, there are cases where the above-mentioned techniques are unable to provide a sufficiently high sense of realism.

[0008] For example, in the real world, sound sources are not point sources, but sound waves propagate from a sound-producing object with a certain size, with specific directional characteristics including reflection and diffraction by that sound-producing object.

[0009] However, although there are currently many attempts to record the sound field of a desired space, even when recording is done for each sound source, i.e., for each audio object, the orientation of each audio object is not taken into account on the playback side, and it may not be possible to achieve a sufficiently high level of realism.

[0010] The present technology has been made in view of such circumstances, and is intended to enable a higher sense of realism to be achieved. [Means for solving the problem]

[0011] A signal processing device according to one aspect of the present technology includes an acquisition unit that acquires metadata including position information indicating the position of an audio object and orientation information indicating the direction of the audio object, as well as audio data of the audio object, and a signal generation unit that generates a playback signal for reproducing the sound of the audio object at the listening position based on listening position information indicating the listening position, listener orientation information indicating the direction of the listener at the listening position, the position information, the orientation information, and the audio data.

[0012] A signal processing method or program according to one aspect of the present technology includes a step of acquiring metadata including position information indicating the position of an audio object and orientation information indicating the direction of the audio object, as well as audio data of the audio object, and generating a playback signal that reproduces the sound of the audio object at the listening position based on listening position information indicating the listening position, listener orientation information indicating the direction of the listener at the listening position, the position information, the orientation information, and the audio data.

[0013] In one aspect of the present technology, metadata including position information indicating the position of an audio object and orientation information indicating the direction of the audio object, and audio data of the audio object are acquired, and a playback signal that reproduces the sound of the audio object at the listening position is generated based on listening position information indicating the listening position, listener orientation information indicating the direction of the listener at the listening position, the position information, the orientation information, and the audio data. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 10 is a diagram illustrating the orientation of objects that make up content. [Figure 2] FIG. 10 is a diagram illustrating the directional characteristics of an object. [Figure 3] FIG. 10 is a diagram illustrating an example of metadata syntax. [Figure 4] FIG. 10 is a diagram illustrating an example of the syntax of directional pattern data. [Figure 5] FIG. 1 illustrates an example of the configuration of a signal processing device. [Figure 6] FIG. 10 is a diagram illustrating relative direction information. [Figure 7] FIG. 10 is a diagram illustrating relative direction information. [Figure 8] FIG. 10 is a diagram illustrating relative direction information. [Figure 9] FIG. 10 is a diagram illustrating relative direction information. [Figure 10] 10 is a flowchart illustrating a content playback process. [Figure 11] FIG. 1 illustrates an example of the configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments to which the present technology is applied will be described with reference to the drawings.

[0016] First Embodiment About this technology This technology relates to a transmission and playback system that appropriately transmits directional pattern data indicating the directional pattern of an audio object, which is a sound source, and reflects the directional pattern of the audio object in the content playback on the content playback side based on the directional pattern data, thereby achieving a higher sense of realism.

[0017] For example, content that reproduces the sound of an audio object (hereinafter also simply referred to as an object) that is a sound source includes fixed viewpoint content and free viewpoint content.

[0018] In fixed viewpoint content, the position of the listener's viewpoint, i.e., the listening position (listening point), is a predetermined fixed position, whereas in free viewpoint content, the listener can freely specify the listening position (viewpoint position) in real time.

[0019] In the real world, each sound source has its own directional characteristics. That is, even if a sound is emitted from the same sound source, the sound transmission characteristics will be different depending on the direction seen from the sound source.

[0020] Therefore, when an object that serves as a sound source in content or a listener at a listening position moves or rotates freely, the way the listener hears the sound of the object changes depending on the directional characteristics of the object.

[0021] In content playback, processing is generally performed to reproduce distance attenuation according to the distance from the listening position to the object.In contrast, this technology plays back content taking into account not only distance attenuation but also the directional characteristics of the object, thereby achieving a higher sense of realism.

[0022] In other words, with this technology, when a listener or object moves or rotates freely, not only the distance between the listener and the object but also the relative direction (orientation) between the listener and the object is taken into consideration, and transmission characteristics according to the distance attenuation and directional characteristics are dynamically added to the content sound for each object.

[0023] For example, the addition of transfer characteristics is realized by gain correction according to distance attenuation and directional characteristics, and processing for wavefront synthesis based on the propagation characteristics of the amplitude and phase of the wavefront taking distance attenuation and directional characteristics into consideration.

[0024] In this technology, directional pattern data is used to add transfer characteristics according to the directional pattern, but if directional pattern data corresponding to each type of target sound source, i.e., each type of object, is prepared, an even greater sense of realism can be achieved.

[0025] For example, directional characteristic data for each type of object can be obtained by recording sound in advance using a microphone array or performing a simulation to determine the transmission characteristics for each direction and distance as the sound emitted from the object propagates through space.

[0026] The directional characteristic data for each type of object is transmitted in advance to the playback device together with the audio data of the content or separately from the audio data.

[0027] When the content is played back, the playback device uses the directional characteristic data to add a transfer characteristic according to the distance to the object and the directional characteristic to the audio data of the object, i.e., the playback signal for playing the sound of the content.

[0028] This makes it possible to play back content with a greater sense of realism.

[0029] With this technology, for each type of sound source (object), transfer characteristics are added according to the relative positional relationship between the listener and the object, i.e., the relative distance and direction. Therefore, even if the distance from the object to the listening position is the same, the way the sound of the object is heard changes depending on the direction from which the sound is heard, making it possible to reproduce a sound field that is closer to reality.

[0030] Examples of content suitable for application of this technology include the following:

[0031] -Content that recreates fields where team sports are played Content that recreates a space with multiple performers, such as in musicals, operas, and plays Content that recreates any space at a live venue or theme park -Content that plays performances by orchestras, marching bands, etc. Games and other content

[0032] For example, in content of a performance by a marching band, the performers may be stationary or may be moving.

[0033] Now, the present technology will be described in further detail below.

[0034] For example, consider an example of content that reproduces a sound field with an arbitrary position on a soccer field as the listening position.

[0035] In this case, for example, as shown in Figure 1, there are players and a referee on the field for each team, and these players and referees are the sound sources, or audio objects.

[0036] In the example shown in Figure 1, each circle in the figure represents a player or a referee, i.e., an object, and the direction of the line segment attached to each circle represents the direction in which the player or referee represented by that circle is facing, i.e., the orientation of the object such as the player or referee.

[0037] Here, each object is at a different position and faces a different direction, and the positions and orientations of these objects change over time, i.e., each object moves and rotates over time.

[0038] For example, object OB11 is a judge, and one example would be to present to the listener as content the video and audio when the position of this object OB11 is the viewing point position (listening position), and the upward direction in the figure, which is the direction of object OB11, is the line of sight.

[0039] In the example in Figure 1, each object is placed on a two-dimensional plane, but in reality, the height of each object's mouth (players and referees), and the height of their feet (where the ball kick sound originates) are different from each other, and the posture of the objects is constantly changing.

[0040] That is, in reality, each object and viewing point (listening position) is placed in a three-dimensional space, and at the same time, these objects and listeners (users) at the viewing points are in various postures and facing in various directions.

[0041] The cases in which directional characteristics according to the object orientation can be reflected in the content can be classified as follows:

[0042] (Case 1) When objects and listening positions are placed on a two-dimensional plane, and only the azimuth angle (yaw) that indicates the direction of the object is taken into account, and the elevation angle (pitch) and tilt angle (roll) are not taken into account. (Case 2) When objects and listening positions are placed in a three-dimensional space, and the azimuth and elevation angles that indicate the object's orientation are taken into account, but the tilt angle that indicates the object's rotation is not taken into account. (Case 3) When objects and listening positions are placed in a three-dimensional space, and Euler angles, which are the azimuth and elevation angles that indicate the object's orientation and the tilt angle that indicates the object's rotation, are taken into consideration.

[0043] This technology can be applied to any of the above cases 1 to 3, and in each of these cases, content is played back taking into account the listening position, object placement, object orientation and rotation (tilt), i.e., rotation angle, as appropriate.

[0044] <About the transmitting device> A transmission and playback system for transmitting and playing back such content may include, for example, a transmitting device that transmits content data and a signal processing device that functions as a playback device that plays back the content based on the content data transmitted from the transmitting device. Note that there may be one or more signal processing devices that function as playback devices.

[0045] A transmitting device on the transmitting side of the transmission and playback system transmits, as content data, for example, audio data for reproducing the sound of one or more objects that make up the content, and metadata of each object (audio data).

[0046] Here, the metadata includes sound source type information, sound source position information, and sound source direction information.

[0047] The sound source type information is ID information that indicates the type of the object that is the sound source.

[0048] For example, the sound source type information may be information specific to the sound source that indicates the type (kind) of the object itself that is the sound source, such as a player or an instrument, or it may be information that indicates the type of sound emitted from the object, such as the voice of a player, the sound of a ball kick, the sound of clapping, or other action sounds.

[0049] Alternatively, the sound source type information may be information indicating the type of the object itself and the type of sound emitted from the object.

[0050] Furthermore, directional characteristic data is prepared for each type indicated by the sound source type information, and on the playback side, a playback signal is generated based on the directional characteristic data determined for the sound source type information, so the sound source type information can also be said to be ID information that indicates the directional characteristic data.

[0051] In the transmitting device, sound source type information is manually or otherwise assigned to each object that constitutes the content, and is included in the metadata of the object.

[0052] The sound source position information included in the metadata is information that indicates the position of the object that is the sound source.

[0053] Here, the sound source location information may be, for example, latitude and longitude indicating an absolute position on the Earth's surface measured (acquired) by a position measurement module such as a GPS (Global Positioning System) module, or coordinates obtained by converting these latitude and longitude into distance.

[0054] Alternatively, the sound source position information may be any information indicating the position of an object, such as coordinates of a coordinate system that uses a predetermined position within the space (target area) in which the content is to be recorded as the reference position.

[0055] Furthermore, when the sound source position information is expressed as coordinates (coordinate information), the coordinates may be coordinates of any coordinate system, such as a polar coordinate system consisting of an azimuth angle, an elevation angle, and a radius, an xyz coordinate system, i.e., a three-dimensional Cartesian coordinate system, or a two-dimensional Cartesian coordinate system.

[0056] Furthermore, the sound source direction information included in the metadata is information that indicates the absolute direction in which an object located at the position indicated by the sound source position information is facing, that is, the direction in front of the object.

[0057] In addition, the sound source direction information may include not only information indicating the orientation of an object but also information indicating the rotation (tilt) of the object, and in the following, the sound source direction information is assumed to include information indicating the orientation of an object and information indicating the rotation of the object.

[0058] Specifically, for example, the sound source direction information includes an azimuth angle ψ indicating the direction of an object in a coordinate system of coordinates as sound source position information. o and elevation angle θ o and the tilt angle φ, which indicates the rotation (tilt) of the object in the coordinate system of the coordinates as the sound source position information. o It includes:

[0059] In other words, the sound source direction information is the azimuth angle ψ that indicates the absolute orientation and rotation of the object. o (yaw), elevation angle θ o (pitch), and the tilt angle φ o For example, sound source direction information can be obtained from a geomagnetic sensor attached to an object or from video data of the object as a subject.

[0060] In the transmitting device, sound source position information and sound source direction information are generated for each object for each discretized unit time such as for each frame of audio data or for each predetermined number of frames, that is, at predetermined time intervals.

[0061] Then, metadata including sound source type information, sound source position information, and sound source direction information is sent (transmitted) to the signal processing device together with the audio data of the object for each unit time, such as for each frame.

[0062] Furthermore, the transmitting device transmits (transmits) the directional pattern data to the signal processing device on the playback side in advance or sequentially for each sound source type indicated by the sound source type information. Note that the signal processing device may obtain the directional pattern data from a device other than the transmitting device.

[0063] The directional characteristic data is data indicating the directional characteristic of an object of the sound source type indicated by the sound source type information, that is, the transfer characteristic in each direction as seen from the object.

[0064] For example, as shown in FIG. 2, each sound source has its own directional characteristic.

[0065] In the example shown in FIG. 2, for example, a whistle as a sound source has a directional characteristic in which sound propagates strongly in the front (forward) direction as indicated by arrow Q11, that is, a sharp front directivity.

[0066] Furthermore, for example, footsteps emitted from spikes or the like as a sound source have directional characteristics (omnidirectional) in which the sound propagates in all directions with the same strength, as indicated by arrow Q12.

[0067] Furthermore, for example, the sound emitted from the mouth of a player as a sound source has directional characteristics in which sound propagates strongly in the front and sides as shown by arrow Q13, that is, has a fairly strong front directivity.

[0068] Such directional pattern data showing the directional characteristics of a sound source can be obtained by, for example, acquiring the characteristics of sound propagation (transmission characteristics) to the surroundings for each type of sound source using a microphone array in an anechoic chamber, etc. Alternatively, directional pattern data can be obtained by performing a simulation on 3D data that mimics the shape of the sound source.

[0069] Specifically, the directional characteristic data is a gain function dir(i, ψ, θ) that is defined as a function of the azimuth angle ψ and elevation angle θ that indicate the direction as seen from the sound source, and is determined for an ID value i that indicates the type of sound source.

[0070] Furthermore, in addition to the azimuth angle ψ and the elevation angle θ, a gain function dir(i, d, ψ, θ) having a discretized distance d from the sound source as an argument may be used as the directivity data.

[0071] In this case, by substituting each argument into the gain function dir(i,d,ψ,θ), a gain value indicating the transfer characteristics (propagation characteristics) of the sound is obtained as the output of the gain function dir(i,d,ψ,θ).

[0072] This gain value indicates the characteristics (transmission characteristics) of sound that is emitted from a sound source of a sound source type with an ID value of i, propagates in the direction of azimuth angle ψ and elevation angle θ as viewed from the sound source, and arrives at a position at a distance d from the sound source (hereinafter referred to as position P).

[0073] Therefore, by performing gain correction on the audio data of the sound source type with an ID value of i based on this gain value, it is possible to reproduce (reproduce) the sound from the sound source of the sound source type with an ID value of i that would actually be heard at position P.

[0074] In particular, in this example, by using the gain value that is the output of the gain function dir(i, d, ψ, θ), it is possible to realize gain correction that adds the transfer characteristics indicated by the directional characteristics that also take into account the distance from the sound source, i.e., distance attenuation.

[0075] The directional characteristic data may be a gain function that indicates a transfer characteristic that takes into account reverberation characteristics, etc. Alternatively, the directional characteristic data may be Ambisonics format data, i.e., data consisting of spherical harmonic coefficients (spherical harmonic spectra) for each direction.

[0076] The transmitting device transmits the directional characteristic data prepared for each type of sound source as described above to the signal processing device on the playback side.

[0077] Here, a specific example of the transmission of metadata and directional characteristic data will be described.

[0078] For example, metadata can be prepared for each frame of a predetermined duration of the audio data of an object, and the metadata can be transmitted to the playback side for each frame using the bitstream syntax shown in Figure 3. In Figure 3, uimsbf is an unsigned integer MSB first, and tcimsbf is a two's complement integer MSB first.

[0079] In the example of FIG. 3, the metadata includes, for each object that constitutes the content, sound source type information "Object_type_index", sound source position information "Object_position[3]", and sound source direction information "Object_direction[3]".

[0080] In particular, in this example, the sound source position information Object_position[3] is the coordinates (x o ,y o ,z o ) and this coordinate (x o ,y o ,z o ) indicates the absolute position of an object in the xyz coordinate system, i.e., object space.

[0081] The sound source direction information Object_direction[3] is an azimuth angle ψ that indicates the absolute direction of the object in the target space. o , elevation angle θ o , and the tilt angle φ o It consists of:

[0082] For example, in free viewpoint content, the viewpoint (listening position) changes over time as the content is played back, so it is advantageous for generating playback signals to express the position of an object using coordinates that indicate an absolute position rather than relative coordinates based on the listening position.

[0083] In contrast, for example, if the content is from a fixed viewpoint, the sound source position information indicating the position of the object can be the coordinates of a polar coordinate system consisting of the azimuth and elevation angles indicating the direction of the object as seen from the listening position, and the radius indicating the distance from the listening position to the object.

[0084] The configuration of the metadata is not limited to the example shown in Fig. 3, and may be any other configuration. Furthermore, the metadata only needs to be transmitted at predetermined time intervals, and does not necessarily need to be transmitted for each frame.

[0085] Furthermore, the directional characteristic data for each sound source type may be stored in metadata and transmitted, or may be transmitted in advance, separately from the metadata and audio data, for example, using the bitstream syntax shown in Figure 4.

[0086] In the example of Figure 4, the directional characteristic data corresponding to the value of the specified sound source type information is transmitted as a gain function "Object_directivity[distance][azimuth][elevation]" whose arguments are the distance from the sound source "distance," and the azimuth and elevation angles "elevation" indicating the direction as seen from the sound source.

[0087] In addition, the directional characteristic data may be in a format in which the sampling intervals of the azimuth angle and elevation angle, which are arguments, are not equal angular intervals, or may be in HOA (Higher Order Ambisonics) format, i.e., Ambisonics format data (spherical harmonic coefficients).

[0088] For example, it is advisable to transmit the directional characteristic data for a general sound source type to the playback side in advance.

[0089] On the other hand, for directional pattern data of sound sources with unusual directional characteristics, such as objects that are not predefined, it is conceivable that the directional pattern data may be included in the metadata shown in Figure 3 and transmitted as metadata.

[0090] In this manner, the metadata, audio data, and directional characteristic data are transmitted from the transmitting device to the signal processing device on the playback side.

[0091] <Configuration example of signal processing device> Next, a signal processing device, which is a device on the playback side, will be described.

[0092] For example, a signal processing device on the playback side is configured as shown in FIG.

[0093] The signal processing device 11 shown in Figure 5 generates a playback signal for playing back the sound of content (object) at the listening position based on directional characteristic data that has been acquired in advance from a transmitting device or the like or that has been shared in advance, and outputs the signal to the playback unit 12.

[0094] For example, the signal processing device 11 generates a playback signal by using the directional characteristic data to perform processing for VBAP (Vector Based Amplitude Panning) or wave field synthesis, convolution processing of HRTF (Head Related Transfer Function), and the like.

[0095] The playback unit 12 is composed of, for example, headphones, earphones, a speaker array consisting of two or more speakers, and plays back the sound of the content based on the playback signal supplied from the signal processing device 11.

[0096] The signal processing device 11 also includes an acquisition unit 21, a listening position designation unit 22, a directional characteristic database unit 23, and a signal generation unit 24.

[0097] The acquisition unit 21 acquires directional pattern data, metadata, and audio data by, for example, receiving data transmitted from a transmitting device or reading data from a transmitting device connected via a wire or the like.

[0098] The timing of acquiring the directional characteristic data and the timing of acquiring the metadata and audio data may be the same or different.

[0099] The acquisition unit 21 supplies the acquired directional pattern data and metadata to the directional pattern database unit 23, and also supplies the acquired metadata and audio data to the signal generation unit 24.

[0100] The listening position designation unit 22 designates a listening position in the target space and the orientation of the listener (user) at that listening position, and supplies listening position information indicating the listening position and listener orientation information indicating the orientation of the listener to the signal generation unit 24 as the designation results.

[0101] The directional characteristic database unit 23 records the directional characteristic data for each of the plurality of sound source types supplied from the acquisition unit 21 .

[0102] Furthermore, when the directional pattern database unit 23 receives sound source type information included in the metadata from the acquisition unit 21, it supplies the signal generation unit 24 with directional pattern data of the sound source type indicated by the received sound source type information from among the multiple directional pattern data stored therein.

[0103] The signal generation unit 24 generates a playback signal based on the metadata and audio data supplied from the acquisition unit 21, the listening position information and listener direction information supplied from the listening position designation unit 22, and the directional characteristic data supplied from the directional characteristic database unit 23, and supplies the generated signal to the playback unit 12.

[0104] The signal generating unit 24 includes a relative distance calculating unit 31 , a relative direction calculating unit 32 , and a directivity rendering unit 33 .

[0105] The relative distance calculation unit 31 calculates the relative distance between the listening position (listener) and the object based on the sound source position information included in the metadata supplied from the acquisition unit 21 and the listening position information supplied from the listening position designation unit 22, and supplies relative distance information indicating the calculation result to the directivity rendering unit 33.

[0106] The relative direction calculation unit 32 calculates the relative direction between the listener and the object based on the sound source position information and sound source direction information included in the metadata supplied from the acquisition unit 21 and the listening position information and listener direction information supplied from the listening position designation unit 22, and supplies relative direction information indicating the calculation result to the directivity rendering unit 33.

[0107] The directivity rendering unit 33 performs rendering processing based on the audio data supplied from the acquisition unit 21, the directional characteristic data supplied from the directional characteristic database unit 23, the relative distance information supplied from the relative distance calculation unit 31, the relative direction information supplied from the relative direction calculation unit 32, and the listening position information and listener direction information supplied from the listening position designation unit 22.

[0108] The directional rendering unit 33 supplies the playback signal obtained by the rendering process to the playback unit 12, which plays back the sound of the content. For example, the directional rendering unit 33 performs rendering processes such as VBAP, wave field synthesis, and HRTF convolution.

[0109] <About each part of the signal processing device> (Listening position designation section) Next, each part of the signal processing device 11 will be described in more detail.

[0110] The listening position designation unit 22 designates the listening position and the direction of the listener in response to a user operation or the like.

[0111] For example, if the content is of a free viewpoint, the user viewing the content, i.e., the listener, may specify an arbitrary listening position or orientation by operating a GUI (Graphical User Interface) or the like in a running service or application.

[0112] In this case, the listening position designation unit 22 sets the listening position and the listener's orientation designated by the user as the listening position (viewpoint position) that serves as the viewpoint of the content and the direction the listener is facing, i.e., the listener's orientation.

[0113] Furthermore, for example, when a user designates a desired player from among a plurality of predetermined players, the position and orientation of the player may be set as the listening position and orientation of the listener.

[0114] Furthermore, the listening position designation unit 22 may execute some kind of automatic route designation program or obtain information indicating the user's position and orientation from a head-mounted display equipped with the playback unit 12, thereby allowing any listening position and orientation of the listener to be designated without requiring user operation.

[0115] In this way, in free viewpoint content, the listening position and the orientation of the listener are set to be arbitrary positions and orientations that can change over time.

[0116] On the other hand, for content with a fixed viewpoint, the listening position designation unit 22 designates a predetermined fixed position and fixed orientation as the listening position and orientation of the listener.

[0117] As a specific example of the listening position information indicating the listening position, for example, coordinates (x,y,z) indicating the listening position in an xyz coordinate system indicating the absolute position on the Earth's surface or an xyz coordinate system indicating the absolute position in the target space are used. v ,y v ,z v ) is possible.

[0118] For example, the listener direction information may be an azimuth angle ψ that indicates the absolute direction of the listener in the xyz coordinate system. v and elevation angle θ v and the tilt angle φ, which is the angle of absolute rotation (tilt) of the listener in the xyz coordinate system. v The information can be the Euler angles.

[0119] In particular, in this case, when the content is of a fixed viewpoint, for example, listening position information (x v ,y v ,z v )=(0,0,0), and the listener direction information (ψ v ,θ v ,φ v )=(0,0,0).

[0120] In the following, the listening position information is expressed as coordinates in the xyz coordinate system (x v ,y v ,z v ), and the listener direction information is the Euler angle (ψ v ,θ v ,φ v ) will be assumed in the following explanation.

[0121] Similarly, in the following, the sound source position information is expressed as coordinates (x o ,y o ,z o ), and the sound source direction information is the Euler angle (ψ o ,θ o ,φ o ) will be assumed in the following explanation.

[0122] (Relative distance calculation section) The relative distance calculation unit 31 calculates the relative distance d from the listening position to each object constituting the content. o It is calculated as:

[0123] Specifically, the relative distance calculation unit 31 calculates the listening position information (x v ,y v ,z v ) and sound source location information (x o ,yo ,z o ) based on the following equation (1), the relative distance d o Calculate the relative distance d o The relative distance information indicating the distance is output.

[0124]

number

[0125] (Relative direction calculation section) Furthermore, the relative direction calculation unit 32 obtains relative direction information that indicates the relative direction between the listener and the object.

[0126] For example, the relative orientation information includes the object orientation angle ψ i_obj , object elevation angle θ i_obj , object rotation azimuth angle ψ_rot i_obj , and the object rotation elevation angle θ_rot i_obj Contains:

[0127] where the object azimuth angle ψ i_obj and object elevation angle θ i_obj are the azimuth and elevation angles, respectively, that indicate the relative direction of the object as seen by the listener.

[0128] Listening location information (x v ,y v ,z v ) is the origin, and the listener direction information (ψ v ,θ v ,φ v ) is called the listener coordinate system. In the listener coordinate system, the direction of the listener, that is, the direction in front of the listener, is the +y direction.

[0129] In this case, the azimuth angle and elevation angle indicating the direction of the object in the listener coordinate system are the object azimuth angle ψ i_obj and object elevation angle θ i_obj This becomes:

[0130] Similarly, the object rotation azimuth angle ψ_rot i_obj and the object rotation elevation angle θ_rot i_obj are the azimuth and elevation angles that indicate the relative direction of the listener (listening position) as seen from the object. In other words, the object rotation azimuth angle ψ_rot i_obj and the object rotation elevation angle θ_rot i_obj It can be said that the information indicates how much the front direction of the object has rotated relative to the listener.

[0131] Sound source location information (x o ,y o ,z o ) is the origin, and the sound source direction information (ψ o ,θ o ,φ o ) is called the object coordinate system. In the object coordinate system, the orientation of the object, that is, the direction facing the object, is the +y direction.

[0132] At this time, the azimuth angle and elevation angle indicating the direction of the listener (listening position) in the object coordinate system are the object rotation azimuth angle ψ_rot i_obj and the object rotation elevation angle θ_rot i_obj This becomes:

[0133] These objects have a rotation azimuth angle ψ_rot i_obj and the object rotation elevation angle θ_rot i_obj are the azimuth angle and elevation angle when referencing the directional characteristic data during rendering processing.

[0134] In the following description, the azimuth angle in each three-dimensional orthogonal coordinate system, such as the xyz coordinate system of the target space, the listener coordinate system, and the object coordinate system, is assumed to be positive in the clockwise direction from the front direction (+y direction).

[0135] For example, in the xyz coordinate system, after projecting a target point of an object onto the xy plane, the angle indicating the position (direction) of the target point after projection based on the +y direction on the xy plane, that is, the angle between the direction of the target point after projection and the +y direction, is the azimuth angle. In this case, the clockwise direction from the +y direction is the positive direction.

[0136] In addition, in the listener coordinate system and the object coordinate system, the orientation of the listener or the object, that is, the direction in front of the listener or the object, is the +y direction.

[0137] The upward direction is assumed to be the positive direction in the elevation angle in each three-dimensional orthogonal coordinate system such as the xyz coordinate system of the target space, the listener coordinate system, and the object coordinate system.

[0138] For example, in the xyz coordinate system, the angle between the xy plane and a line passing through the origin of the xyz coordinate system and a target point such as an object is the elevation angle.

[0139] Furthermore, when a target point of an object or the like is projected onto the xy plane, if the plane containing the origin of the xyz coordinate system, the target point, and the projected target point is called plane A, then on plane A, the +z direction from the xy plane is considered to be the positive direction of the elevation angle.

[0140] In the case of a listener coordinate system or an object coordinate system, for example, the object or listening position is the target point.

[0141] Furthermore, the tilt angle in each three-dimensional Cartesian coordinate system, such as the xyz coordinate system of the target space, the listener coordinate system, and the object coordinate system, is considered to be a positive rotation when, after the elevation angle rotation, it rotates to the upper right with the +y direction as the front direction.

[0142] Although the azimuth angle, elevation angle, and tilt angle, which indicate the listening position and object direction in a three-dimensional Cartesian coordinate system, are defined as above, generality is not lost even if other definitions are used, such as when using quaternions or rotation matrices.

[0143] Here, the relative distance d o and the object azimuth angle ψ i_obj , object elevation angle θ i_obj , object rotation azimuth angle ψ_rot i_obj , object rotation elevation angle θ_rot i_obj A specific example of this will be described.

[0144] First, a case will be described in which only the azimuth angle is taken into consideration in the sound source direction information and the listener direction information, and the elevation angle and tilt angle are not taken into consideration, that is, a two-dimensional case.

[0145] For example, as shown in FIG. 6, it is assumed that the position of point P21 in an xy coordinate system with origin O as the reference is the listening position, and an object is located at point P22.

[0146] Also, the direction of a line segment W11 passing through point P21, more specifically, the direction from point P21 to the end point of line segment W11 on the opposite side of point P21, is assumed to be the direction indicating the orientation of the listener.

[0147] Similarly, the direction of a line segment W12 passing through point P22 is assumed to be the direction indicating the orientation of the object. Furthermore, a line passing through points P21 and P22 is assumed to be a line L11.

[0148] In this case, the distance between points P21 and P22 is the relative distance d o It is said that.

[0149] The angle between the line segment W11 and the straight line L11, that is, the angle indicated by the arrow K11, is the object azimuth angle ψ i_obj Similarly, the angle between the line segment W12 and the straight line L11, that is, the angle indicated by the arrow K12, is the object rotation azimuth angle ψ_rot i_obj This becomes:

[0150] Furthermore, if the target space is three-dimensional, the relative distance d o and the object azimuth angle ψ i_obj , object elevation angle θ i_obj , object rotation azimuth angle ψ_rot i_obj, object rotation elevation angle θ_rot i_obj 7 to 9. In addition, in Fig. 7 to 9, the same reference numerals are used to designate corresponding parts, and the description thereof will be omitted as appropriate.

[0151] For example, as shown in FIG. 7, in an xyz coordinate system with origin O as the reference, the positions of points P31 and P32 are the listening position and the object position, respectively, and the line passing through points P31 and P32 is taken as line L31.

[0152] In addition, the xy plane of the xyz coordinate system is used as the listener direction information (ψ v ,θ v ,φ v ), and then rotate the origin O by the angle indicated by the listening position information (x v ,y v ,z v ) is translated to the position indicated by the plane PF11. This plane PF11 is the xy plane of the listener coordinate system.

[0153] Similarly, the xy plane of the xyz coordinate system is used as the sound source direction information (ψ o ,θ o ,φ o ), and then rotate the origin O by the angle indicated by the sound source position information (x o ,y o ,z o ) is translated to the position indicated by the arrow PF12. This plane PF12 is the xy plane of the object coordinate system.

[0154] The direction of the line segment W21 passing through the point P31, more specifically, the direction from the point P31 to the end point of the line segment W21 opposite to the point P31, is determined as the listener direction information (ψ v ,θ v ,φ v ) is the direction indicating the orientation of the listener.

[0155] Similarly, the direction of the line segment W22 passing through the point P32 is determined as the sound source direction information (ψ o ,θ o ,φ o) is the direction indicating the orientation of the object.

[0156] In this case, the distance between points P31 and P32 is the relative distance d o It is said that.

[0157] Furthermore, as shown in FIG. 8, if the line obtained by projecting the line L31 onto the plane PF11 is a line L41, the angle formed between the line L41 and the line segment W21 on the plane PF11, that is, the angle indicated by the arrow K21, is the object azimuth angle ψ i_obj This becomes:

[0158] Furthermore, the angle between the line L41 and the line L31, i.e., the angle indicated by the arrow K22, is the object elevation angle θ i_obj In other words, the object elevation angle θ i_obj is the angle between the plane PF11 and the line L31.

[0159] On the other hand, as shown in FIG. 9, if the line obtained by projecting the line L31 onto the plane PF12 is a line L51, the angle formed between the line L51 and the line segment W22 on the plane PF12, that is, the angle indicated by the arrow K31, is the object rotation azimuth angle ψ_rot i_obj This becomes:

[0160] Furthermore, the angle between the line L51 and the line L31, i.e., the angle indicated by the arrow K32, is the object rotation elevation angle θ_rot i_obj In other words, the object rotation elevation angle θ_rot i_obj is the angle between the plane PF12 and the line L31.

[0161] The object azimuth angle ψ described above i_obj , object elevation angle θ i_obj , object rotation azimuth angle ψ_rot i_obj , and the object rotation elevation angle θ_rot i_obj Specifically, the relative direction information can be calculated, for example, as follows.

[0162] For example, the rotation matrix that describes rotation in three-dimensional space is as shown in the following equation (2).

[0163]

number

[0164] In equation (2), the coordinates (x, y, z) in the X1Y1Z1 space, which is a three-dimensional Cartesian coordinate system space with the predetermined X1, Y1, and Z1 axes as its axes, are rotated by the rotation matrix to obtain the rotated coordinates (x', y', z').

[0165] That is, in the calculation shown in formula (2), the second matrix from the right on the right side is a rotation matrix that rotates the X1Y1Z1 space by an angle φ around the Z1 axis in the X1Y1 plane to obtain the rotated X2Y2Z1 space. In other words, the second rotation matrix from the right on the right side rotates the coordinates (x, y, z) by an angle -φ on the X1Y1 plane.

[0166] Furthermore, the third matrix from the right on the right side of equation (2) is a rotation matrix that rotates the X2Y2Z1 space by an angle θ around the X2 axis in the Y2Z1 plane, obtaining the rotated X2Y3Z2 space.

[0167] Furthermore, the fourth matrix from the right on the right side of equation (2) is a rotation matrix that rotates the X2Y3Z2 space by an angle ψ around the Y3 axis within the X2Z2 plane, obtaining the rotated X3Y3Z3 space.

[0168] In the relative direction calculation unit 32, the rotation matrix shown in equation (2) is used to generate relative direction information.

[0169] Specifically, the relative direction calculation unit 32 calculates the sound source position information (x o ,y o ,z o ) and listener direction information (ψ v ,θ v ,φ v ) is calculated using the following equation (3), and the coordinates (xo ,y o ,z o ) coordinates after rotation (x o ',y o ',z o ').

[0170]

number

[0171] In the calculation of equation (3), φ=-φ v , θ=-θ v , and ψ=-ψ v Then, the rotation matrix is ​​used for calculation.

[0172] The coordinates thus obtained (x o ',y o ',z o ') are coordinates that indicate the position of the object in the listener's coordinate system. However, the origin of the listener's coordinate system here is not the listening position but the origin O of the xyz coordinate system of the target space.

[0173] Next, the relative direction calculation unit 32 calculates the listening position information (x v ,y v ,z v ) and listener direction information (ψ v ,θ v ,φ v ) is calculated using the following equation (4), and the coordinates (x v ,y v ,z v ) coordinates after rotation (x v ',y v ',z v ').

[0174]

number

[0175] In the calculation of equation (4), φ=-φ v , θ=-θ v , and ψ=-ψ vThen, the rotation matrix is ​​used for calculation.

[0176] The coordinates thus obtained (x v ',y v ',z v ') are coordinates that indicate the listening position in the listener coordinate system. However, the origin of the listener coordinate system here is not the listening position but the origin O of the xyz coordinate system of the target space.

[0177] Furthermore, the relative direction calculation unit 32 calculates the coordinates (x o ',y o ',z o ') and the coordinates (x v ',y v ',z v ') and calculate the following equation (5).

[0178]

number

[0179] By calculating equation (5), the coordinates (x o '',y o '',z o This coordinate (x o '',y o '',z o '') are coordinates that indicate the relative position of an object as seen by the listener.

[0180] The relative direction calculation unit 32 calculates the coordinates (x o '',y o '',z o ' '), the following equations (6) and (7) are calculated, and the object azimuth angle ψ i_obj and object elevation angle θ i_obj get.

[0181]

number

[0182]

number

[0183] In equation (6), the x and y coordinates are x o '' and y o '' based on the object azimuth angle ψ i_obj is required.

[0184] More specifically, when calculating equation (6), y o '' sign and x o Based on the result of the 0 judgment for '', case distinction processing is performed, and depending on the result of the case distinction, exception processing is performed to determine the object azimuth angle ψ i_obj is calculated, but a detailed description thereof will be omitted here.

[0185] Also, in equation (7), the coordinates (x o '',y o '',z o '') based on the object elevation angle θ i_obj In more detail, when calculating equation (7), z o '' sign and (x o '' 2 +y o '' 2 ) is judged to be 0, and exception handling is performed depending on the result of the judgement. i_obj is calculated, but a detailed description thereof will be omitted here.

[0186] From the above calculation, the object azimuth angle ψ i_obj and the object elevation angle θ i_obj is calculated, the relative orientation calculation unit 32 performs a similar calculation to obtain the object rotation orientation angle ψ_rot i_obj and the object rotation elevation angle θ_rot i_obj Ask for.

[0187] That is, the relative direction calculation unit 32 calculates the listening position information (x v ,y v ,z v ) and sound source direction information (ψ o ,θ o ,φ o ) is calculated using the following equation (8), and the coordinates (x v ,y v ,z v ) coordinates after rotation (x v ',y v ',z v ').

[0188]

number

[0189] In the calculation of equation (8), φ=-φ o , θ=-θ o , and ψ=-ψ o Then, the rotation matrix is ​​used for calculation.

[0190] The coordinates thus obtained (x v ',y v ',z v ') are coordinates that indicate the listening position (listener's position) in the object coordinate system. However, the origin of the object coordinate system here is not the object position but the origin O of the xyz coordinate system of the target space.

[0191] Next, the relative direction calculation unit 32 calculates the sound source position information (x o ,y o ,z o ) and sound source direction information (ψ o ,θ o ,φ o ) is calculated using the following equation (9), and the coordinates (x o ,y o ,z o ) coordinates after rotation (x o ',y o ',z o ').

[0192]

number

[0193] In the calculation of equation (9), φ=-φ o , θ=-θ o , and ψ=-ψ o Then, the rotation matrix is ​​used for calculation.

[0194] The coordinates thus obtained (x o ',y o ',z o ') are coordinates that indicate the position of the object in the object coordinate system. However, the origin of the object coordinate system here is not the position of the object, but the origin O of the xyz coordinate system of the target space.

[0195] Furthermore, the relative direction calculation unit 32 calculates the coordinates (x v ',y v ',z v ') and the coordinates (x o ',y o ',z o ') and calculate the following equation (10).

[0196]

number

[0197] By calculating equation (10), the coordinate (x) indicating the listening position in the object coordinate system with the object position as the origin is obtained. v '',y v '',z v This coordinate (x v '',y v '',z v '') are coordinates that indicate the relative position of the listening position from the object's perspective.

[0198] The relative direction calculation unit 32 calculates the coordinates (x v '',yv '',z v '') and calculate the object rotation azimuth angle ψ_rot using the following equations (11) and (12). i_obj and the object rotation elevation angle θ_rot i_obj get.

[0199]

number

[0200]

number

[0201] In equation (11), the same calculation as in equation (6) is performed, and the object rotation azimuth angle ψ_rot i_obj In addition, in equation (12), the same calculation as in equation (7) is performed, and the object rotation elevation angle θ_rot i_obj is required.

[0202] The relative direction calculation unit 32 performs the above-described processing for each frame of audio data for a plurality of objects.

[0203] This allows us to calculate the object azimuth angle ψ for each object for each frame. i_obj , object elevation angle θ i_obj , object rotation azimuth angle ψ_rot i_obj , and the object rotation elevation angle θ_rot i_obj Relative direction information consisting of

[0204] By using the relative direction information obtained in this way, the sound image of each object can be localized in accordance with the listening position, the listener's orientation, and the movement and rotation of the object, resulting in a greater sense of realism.

[0205] (Directional pattern database section) The directional characteristic database unit 23 records directional characteristic data for each type of object, that is, for each type of sound source.

[0206] This directivity data is a function that uses the azimuth angle and elevation angle as arguments when viewed from the object to obtain the gain and spherical harmonic coefficient of the propagation direction indicated by the azimuth angle and elevation angle.

[0207] The directional characteristic data may be data in table format instead of a function, i.e., a table in which the azimuth angle and elevation angle as seen from the object correspond to the gain and spherical harmonic coefficient of the propagation direction indicated by those azimuth angle and elevation angle.

[0208] (Directional Rendering) The directional rendering unit 33 performs rendering processing based on the audio data of each object, the directional characteristic data, relative distance information, and relative direction information obtained for each object, as well as the listening position information and listener direction information, and generates a playback signal corresponding to the playback unit 12, which is the target device.

[0209] <Explanation of Content Playback Processing> Next, the operation of the signal processing device 11 will be described.

[0210] That is, the content playback process by the signal processing device 11 will be described below with reference to the flowchart of FIG.

[0211] In this description, it is assumed that the content to be played back is free viewpoint content, and that directional pattern data for each sound source type has been acquired in advance and recorded in the directional pattern database unit 23.

[0212] In step S11, the acquisition unit 21 acquires metadata and audio data for one frame of each object constituting the content from the transmission device. In other words, metadata and audio data are acquired at predetermined time intervals.

[0213] The acquisition unit 21 supplies the sound source type information included in the metadata of each acquired object to the directional characteristic database unit 23, and also supplies the audio data of each acquired object to the directivity rendering unit 33.

[0214] The acquisition unit 21 also acquires the sound source position information (x o ,y o ,z o ) to the relative distance calculation unit 31 and the relative direction calculation unit 32, and also supplies the sound source direction information (ψ o ,θ o ,φ o ) is supplied to the relative direction calculation unit 32.

[0215] In step S12, the listening position designation unit 22 designates the listening position and the orientation of the listener.

[0216] That is, the listening position designation unit 22 determines the listening position and the direction of the listener in response to the listener's operation, etc., and outputs listening position information (x v ,y v ,z v ) and listener direction information (ψ v ,θ v ,φ v )

[0217] The listening position designation unit 22 uses the obtained listening position information (x v ,y v ,z v ) is supplied to a relative distance calculation unit 31, a relative direction calculation unit 32, and a directivity rendering unit 33, and the obtained listener direction information (ψ v ,θ v ,φ v ) is supplied to the relative direction calculation unit 32 and the directivity rendering unit 33.

[0218] If the content is for a fixed viewpoint, for example, the listening position information is set to (0,0,0), and the listener direction information is also set to (0,0,0).

[0219] In step S13, the relative distance calculation unit 31 calculates the sound source position information (x o ,y o ,z o ) and the listening position information (x v ,y v ,z v ) and the relative distance d based on o and supplies relative distance information indicating the calculation result to the directional rendering unit 33. For example, in step S13, the above-mentioned formula (1) is calculated for each object, and the relative distance d o is calculated.

[0220] In step S14, the relative direction calculation unit 32 calculates the sound source position information (x o ,y o ,z o ) and sound source direction information (ψ o ,θ o ,φ o ) and the listening position information (x v ,y v ,z v ) and listener direction information (ψ v ,θ v ,φ v ), the relative direction between the listener and the object is calculated, and relative direction information indicating the calculation result is supplied to the directivity rendering unit 33.

[0221] For example, the relative azimuth calculation unit 32 calculates the above-mentioned equations (3) to (7) for each object to obtain the object azimuth angle ψ i_obj and object elevation angle θ i_obj Calculate.

[0222] Furthermore, for example, the relative orientation calculation unit 32 calculates the above-described equations (8) to (12) for each object to obtain the object rotation orientation angle ψ_rot for each object. i_obj and the object rotation elevation angle θ_rot i_objCalculate.

[0223] The relative azimuth calculation unit 32 calculates the object azimuth angle ψ obtained for each object. i_obj , object elevation angle θ i_obj , object rotation azimuth angle ψ_rot i_obj , and the object rotation elevation angle θ_rot i_obj The information consisting of the above is supplied to the directivity rendering unit 33 as relative direction information.

[0224] In step S15, the directivity rendering unit 33 acquires the directivity data from the directivity database unit 23.

[0225] For example, when metadata is acquired for each object in step S11 and sound source type information included in the metadata is supplied to the directional pattern database unit 23, the directional pattern database unit 23 outputs directional pattern data for each object.

[0226] That is, for each piece of sound source type information supplied from the acquisition unit 21, the directional characteristic database unit 23 reads out the directional characteristic data of the sound source type indicated by the sound source type information from among the multiple directional characteristic data recorded, and outputs the readout directional characteristic data to the directional rendering unit 33.

[0227] The directivity rendering unit 33 obtains the directivity data of each object by acquiring the directivity data output from the directivity database unit 23 for each object in this manner.

[0228] In step S16, the directivity rendering unit 33 converts the audio data supplied from the acquisition unit 21, the directional characteristic data supplied from the directional characteristic database unit 23, the relative distance information supplied from the relative distance calculation unit 31, the relative direction information supplied from the relative direction calculation unit 32, and the listening position information (x v ,y v ,z v ) and listener direction information (ψ v ,θ v,φ v ) to perform rendering processing.

[0229] In addition, the listening position information (x v ,y v ,z v ) and listener direction information (ψ v ,θ v ,φ v ) may be used in the rendering process as needed, but does not necessarily have to be used in the rendering process.

[0230] For example, the directional rendering unit 33 performs rendering processes such as VBAP, wave field synthesis, and HRTF convolution to generate a playback signal for playing back the sound of an object (content) at the listening position.

[0231] Here, an example will be described in which VBAP is performed as the rendering process, and in this case, the playback unit 12 is assumed to be configured with a plurality of speakers.

[0232] For simplicity of explanation, the case where the content is made up of one object will be explained as an example.

[0233] First, the directivity rendering unit 33 calculates the relative distance d o The following equation (13) is calculated based on the gain value gain i_obj Calculate.

[0234]

number

[0235] In addition, in equation (13), power (d o ,2.0) is the relative distance d o Here, an example in which the distance square law is used will be described, but the calculation of the gain value that reproduces the distance attenuation is not limited to this, and any other method may be used.

[0236] Next, the directivity rendering unit 33 calculates, for example, the object rotation azimuth angle ψ_rot included in the relative azimuth information. i_obj and the object rotation elevation angle θ_rot i_obj The gain value dir_gain according to the directional characteristics of the object is calculated by the following equation (14) based on i_obj Calculate.

[0237]

number

[0238] In equation (14), dir(i,ψ_rot i_obj ,θ_rot i_obj ) indicates a gain function corresponding to the value i of the sound source type information supplied as directional characteristic data.

[0239] Therefore, in the calculation of equation (14), the directional rendering unit 33 calculates the object rotation azimuth angle ψ_rot i_obj , and the object rotation elevation angle θ_rot i_obj is substituted into the gain function and the calculation is performed, and the gain value dir_gain is obtained as the result. i_obj get.

[0240] That is, in equation (14), the object rotation azimuth angle ψ_rot i_obj , and the object rotation elevation angle θ_rot i_obj and the directional characteristic data to calculate the gain value dir_gain i_obj is obtained.

[0241] The gain value dir_gain obtained in this way i_obj This realizes gain correction for adding the transfer characteristics of sound propagating from an object to a listener, in other words, gain correction for reproducing sound propagation according to the directional characteristics of the object.

[0242] As described above, the argument (variable) of the gain function as directional characteristic data includes the distance from the object, and the gain value dir_gain, which is the output of the gain function, is i_obj In this case, the gain function may be modified so that the relative distance d indicated by the relative distance information is used as the distance, which is the argument of the gain function. o will be used.

[0243] Furthermore, the directivity rendering unit 33 calculates the object azimuth angle ψ included in the relative azimuth information. i_obj and object elevation angle θ i_obj Based on this, the reproduction gain value VBAP_gain of the channel corresponding to each of the multiple speakers that constitute the reproduction unit 12 is calculated by VBAP. i_spk Ask for.

[0244] Then, the directional rendering unit 33 converts the audio data obj_audio of the object into i_obj , distance attenuation gain value gain i_obj , directional characteristic gain value dir_gain i_obj , and the playback gain value VBAP_gain of the channel corresponding to the speaker i_spk The following equation (15) is calculated based on the i_spk Ask for.

[0245]

number

[0246] Here, the calculation of equation (15) is performed for each combination of the speaker that constitutes the playback unit 12 and the object that constitutes the content, and the playback signal speaker_signal i_spk is required.

[0247] This achieves gain correction to reproduce distance attenuation, gain correction to reproduce sound propagation according to directional characteristics, and VBAP processing to localize the sound image at the desired position.

[0248] In contrast, the gain value dir_gain obtained from the directional characteristic data i_obj is a gain value that takes into account both the directional characteristics and distance attenuation, that is, the relative distance d indicated by the relative distance information as an argument of the gain function o If the value is included, the calculation of the following equation (16) is performed.

[0249] That is, the directional rendering unit 33 converts the audio data obj_audio of the object into i_obj , directional characteristic gain value dir_gain i_obj , and the playback gain value VBAP_gain i_spk The following equation (16) is calculated based on the i_spk Ask for.

[0250]

number

[0251] Once the playback signal is obtained in this manner, the directivity rendering unit 33 finally converts the playback signal speaker_signal obtained for the current frame into i_spk and the playback signal speaker_signal of the frame immediately before the current frame i_spk The signals are overlap-added to obtain the final reproduced signal.

[0252] Although the example described here is one in which VBAP is performed as the rendering process, a playback signal can also be obtained by similar processing when HRTF convolution processing is performed as the rendering process.

[0253] Here, we will explain how to generate a headphone playback signal that takes into account the directional characteristics of an object by using an HRTF database consisting of HRTFs for each user according to the distance, azimuth angle, and elevation angle that indicate the relative positional relationship between the object and the user (listener).

[0254] In particular, it is assumed here that an HRTF database consisting of HRTFs from virtual speakers corresponding to the real speakers at the time of HRTF measurement is held in the directivity rendering unit 33, and the playback unit 12 is a headphone.

[0255] Here, the case where an HRTF database is prepared for each user in consideration of differences in the characteristics of each individual user will be described, but a common HRTF database may also be used for all users.

[0256] In this example, the personal ID information identifying the individual user is j, and the azimuth and elevation angles indicating the direction of sound arrival from the sound source (virtual speaker), i.e., the object to the user's ear, are respectively ψ L and ψ R and θ L and θ R Here, the azimuth angle ψ L and elevation angle θ L are the azimuth and elevation angles indicating the direction of arrival at the user's left ear, and the azimuth angle ψ R and elevation angle θ R are the azimuth and elevation angles indicating the direction of arrival at the user's right ear.

[0257] In addition, the HRTF, which is the transfer characteristic from the sound source to the user's left ear, is specifically defined as HRTF(j,ψ L ,θ L ), and the HRTF, which is the transfer characteristic from the sound source to the user's right ear, is specifically referred to as HRTF(j,ψ R ,θ R ) will be written as

[0258] It is also possible to prepare HRTFs for each of the user's left and right ears for each direction of arrival and distance to the sound source, and to reproduce distance attenuation by convolution of the HRTFs.

[0259] Furthermore, the directional characteristic data may be a function that indicates the transfer characteristics from the sound source to each direction, or may be a gain function as in the above-mentioned VBAP example. However, the argument of the function is the object rotation azimuth angle ψ_rot i_obj and the object rotation elevation angle θ_rot i_obj is used.

[0260] Alternatively, the object rotation azimuth angle and the object rotation elevation angle may be calculated for each ear, taking into account the convergence angle of the user's ears relative to the object, i.e., the difference in the angle at which sound arrives from the object to the user's ears depending on the width of the user's face.

[0261] The convergence angle here is the angle formed by a line connecting the left ear of the user (listener) and the object and a line connecting the right ear of the user and the object.

[0262] In the following, of the object rotation azimuth angle and object rotation elevation angle that constitute the relative orientation information, the object rotation azimuth angle ψ_rot i_obj_l and the object rotation elevation angle θ_rot i_obj_l It will be written as follows.

[0263] Similarly, hereinafter, of the object rotation azimuth angle and object rotation elevation angle that constitute the relative orientation information, the object rotation azimuth angle ψ_rot obtained for the user's right ear is referred to as the object rotation azimuth angle ψ_rot i_obj_r and the object rotation elevation angle θ_rot i_obj_r It will be written as follows.

[0264] First, the directivity rendering unit 33 performs the calculation of the above-mentioned equation (13) to obtain a gain value gain for reproducing distance attenuation. i_obj Calculate.

[0265] In addition, the HRTF database provides HRTFs for each sound arrival direction and distance to the sound source, and if distance attenuation can be reproduced by convolution of HRTFs, the gain value i_obj Alternatively, the reproduction of distance attenuation may be realized by convolution of the transfer characteristic obtained from the directional characteristic data, rather than by convolution of the HRTF.

[0266] Next, the directivity rendering unit 33 acquires a transfer characteristic according to the directivity of the object based on, for example, the directivity data and the relative direction information.

[0267] For example, when a function for obtaining transfer characteristics is supplied as directional characteristic data and the function takes distance, azimuth angle, and elevation angle as arguments, the directivity rendering unit 33 calculates the following equation (17) based on the relative distance information, relative azimuth information, and directional characteristic data.

[0268]

number

[0269] That is, in equation (17), the directional rendering unit 33 calculates the relative distance d o d i_obj Let's say.

[0270] The directivity rendering unit 33 then uses the function dir(i,d i_obj ,ψ_rot i_obj_l ,θ_rot i_obj_l ) at a relative distance d o , object rotation azimuth angle ψ_rot i_obj_l , and the object rotation elevation angle θ_rot i_obj_l Substituting the transfer characteristic of the left ear, dir_func i_obj_l get.

[0271] Similarly, the directivity rendering unit 33 calculates the function dir(i,di_obj ,ψ_rot i_obj_r ,θ_rot i_obj_r ) at a relative distance d o , object rotation azimuth angle ψ_rot i_obj_r , and the object rotation elevation angle θ_rot i_obj_r Substituting the transfer characteristic of the right ear, dir_func i_obj_r get.

[0272] In this case, the transfer function dir_func i_obj_l and transfer characteristics dir_func i_obj_r By convolution, the distance attenuation can also be reproduced.

[0273] Furthermore, the directional rendering unit 33 calculates the object azimuth angle ψ i_obj and object elevation angle θ i_obj Based on this, the HRTF for the left ear (j, ψ L ,θ L ) and the right-ear HRTF(j,ψ R ,θ R ) is obtained. Here, for example, ψ L =ψ i_obj and θ L =θ i_obj HRTF(j,ψ L ,θ L ) is read from the HRTF database. Note that the object azimuth angle and object elevation angle may also be calculated for each of the left and right ears.

[0274] Once the transfer characteristics and HRTFs of the left and right ears are obtained through the above processing, the transfer characteristics and HRTFs are combined with the object's audio data obj_audio i_obj Based on this, playback signals for the left and right ears to be supplied to headphones as the playback unit 12 are calculated.

[0275] Specifically, for example, the transfer characteristic dir_func obtained from the directional characteristic data i_obj_l and transfer characteristic dir_func i_obj_rWhen both the directivity and the distance attenuation are taken into consideration, that is, when the transfer characteristic is calculated by the formula (17), the directivity rendering unit 33 calculates the reproduction signal HPout for the left ear by the following formula (18): L and the playback signal HPout for the right ear R Ask for.

[0276]

number

[0277] In addition, in equation (18), * represents convolution processing.

[0278] So here the audio data obj_audio i_obj For the transfer characteristic dir_func i_obj_l and HRTF(j,ψ L ,θ L ) is convolved to generate the playback signal HPout for the left ear. L Similarly, the audio data obj_audio i_obj For the transfer characteristic dir_func i_obj_r and HRTF(j,ψ R ,θ R ) is convolved to generate the playback signal HPout for the right ear R Furthermore, even when distance attenuation is reproduced using HRTF, the reproduced signal can be obtained by a calculation similar to that of equation (18).

[0279] On the other hand, if the transfer characteristics or HRTF obtained from the directivity data do not take into account distance attenuation, the directivity rendering unit 33 calculates the reproduction signal by performing the following equation (19).

[0280]

number

[0281] In equation (19), in addition to the convolution process performed in equation (18), the audio data obj_audioi_obj The gain value for reproducing distance attenuation is i_obj The left ear playback signal HPout is also convoluted with the L and the playback signal HPout for the right ear R This gain value is i_obj is obtained by the above equation (13).

[0282] The above processing results in a reproduced signal HPout L and playback signal HPout R is obtained, the directivity rendering unit 33 performs overlap addition with the playback signal of the previous frame to obtain the final playback signal HPout L and playback signal HPout R Let's say.

[0283] Furthermore, when processing for wave field synthesis is performed as rendering processing, i.e., when a sound field including the sound of an object is formed by wave field synthesis using multiple speakers as the playback unit 12, a playback signal is generated as follows.

[0284] Here, an example will be described in which a speaker drive signal to be supplied to a speaker constituting the playback unit 12 is generated as a playback signal using a spherical harmonic function.

[0285] The external sound field at a position outside a certain radius r from a given sound source, that is, a position where the radius (distance) from the sound source is r' (where r'>r) and the azimuth and elevation angles indicating the direction as seen from the sound source are ψ and θ, i.e., sound pressure p(r',ψ,θ), can be expressed by the following equation (20):

[0286]

number

[0287] In addition, in equation (20), Y n m (ψ,θ) is a spherical harmonic function, and n and m indicate the degree and order of the spherical harmonic function. n(1) (kr) is the first kind of spherical Hankel function, where k is the wave number.

[0288] Furthermore, in equation (20), X(k) represents the reproduced signal expressed in the frequency domain, and P nm (r) shows the spherical harmonic spectrum for a sphere of radius (distance) r, where the frequency domain signal X(k) corresponds to the audio data of the object.

[0289] For example, if the measurement microphone array for measuring directional characteristics is spherical with a radius of r, it is possible to measure the sound pressure at a position of radius r of sound propagating in all directions from a sound source at the center of the sphere (measurement microphone array). In particular, since directional characteristics differ depending on the sound source, measuring the sound from the sound source at each position will allow for the acquisition of observed sound containing directional characteristic information.

[0290] Spherical harmonic spectrum P nm (r) can be written as the following equation (21) using the measured observed sound pressure p(r, ψ, θ) measured by such a measurement microphone array.

[0291]

number

[0292] In equation (21), ∂Ω represents the integral range, particularly the integral over the radius r.

[0293] Such a spherical harmonic spectrum P nm (r) is data indicating the directional characteristics of the sound source. Therefore, for example, for each type of sound source, the spherical harmonic spectrum P nm If (r) is measured in advance, the function shown in the following equation (22) can be used to calculate the directional characteristic data dir(i_obj,d i_obj ) can be used as

[0294]

number

[0295] In equation (22), i_obj indicates the type of sound source, and d i_obj indicates the distance from the sound source, and this distance d i_obj is the relative distance d o The directional pattern data dir(i_obj,d i_obj ) is data that indicates the transmission characteristics in each direction determined by the azimuth angle ψ and the elevation angle θ, that is, in all directions, taking into account the amplitude and phase.

[0296] If there is no change in the relative positional relationship between the object and the listening position, a reproduced signal that also takes into account the directional characteristics can be obtained using the above equation (20).

[0297] However, even if the relative positional relationship between the object and the listening position changes, the directional characteristic data dir(i_obj, d i_obj ) with respect to the object rotation azimuth angle ψ_rot i_obj and the object rotation elevation angle θ_rot i_obj By performing a rotation operation based on the azimuth angle ψ, elevation angle θ, and distance d i_obj The point determined by (d i_obj ,ψ,θ) i_obj ,ψ,θ) can be obtained.

[0298]

number

[0299] When calculating equation (23), the distance d i_obj as the relative distance d o is assigned, and the object's audio data is assigned to X(k), and the sound pressure p(d i_obj ,ψ,θ) is obtained. Then, the sound pressure p(d i_obj,ψ,θ) to find the sum of the points (d i_obj , ψ, θ), that is, the reproduced signal, is obtained.

[0300] Therefore, when generating a reproduction signal for wave field synthesis, the calculation of equation (23) is performed for each wave number k for each object in the process of step S16, and a reproduction signal is generated based on the calculation result.

[0301] Once the reproduction signal to be supplied to the reproduction unit 12 is obtained by the rendering process described above, the process proceeds from step S16 to step S17.

[0302] In step S17, the directional rendering unit 33 supplies the playback signal obtained by the rendering process to the playback unit 12, causing it to output sound. As a result, the sound of the content, that is, the sound of the object, is played back.

[0303] In step S18, the signal generation unit 24 determines whether or not to end the process of reproducing the sound of the content. For example, when the process has been performed for all frames and the reproduction of the content has ended, it is determined that the process is to end.

[0304] If it is determined in step S18 that the process is not yet finished, the process returns to step S11, and the above-described process is repeated.

[0305] On the other hand, if it is determined in step S18 that the process is to be ended, the content playback process ends.

[0306] In this way, the signal processing device 11 generates relative distance information and relative direction information, and performs rendering processing that takes into account the directional characteristics using the relative distance information and relative direction information. In this way, sound propagation according to the directional characteristics of the object can be reproduced, providing a higher sense of realism.

[0307] <Example of computer configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0308] FIG. 11 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes according to a program.

[0309] In the computer, a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, and a RAM (Random Access Memory) 503 are interconnected by a bus 504.

[0310] An input / output interface 505 is further connected to the bus 504. An input unit 506, an output unit 507, a recording unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.

[0311] The input unit 506 includes a keyboard, a mouse, a microphone, an image sensor, etc. The output unit 507 includes a display, a speaker, etc. The recording unit 508 includes a hard disk, a non-volatile memory, etc. The communication unit 509 includes a network interface, etc. The drive 510 drives a removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0312] In a computer configured as described above, the CPU 501 performs the above-described series of processes by, for example, loading a program recorded in the recording unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executing it.

[0313] The program executed by the computer (CPU 501) can be provided by being recorded on a removable recording medium 511 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0314] In a computer, a program can be installed in the recording unit 508 via the input / output interface 505 by inserting a removable recording medium 511 into the drive 510. The program can also be received by the communication unit 509 via a wired or wireless transmission medium and installed in the recording unit 508. Alternatively, the program can be installed in the ROM 502 or the recording unit 508 in advance.

[0315] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0316] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0317] For example, this technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0318] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by multiple devices.

[0319] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0320] Furthermore, the present technology can also be configured as follows.

[0321] (1) an acquisition unit that acquires metadata including location information indicating a location of an audio object and orientation information indicating a direction of the audio object, and audio data of the audio object; a signal generating unit that generates a playback signal for playing back the sound of the audio object at the listening position based on listening position information indicating a listening position, listener direction information indicating a direction of the listener at the listening position, the position information, the direction information, and the audio data; A signal processing device comprising: (2) The acquisition unit acquires the metadata at predetermined time intervals. The signal processing device according to (1). (3) The signal generating unit generates the playback signal based on directional characteristic data indicating directional characteristics of the audio object, the listening position information, the listener direction information, the position information, the direction information, and the audio data. A signal processing device according to (1) or (2). (4) The signal generation unit generates the playback signal based on the directional characteristic data determined for the type of the audio object. A signal processing device according to (3). (5) The orientation information is information including an azimuth angle indicating the direction of the audio object. A signal processing device according to (3) or (4). (6) The orientation information includes an azimuth angle and an elevation angle that indicate the direction of the audio object. A signal processing device according to (3) or (4). (7) The orientation information includes an azimuth angle and an elevation angle that indicate the direction of the audio object, and a tilt angle that indicates the rotation of the audio object. A signal processing device according to (3) or (4). (8) The listening position information is information indicating a predetermined fixed listening position, and the listener direction information is information indicating a predetermined fixed orientation of the listener. A signal processing device according to any one of (3) to (7). (9) The position information is information consisting of an azimuth angle and an elevation angle that indicate the direction of the audio object as seen from the listening position, and a radius that indicates the distance from the listening position to the audio object. (8) A signal processing device according to (8). (10) The listening position information is information indicating an arbitrary listening position, and the listener direction information is information indicating an arbitrary direction of the listener. A signal processing device according to any one of (3) to (7). (11) The position information is a coordinate in a Cartesian coordinate system indicating the position of the audio object. The signal processing device according to (10). (12) The signal generation unit The directional characteristic data; relative distance information indicating a relative distance between the audio object and the listening position, the relative distance information being obtained from the listening position information and the position information; relative orientation information indicating a relative direction between the audio object and the listener, the relative orientation information being obtained from the listening position information, the listener orientation information, the position information, and the orientation information; The audio data; generating the reproduction signal based on A signal processing device according to any one of (3) to (11). (13) The relative direction information includes an azimuth angle and an elevation angle that indicate the relative direction between the audio object and the listener. The signal processing device according to (12). (14) The relative direction information includes information indicating a direction of the listener as seen from the audio object and information indicating a direction of the audio object as seen from the listener. The signal processing device according to (12) or (13). (15) The signal generation unit generates the playback signal based on information indicating a transfer characteristic of the direction of the listener as seen from the audio object, the transfer characteristic being obtained from the directional characteristic data and information indicating the direction of the listener as seen from the audio object. The signal processing device according to (14). (16) The signal processing device acquiring metadata including location information indicating a location of an audio object and orientation information indicating a direction of the audio object, and audio data of the audio object; generating a playback signal for playing back the sound of the audio object at the listening position based on listening position information indicating the listening position, listener direction information indicating the direction of the listener at the listening position, the position information, the direction information, and the audio data; Signal processing methods. (17) acquiring metadata including location information indicating a location of an audio object and orientation information indicating a direction of the audio object, and audio data of the audio object; generating a playback signal for playing back the sound of the audio object at the listening position based on listening position information indicating the listening position, listener direction information indicating the direction of the listener at the listening position, the position information, the direction information, and the audio data; A program that causes a computer to execute a process that includes steps. [Explanation of symbols]

[0322] 11 signal processing device, 21 acquisition unit, 22 listening position designation unit, 23 directional characteristic database unit, 24 signal generation unit, 31 relative distance calculation unit, 32 relative direction calculation unit, 33 directional characteristic rendering unit

Claims

1. an acquisition unit that acquires metadata including location information indicating a location of an audio object and orientation information indicating a direction of the audio object, and audio data of the audio object; a signal generating unit that generates a playback signal for playing back the sound of the audio object at the listening position based on listening position information indicating a listening position, listener direction information indicating a direction of the listener at the listening position, the position information, the direction information, and the audio data; A signal processing device comprising:

2. The acquisition unit acquires the metadata at predetermined time intervals. The signal processing device according to claim 1 .

3. The signal generating unit generates the playback signal based on directional characteristic data indicating directional characteristics of the audio object, the listening position information, the listener direction information, the position information, the direction information, and the audio data. The signal processing device according to claim 1 .

4. The signal generation unit generates the playback signal based on the directional characteristic data determined for the type of the audio object. The signal processing device according to claim 3 .

5. The orientation information is information including an azimuth angle indicating the direction of the audio object. The signal processing device according to claim 3 .

6. The orientation information includes an azimuth angle and an elevation angle that indicate the direction of the audio object. The signal processing device according to claim 3 .

7. The orientation information includes an azimuth angle and an elevation angle that indicate the direction of the audio object, and a tilt angle that indicates the rotation of the audio object. The signal processing device according to claim 3 .

8. The listening position information is information indicating a predetermined fixed listening position, and the listener direction information is information indicating a predetermined fixed orientation of the listener. The signal processing device according to claim 3 .

9. The position information is information consisting of an azimuth angle and an elevation angle that indicate the direction of the audio object as seen from the listening position, and a radius that indicates the distance from the listening position to the audio object. The signal processing device according to claim 8 .

10. The listening position information is information indicating an arbitrary listening position, and the listener direction information is information indicating an arbitrary direction of the listener. The signal processing device according to claim 3 .

11. The position information is a coordinate in a Cartesian coordinate system indicating the position of the audio object. The signal processing device according to claim 10.

12. The signal generation unit The directional characteristic data; relative distance information indicating a relative distance between the audio object and the listening position, the relative distance information being obtained from the listening position information and the position information; relative orientation information indicating a relative direction between the audio object and the listener, the relative orientation information being obtained from the listening position information, the listener orientation information, the position information, and the orientation information; The audio data; generating the reproduction signal based on The signal processing device according to claim 3 .

13. The relative direction information includes an azimuth angle and an elevation angle that indicate the relative direction between the audio object and the listener. The signal processing device according to claim 12.

14. The relative direction information includes information indicating a direction of the listener as seen from the audio object and information indicating a direction of the audio object as seen from the listener. The signal processing device according to claim 12.

15. The signal generation unit generates the playback signal based on information indicating a transfer characteristic of the direction of the listener as seen from the audio object, the transfer characteristic being obtained from the directional characteristic data and information indicating the direction of the listener as seen from the audio object. The signal processing device according to claim 14.

16. The signal processing device acquiring metadata including location information indicating a location of an audio object and orientation information indicating a direction of the audio object, and audio data of the audio object; generating a playback signal for playing back the sound of the audio object at the listening position based on listening position information indicating the listening position, listener direction information indicating the direction of the listener at the listening position, the position information, the direction information, and the audio data; Signal processing methods.

17. acquiring metadata including location information indicating a location of an audio object and orientation information indicating a direction of the audio object, and audio data of the audio object; generating a playback signal for playing back the sound of the audio object at the listening position based on listening position information indicating the listening position, listener direction information indicating the direction of the listener at the listening position, the position information, the direction information, and the audio data; A program that causes a computer to execute a process that includes steps.

Images