Signal processing device, method, and program

Abstract

To provide a higher sense of realism.SOLUTION: A signal processing device includes an acquisition unit that acquires metadata including position information indicating a position of an audio object and orientation information indicating a direction of the audio object, and audio data of the audio object, and a signal generation unit that generates a playback signal for reproducing sound of the audio object at the listening position, on the basis of listening position information indicating a listening position, listener orientation information indicating the orientation of a listener at the listening position, the position information, the orientation information, and the audio data. The present technology can be applied to a transmission playback system.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present technology relates to a signal processing device, method, and program, and in particular to a signal processing device, method, and program that enable a higher sense of realism to be obtained. [Background technology]

[0002] For example, when reproducing a sound field from a free viewpoint, such as a bird's-eye view or walk-through, it is important to record target sounds such as human voices, the sounds of athletes' movements, such as the sound of a ball being kicked in sports, or the sounds of musical instruments, with as high a signal-to-noise ratio as possible.

[0003] At the same time, it is necessary to reproduce the sound with accurate localization for each sound source of the target sound, and to follow the sound image localization that accompanies the movement of the viewpoint or sound source.

[0004] Meanwhile, in the case of free viewpoint or fixed viewpoint content, there is a demand for technology that allows a higher sense of realism to be achieved, and many such technologies have been proposed.

[0005] For example, as a technology related to reproducing a sound field from a free viewpoint, a technology has been proposed in which, when a user can freely specify a listening position, gain correction and frequency characteristic correction are performed depending on the distance from the changed listening position to an audio object (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2015 / 107926 Summary of the Invention [Problem to be solved by the invention]

[0007] However, there are cases where the above-mentioned techniques are unable to provide a sufficiently high sense of realism.

[0008] For example, in the real world, sound sources are not point sources, but sound waves propagate from a sound-producing object with some size with specific directional characteristics including reflection and diffraction by the sound-producing object.

[0009] However, although there have been many attempts to record the sound field of a desired space, even when recording is done for each sound source, i.e., for each audio object, the orientation of each audio object is not taken into account on the playback side, and a sufficiently high level of realism may not be achieved.

[0010] The present technology has been made in view of such circumstances, and is intended to enable a higher sense of realism to be obtained. [Means for solving the problem]

[0011] A signal processing device according to one aspect of the present technology includes an acquisition unit that acquires metadata including position information indicating a position of an audio object and orientation information indicating a direction of the audio object, and audio data of the audio object, and a signal generation unit that generates a playback signal for reproducing the sound of the audio object at the listening position based on listening position information indicating a listening position, listener orientation information indicating the direction of the listener at the listening position, the position information, the orientation information, and the audio data.

[0012] A signal processing method or program of one aspect of the present technology includes a step of acquiring metadata including position information indicating a position of an audio object and orientation information indicating a direction of the audio object, as well as audio data of the audio object, and generating a playback signal that reproduces the sound of the audio object at the listening position based on listening position information indicating a listening position, listener orientation information indicating the orientation of the listener at the listening position, the position information, the orientation information, and the audio data.

[0013] In one aspect of the present technology, metadata including position information indicating the position of an audio object and orientation information indicating the direction of the audio object, and audio data of the audio object are acquired, and a playback signal for reproducing the sound of the audio object at the listening position is generated based on listening position information indicating the listening position, listener orientation information indicating the direction of the listener at the listening position, the position information, the orientation information, and the audio data. [Brief description of the drawings]

[0014] [Figure 1] FIG. 2 is a diagram illustrating the orientation of objects constituting a content. [Diagram 2] FIG. 2 is a diagram illustrating the directional characteristics of an object. [Diagram 3] FIG. 11 is a diagram illustrating an example of metadata syntax. [Figure 4] FIG. 13 is a diagram illustrating an example of the syntax of directional pattern data. [Diagram 5] FIG. 1 illustrates an example of the configuration of a signal processing device. [Figure 6] FIG. 4 is a diagram illustrating relative direction information. [Figure 7] FIG. 4 is a diagram illustrating relative direction information. [Figure 8] FIG. 4 is a diagram illustrating relative direction information. [Figure 9] FIG. 4 is a diagram illustrating relative direction information. [Figure 10] 11 is a flowchart illustrating a content playback process. [Figure 11] FIG. 1 illustrates an example of the configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, embodiments to which the present technology is applied will be described with reference to the drawings.

[0016] First embodiment About this technology This technology relates to a transmission and playback system that appropriately transmits directional characteristic data indicating the directional characteristics of an audio object, which is a sound source, and reflects the directional characteristics of the audio object in the content playback on the content playback side based on the directional characteristic data, thereby achieving a higher sense of realism.

[0017] For example, content that reproduces the sound of an audio object (hereinafter, also simply referred to as an object) that is a sound source can be classified into fixed viewpoint content and free viewpoint content.

[0018] In fixed viewpoint content, the position of the listener's viewpoint, i.e., the listening position (listening point), is a predetermined fixed position, whereas in free viewpoint content, the listener (user) can freely specify the listening position (viewpoint position) in real time.

[0019] In the real world, each sound source has its own directional characteristics. That is, even if a sound is emitted from the same sound source, the sound transmission characteristics differ depending on the direction seen from the sound source.

[0020] Therefore, when an object serving as a sound source in content or a listener at a listening position moves or rotates freely, the way in which the sound of the object is heard by the listener changes depending on the directional characteristics of the object.

[0021] In content playback, a process is generally performed to reproduce distance attenuation according to the distance from the listening position to the object. In contrast, this technology reproduces content taking into account not only distance attenuation but also the directional characteristics of the object, making it possible to achieve a higher sense of realism.

[0022] In other words, with this technology, when a listener or object moves or rotates freely, not only the distance between the listener and the object but also the relative direction (orientation) between the listener and the object is taken into consideration, and transmission characteristics are dynamically added to the content sound according to the distance attenuation and directional characteristics for each object.

[0023] For example, the addition of transfer characteristics is realized by gain correction according to distance attenuation and directional characteristics, and processing for wavefront synthesis based on the propagation characteristics of the amplitude and phase of a wavefront taking into account distance attenuation and directional characteristics.

[0024] In this technology, directional pattern data is used to add transfer characteristics according to the directional pattern, but if directional pattern data corresponding to each type of target sound source, i.e., object, is prepared, it is possible to obtain an even higher sense of realism.

[0025] For example, directional characteristic data for each type of object can be obtained by recording sound in advance using a microphone array or performing a simulation to determine the transmission characteristics in each direction and at each distance as the sound emitted from the object propagates through space.

[0026] The directional characteristic data for each type of object is transmitted in advance to the playback device together with the audio data of the content or separately from the audio data.

[0027] When the content is played back, the playback device uses the directional characteristic data to add a transfer characteristic according to the distance to the object and its directional characteristic to the audio data of the object, i.e., the playback signal for playing the sound of the content.

[0028] This makes it possible to play back content with a greater sense of realism.

[0029] With this technology, for each type of sound source (object), a transfer characteristic is added according to the relative positional relationship between the listener and the object, that is, the relative distance and direction. Therefore, even if the distance from the object to the listening position is the same, the way the sound of the object is heard changes depending on the direction from which the sound is heard, making it possible to reproduce a sound field closer to reality.

[0030] Examples of content suitable for application of this technology include the following:

[0031] -Content that recreates fields where team sports are played · Content that recreates the space of musicals, operas, plays, etc., where multiple performers are present · Content that recreates any space at a live venue or theme park -Content that plays performances by orchestras, marching bands, etc. Games and other content

[0032] For example, in the content of a performance by a marching band or the like, the performers may be stationary or may be moving.

[0033] Now, the present technology will be described in further detail below.

[0034] For example, consider an example of content that reproduces a sound field with an arbitrary position on a soccer field as the listening position.

[0035] In this case, for example, as shown in Figure 1, there are players and a referee on the field for each team, and these players and the referee are the sound sources, that is, the audio objects.

[0036] In the example shown in Figure 1, each circle in the figure represents a player or a referee, that is, an object, and the direction of the line segment added to each circle represents the direction in which the player or referee represented by that circle is facing, that is, the orientation of the object such as the player or referee.

[0037] Here, each object is in a different position and faces a different direction, and the positions and orientations of these objects change over time, i.e., each object moves and rotates over time.

[0038] For example, object OB11 is a judge, and the position of this object OB11 is the viewing point position (listening position), and the video and audio when the orientation of object OB11, which is the upward direction in the figure, is the line of sight, can be presented to the listener as content.

[0039] In the example in Figure 1, each object is placed on a two-dimensional plane, but in reality, the height of each object's mouth (player or referee), and the height of the feet from which the sound of a ball kick originates, are different from one another, and the postures of the objects are constantly changing.

[0040] That is, in reality, each object and viewing point (listening position) is placed in a three-dimensional space, and at the same time, those objects and listeners (users) at the viewing points are in various postures and facing various directions.

[0041] The cases in which directional characteristics according to the object orientation can be reflected in the content can be classified as follows:

[0042] (Case 1) When objects and listening positions are placed on a two-dimensional plane, only the azimuth angle (yaw) that indicates the direction of the object is taken into account, and the elevation angle (pitch) and tilt angle (roll) are not taken into account. (Case 2) When objects and listening positions are placed in a three-dimensional space, the azimuth and elevation angles that indicate the object's direction are taken into account, but the tilt angle that indicates the object's rotation is not taken into account. (Case 3) When objects and listening positions are placed in a three-dimensional space, and Euler angles, consisting of azimuth and elevation angles that indicate the direction of the object, and tilt angles that indicate the rotation of the object, are taken into consideration.

[0043] This technology can be applied to any of the above cases 1 to 3, and in each of these cases, content is played back taking into account the listening position, object placement, and object orientation and rotation (tilt), i.e., the rotation angle, as appropriate.

[0044] <Transmitting device> A transmission and playback system for transmitting and playing such content includes, for example, a transmitting device for transmitting content data, and a signal processing device that functions as a playback device for playing content based on the content data transmitted from the transmitting device. Note that there may be one or more signal processing devices that function as a playback device.

[0045] A transmitting device on the transmitting side of the transmission and playback system transmits, as content data, for example, audio data for reproducing the sound of one or more objects constituting the content, and metadata of each object (audio data).

[0046] Here, the metadata includes sound source type information, sound source position information, and sound source direction information.

[0047] The sound source type information is ID information that indicates the type of the object that is the sound source.

[0048] For example, the sound source type information may be information specific to the sound source indicating the type (kind) of the object itself that is the sound source, such as a player or an instrument, or it may be information indicating the type of sound emitted from the object, such as the voice of a player, the sound of a ball kick, the sound of applause, or other action sounds.

[0049] Alternatively, the sound source type information may be information indicating the type of the object itself and the type of sound emitted from the object.

[0050] Furthermore, directional characteristic data is prepared for each type indicated by the sound source type information, and on the playback side, a playback signal is generated based on the directional characteristic data determined for the sound source type information, so the sound source type information can also be said to be ID information indicating the directional characteristic data.

[0051] In the transmitting device, sound source type information is manually or otherwise assigned to each object constituting the content, and is included in the metadata of the object.

[0052] Moreover, the sound source position information included in the metadata is information that indicates the position of an object that is a sound source.

[0053] Here, the sound source location information may be, for example, latitude and longitude indicating an absolute position on the Earth's surface measured (acquired) by a position measurement module such as a GPS (Global Positioning System) module, or coordinates obtained by converting these latitude and longitude into distance.

[0054] Alternatively, the sound source position information may be any information indicating the position of an object, such as the coordinates of a coordinate system that uses a predetermined position within the space (target area) in which the content is to be recorded as the reference position.

[0055] Furthermore, when the sound source position information is expressed as coordinates (coordinate information), the coordinates may be coordinates of any coordinate system, such as a polar coordinate system consisting of azimuth angle, elevation angle, and radius, an xyz coordinate system, i.e., a three-dimensional Cartesian coordinate system, or a two-dimensional Cartesian coordinate system.

[0056] Furthermore, the sound source direction information included in the metadata is information that indicates the absolute direction in which an object located at the position indicated by the sound source position information is facing, that is, the direction in front of the object.

[0057] In addition, the sound source direction information may include not only information indicating the orientation of an object, but also information indicating the rotation (tilt) of the object. In the following, the sound source direction information is assumed to include information indicating the orientation of an object and information indicating the rotation of the object.

[0058] Specifically, for example, the sound source direction information includes an azimuth angle ψ indicating the direction of an object in a coordinate system as sound source position information. o and the elevation angle θ o and the tilt angle φ, which indicates the rotation (tilt) of the object in the coordinate system of the coordinates as the sound source position information. o The following are included:

[0059] In other words, the sound source direction information is the azimuth angle ψ that indicates the absolute direction and rotation of the object. o (yaw), elevation angle θ o (pitch), and the inclination angle φ o For example, sound source direction information can be obtained from a geomagnetic sensor attached to an object, video data in which the object is the subject, or the like.

[0060] In the transmitting device, sound source position information and sound source direction information are generated for each object for each discretized unit of time, such as for each frame of audio data or for each predetermined number of frames, that is, at predetermined time intervals.

[0061] Then, metadata including sound source type information, sound source position information, and sound source direction information is transmitted (transmitted) to the signal processing device together with the audio data of the object for each unit time, such as for each frame.

[0062] Furthermore, the transmitting device transmits (transmits) the directional characteristic data to the signal processing device on the playback side in advance or sequentially for each sound source type indicated by the sound source type information. Note that the signal processing device may obtain the directional characteristic data from a device or the like different from the transmitting device.

[0063] The directional characteristic data is data indicating the directional characteristic of an object of the sound source type indicated by the sound source type information, that is, the transmission characteristic in each direction as seen from the object.

[0064] For example, as shown in FIG. 2, each sound source has a directional characteristic that is specific to that sound source.

[0065] In the example shown in FIG. 2, for example, a whistle as a sound source has directional characteristics in which sound propagates strongly in the front (forward) direction as indicated by arrow Q11, that is, has sharp front directivity.

[0066] Furthermore, for example, footsteps emitted from spikes or the like as a sound source have directional characteristics (omnidirectional) in which the sound propagates with equal strength in all directions, as indicated by arrow Q12.

[0067] Furthermore, for example, the sound emitted from the mouth of a player as a sound source has directional characteristics in which the sound propagates strongly in the front and sides as shown by arrow Q13, that is, has a fairly strong front directivity.

[0068] Such directional pattern data showing the directional pattern of a sound source can be obtained by acquiring the characteristics of sound propagation (transmission characteristics) to the surroundings for each type of sound source using a microphone array in, for example, an anechoic chamber, etc. In addition, the directional pattern data can also be obtained by performing a simulation on 3D data that mimics the shape of the sound source.

[0069] Specifically, the directional characteristic data is a gain function dir(i, ψ, θ) defined as a function of the azimuth angle ψ and elevation angle θ indicating the direction as seen from the sound source, for example, determined for an ID value i indicating the type of sound source.

[0070] Furthermore, in addition to the azimuth angle ψ and the elevation angle θ, a gain function dir(i, d, ψ, θ) having a discretized distance d from a sound source as an argument may be used as the directivity data.

[0071] In this case, by substituting each argument into the gain function dir(i, d, ψ, θ), a gain value indicating the transfer characteristics (propagation characteristics) of the sound is obtained as the output of the gain function dir(i, d, ψ, θ).

[0072] This gain value indicates the characteristics (transmission characteristics) of sound that is emitted from a sound source of a sound source type with an ID value of i, propagates in the direction of azimuth angle ψ and elevation angle θ from the sound source, and arrives at a position at a distance d from the sound source (hereinafter referred to as position P).

[0073] Therefore, by performing gain correction on the audio data of a sound source type having an ID value of i based on this gain value, it is possible to reproduce (reproduce) the sound from the sound source of the sound source type having an ID value of i that would actually be heard at position P.

[0074] In particular, in this example, by using the gain value that is the output of the gain function dir(i, d, ψ, θ), it is possible to realize gain correction that adds a transfer characteristic indicated by the directional characteristic that also takes into account the distance from the sound source, i.e., distance attenuation.

[0075] The directional characteristic data may be a gain function indicating a transfer characteristic taking into consideration reverberation characteristics, etc. Alternatively, the directional characteristic data may be Ambisonics format data, that is, data consisting of spherical harmonic coefficients (spherical harmonic spectrum) in each direction.

[0076] The transmitting device transmits the directional characteristic data prepared for each sound source type as described above to the signal processing device on the reproduction side.

[0077] Here, a specific example of the transmission of metadata and directional characteristic data will be described.

[0078] For example, it is conceivable to prepare metadata for each frame of a predetermined duration of audio data of an object, and transmit the metadata for each frame to the playback side using the bitstream syntax shown in Figure 3. In Figure 3, uimsbf is an unsigned integer MSB first, and tcimsbf is a two's complement integer MSB first.

[0079] In the example of FIG. 3, the metadata includes, for each object constituting the content, sound source type information "Object_type_index", sound source position information "Object_position[3]", and sound source direction information "Object_direction[3]".

[0080] In particular, in this example, the sound source position information Object_position[3] is the coordinates (x o ,y o ,z o ) is the coordinate (x o ,y o ,z o ) indicates the absolute position of an object in the xyz coordinate system, i.e., object space.

[0081] The sound source direction information Object_direction[3] is an azimuth angle ψ o , elevation angle θ o , and the tilt angle φ o It consists of:

[0082] For example, in the case of free viewpoint content, the viewpoint (listening position) changes over time during content playback, so it is advantageous for generating playback signals to express the position of an object using coordinates that indicate an absolute position rather than relative coordinates based on the listening position.

[0083] In contrast, for example, if the content is from a fixed viewpoint, the sound source position information indicating the position of the object can be the coordinates of a polar coordinate system consisting of the azimuth and elevation angles indicating the direction of the object as seen from the listening position, and a radius indicating the distance from the listening position to the object.

[0084] The configuration of the metadata is not limited to the example shown in Fig. 3, and may be any other configuration. Also, the metadata only needs to be transmitted at a predetermined time interval, and does not necessarily need to be transmitted for each frame.

[0085] Furthermore, the directional characteristic data for each sound source type may be stored in metadata and transmitted, or may be transmitted in advance separately from the metadata and audio data, for example in the bitstream syntax shown in FIG. 4.

[0086] In the example of Figure 4, the directional characteristic data corresponding to the value of the specified sound source type information is transmitted as a gain function "Object_directivity[distance][azimuth][elevation]" whose arguments are the distance from the sound source "distance," and the azimuth and elevation angles "azimuth" indicating the direction as seen from the sound source.

[0087] In addition, the directional characteristic data may be in a format in which the sampling intervals of the azimuth angle and elevation angle, which are arguments, are not equal angular intervals, or may be in HOA (Higher Order Ambisonics) format, i.e., Ambisonics format data (spherical harmonic coefficients).

[0088] For example, for directional characteristic data of general sound source types, it is advisable to transmit the directional characteristic data to the playback side in advance.

[0089] On the other hand, for directional pattern data of sound sources having unusual directional patterns, such as objects that are not predefined, it is conceivable that the directional pattern data is included in the metadata shown in Figure 3 and transmitted as metadata.

[0090] In this manner, the metadata, audio data, and directional characteristic data are transmitted from the transmitting device to the signal processing device on the playback side.

[0091] <Configuration example of signal processing device> Next, a signal processing device, which is a device on the reproducing side, will be described.

[0092] For example, a signal processing device on the playback side is configured as shown in FIG.

[0093] The signal processing device 11 shown in Figure 5 generates a playback signal for playing back the sound of content (object) at the listening position based on directional characteristic data that has been obtained in advance from a transmitting device or the like or that has been shared in advance, and outputs the playback signal to the playback unit 12.

[0094] For example, the signal processing device 11 generates a playback signal by using the directional characteristic data to perform processing for vector based amplitude panning (VBAP) or wave field synthesis, convolution processing of a head related transfer function (HRTF), and the like.

[0095] The playback unit 12 is composed of, for example, headphones, earphones, a speaker array consisting of two or more speakers, and plays back the sound of the content based on the playback signal supplied from the signal processing device 11.

[0096] The signal processing device 11 also includes an acquisition unit 21, a listening position designation unit 22, a directional characteristic database unit 23, and a signal generation unit 24.

[0097] The acquisition unit 21 acquires directional pattern data, metadata, and audio data, for example, by receiving data transmitted from a transmitting device or reading data from a transmitting device connected by wire or the like.

[0098] The timing of acquiring the directional characteristic data and the timing of acquiring the metadata and audio data may be the same or different.

[0099] The acquisition unit 21 supplies the acquired directional pattern data and metadata to a directional pattern database unit 23 , and also supplies the acquired metadata and audio data to a signal generation unit 24 .

[0100] The listening position designation unit 22 designates a listening position in the target space and the orientation of a listener (user) at that listening position, and supplies listening position information indicating the listening position and listener orientation information indicating the orientation of the listener to the signal generation unit 24 as the designation results.

[0101] The directional characteristic database unit 23 records the directional characteristic data for each of the multiple sound source types supplied from the acquisition unit 21 .

[0102] In addition, when the directional characteristic database unit 23 receives sound source type information included in the metadata from the acquisition unit 21, the directional characteristic database unit 23 supplies the signal generation unit 24 with directional characteristic data of the sound source type indicated by the supplied sound source type information from among the multiple directional characteristic data recorded.

[0103] The signal generation unit 24 generates a playback signal based on the metadata and audio data supplied from the acquisition unit 21, the listening position information and listener direction information supplied from the listening position designation unit 22, and the directional characteristic data supplied from the directional characteristic database unit 23, and supplies the playback signal to the playback unit 12.

[0104] The signal generating unit 24 includes a relative distance calculating unit 31 , a relative direction calculating unit 32 , and a directivity rendering unit 33 .

[0105] The relative distance calculation unit 31 calculates the relative distance between the listening position (listener) and the object based on the sound source position information included in the metadata supplied from the acquisition unit 21 and the listening position information supplied from the listening position designation unit 22, and supplies relative distance information indicating the calculation result to the directivity rendering unit 33.

[0106] The relative direction calculation unit 32 calculates the relative direction between the listener and the object based on the sound source position information and sound source direction information contained in the metadata supplied from the acquisition unit 21 and the listening position information and listener direction information supplied from the listening position designation unit 22, and supplies relative direction information indicating the calculation result to the directivity rendering unit 33.

[0107] The directivity rendering unit 33 performs rendering processing based on the audio data supplied from the acquisition unit 21, the directional characteristic data supplied from the directional characteristic database unit 23, the relative distance information supplied from the relative distance calculation unit 31, the relative direction information supplied from the relative direction calculation unit 32, and the listening position information and listener direction information supplied from the listening position designation unit 22.

[0108] The directivity rendering unit 33 supplies the playback signal obtained by the rendering process to the playback unit 12, and plays back the sound of the content. For example, the directivity rendering unit 33 performs rendering processes such as VBAP, processes for wave field synthesis, and HRTF convolution processes.

[0109] <About each part of the signal processing device> (Listening position designation section) Next, each unit of the signal processing device 11 will be described in more detail.

[0110] The listening position designation unit 22 designates the listening position and the direction of the listener in response to a user operation or the like.

[0111] For example, if the content is of a free viewpoint, the user watching the content, i.e., the listener, may specify an arbitrary listening position or orientation by operating a GUI (Graphical User Interface) or the like in a running service or application.

[0112] In this case, the listening position designation unit 22 sets the listening position and the listener's orientation designated by the user as the listening position (viewpoint position) that is the viewpoint of the content and the direction the listener is facing, that is, the listener's orientation.

[0113] Also, for example, when a user specifies a desired player from among a plurality of predetermined players, the position and orientation of the player may be set as the listening position and orientation of the listener.

[0114] Furthermore, the listening position designation unit 22 may execute some kind of automatic route designation program or obtain information indicating the user's position and orientation from a head-mounted display in which the playback unit 12 is provided, thereby allowing the designation of any listening position and orientation of the listener without requiring user operation.

[0115] In this way, in free viewpoint content, the listening position and the orientation of the listener are arbitrary positions and orientations that can change over time.

[0116] On the other hand, for fixed viewpoint content, the listening position designation unit 22 designates a predetermined fixed position and fixed orientation as the listening position and orientation of the listener.

[0117] As a specific example of the listening position information indicating the listening position, for example, the coordinates (x,y,z) indicating the listening position in the xyz coordinate system indicating the absolute position on the Earth's surface or the xyz coordinate system indicating the absolute position in the target space are v ,y v ,z v ) is possible.

[0118] For example, the listener direction information is an azimuth angle ψ v and the elevation angle θ v and the tilt angle φ, which is the angle of absolute rotation (tilt) of the listener in the xyz coordinate system. v The information can be the Euler angles.

[0119] In particular, in this case, when the content is of a fixed viewpoint, for example, the listening position information (x v ,y v ,z v )=(0,0,0), and the listener direction information (ψ v ,θ v ,φ v )=(0,0,0).

[0120] In the following, the listening position information is expressed as coordinates (x v ,y v ,z v ), and the listener direction information is the Euler angle (ψ v ,θ v ,φ v ) will be assumed in the following explanation.

[0121] Similarly, the sound source position information is expressed as coordinates (x o ,y o ,z o ), and the sound source direction information is the Euler angle (ψ o ,θ o ,φ o ) will be assumed in the following explanation.

[0122] (Relative distance calculation part) In the relative distance calculation unit 31, for each object constituting the content, the distance from the listening position to the object is calculated as a relative distance d o It is calculated as:

[0123] Specifically, the relative distance calculation unit 31 calculates the listening position information (x v ,y v ,z v ) and sound source location information (x o ,yo ,z o ) based on the above, the relative distance d is calculated by the following formula (1): o Calculate the relative distance d o The relative distance information indicating the distance is output.

[0124]

number

[0125] (Relative direction calculation section) Furthermore, the relative direction calculation unit 32 determines relative direction information that indicates the relative direction between the listener and the object.

[0126] For example, the relative orientation information includes the object azimuth angle ψ i_obj , object elevation angle θ i_obj , Object rotation azimuth angle ψ_rot i_obj , and the object rotation elevation angle θ_rot i_obj Contains:

[0127] Here, the object azimuth angle ψ i_obj and the object elevation angle θ i_obj are the azimuth and elevation angles, respectively, that indicate the relative direction of the object as seen by the listener.

[0128] Listening location information (x v ,y v ,z v ) is the origin, and the listener's direction information (ψ v ,θ v ,φ v ) is called the listener coordinate system. In the listener coordinate system, the direction of the listener, that is, the direction in front of the listener, is the +y direction.

[0129] In this case, the azimuth angle and elevation angle indicating the direction of the object in the listener coordinate system are the object azimuth angle ψ i_obj and the object elevation angle θ i_obj It becomes.

[0130] Similarly, the object rotation azimuth angle ψ_rot i_obj and the object rotation elevation angle θ_rot i_obj are the azimuth and elevation angles that indicate the relative direction of the listener (listening position) from the object. In other words, the object rotation azimuth ψ_rot i_obj and the object rotation elevation angle θ_rot i_obj It can be said that the information indicates how much the front direction of the object has rotated with respect to the listener.

[0131] Sound source location information (x o ,y o ,z o ) is the origin, and the sound source direction information (ψ o ,θ o ,φ o ) is called the object coordinate system. In the object coordinate system, the orientation of the object, that is, the direction of the front of the object, is the +y direction.

[0132] At this time, the azimuth angle and elevation angle indicating the direction of the listener (listening position) in the object coordinate system are the object rotation azimuth angle ψ_rot i_obj and the object rotation elevation angle θ_rot i_obj It becomes.

[0133] These objects rotate azimuth ψ_rot i_obj and the object rotation elevation angle θ_rot i_obj are the azimuth angle and elevation angle when referring to the directional characteristic data during rendering processing.

[0134] In the following description, the azimuth angle in each three-dimensional orthogonal coordinate system, such as the xyz coordinate system of the target space, the listener coordinate system, and the object coordinate system, is positive in the clockwise direction from the front direction (+y direction).

[0135] For example, in the xyz coordinate system, after projecting a target point of an object onto the xy plane, the angle indicating the position (direction) of the target point after projection based on the +y direction on the xy plane, that is, the angle between the direction of the target point after projection and the +y direction, is the azimuth angle. In this case, the clockwise direction from the +y direction is the positive direction.

[0136] In addition, in the listener coordinate system and the object coordinate system, the orientation of the listener or the object, that is, the front direction of the listener or the object, is the +y direction.

[0137] The elevation angle in each three-dimensional orthogonal coordinate system, such as the xyz coordinate system of the target space, the listener coordinate system, and the object coordinate system, is defined as a positive direction in the upward direction.

[0138] For example, in the xyz coordinate system, the elevation angle is the angle between a line passing through the origin of the xyz coordinate system and a target point such as an object, and the xy plane.

[0139] In addition, when a target point of an object or the like is projected onto the xy plane, if the plane containing the origin of the xyz coordinate system, the target point, and the projected target point is defined as plane A, then on plane A, the +z direction from the xy plane is considered to be the positive direction of the elevation angle.

[0140] In addition, for example, in the listener coordinate system or the object coordinate system, the object or the listening position is the target point.

[0141] Furthermore, the tilt angle in each three-dimensional Cartesian coordinate system, such as the xyz coordinate system of the target space, the listener coordinate system, and the object coordinate system, is considered to be a positive rotation when, after the elevation angle rotation, it rotates to the upper right with the +y direction as the front direction.

[0142] Note that, although the azimuth, elevation, and tilt angles indicating the listening position and the direction of an object in a three-dimensional Cartesian coordinate system have been defined as above, generality is not lost even if other definitions are used, such as when quaternions or rotation matrices are used.

[0143] Here, the relative distance d o and object azimuth angle ψ i_obj , object elevation angle θ i_obj , Object rotation azimuth angle ψ_rot i_obj , Object rotation elevation angle θ_rot i_obj A specific example of the above will be described.

[0144] First, a case will be described in which only the azimuth angle is taken into consideration in the sound source direction information and the listener direction information, and the elevation angle and tilt angle are not taken into consideration, that is, a two-dimensional case.

[0145] For example, as shown in FIG. 6, assume that the position of point P21 in an xy coordinate system with origin O as a reference is the listening position, and an object is located at point P22.

[0146] Also, the direction of a line segment W11 passing through point P21, more specifically, the direction from point P21 to the end point of line segment W11 on the opposite side to point P21, is assumed to be the direction indicating the orientation of the listener.

[0147] Similarly, the direction of a line segment W12 passing through point P22 is assumed to be the direction indicating the orientation of the object. Furthermore, a straight line passing through points P21 and P22 is assumed to be a straight line L11.

[0148] In this case, the distance between points P21 and P22 is the relative distance d o It is said.

[0149] The angle between the line segment W11 and the straight line L11, that is, the angle indicated by the arrow K11, is the object azimuth angle ψ i_obj Similarly, the angle between the line segment W12 and the straight line L11, that is, the angle indicated by the arrow K12, is the object rotation azimuth angle ψ_rot i_obj It becomes.

[0150] Furthermore, if the target space is three-dimensional, the relative distance d o and object azimuth angle ψ i_obj , object elevation angle θ i_obj , Object rotation azimuth angle ψ_rot i_obj, Object rotation elevation angle θ_rot i_obj 7 to 9. In addition, in Fig. 7 to Fig. 9, the same reference numerals are used to designate corresponding parts, and the description thereof will be omitted as appropriate.

[0151] For example, as shown in FIG. 7, in an xyz coordinate system with origin O as a reference, the positions of points P31 and P32 are the listening position and the position of an object, respectively, and a straight line passing through points P31 and P32 is defined as a straight line L31.

[0152] In addition, the xy plane of the xyz coordinate system is used as the listener's orientation information (ψ v ,θ v ,φ v ), and then the origin O is rotated by the angle indicated by the listening position information (x v ,y v ,z v ) is translated to the position indicated by PF11. This plane PF11 is the xy plane of the listener coordinate system.

[0153] Similarly, the xy plane of the xyz coordinate system is expressed as the sound source direction information (ψ o ,θ o ,φ o ), and then the origin O is rotated by the angle indicated by the sound source position information (x o ,y o ,z o ) is translated to the position indicated by the arrow PF12. This plane PF12 is the xy plane of the object coordinate system.

[0154] The direction of the line segment W21 passing through the point P31, more specifically, the direction from the point P31 to the end point of the line segment W21 opposite to the point P31, is the listener direction information (ψ v ,θ v ,φ v ) is the direction indicating the orientation of the listener.

[0155] Similarly, the direction of the line segment W22 passing through the point P32 is the sound source direction information (ψ o ,θ o ,φ o) is the direction indicating the orientation of the object.

[0156] In this case, the distance between points P31 and P32 is the relative distance d o It is said.

[0157] As shown in FIG. 8, if the straight line L31 projected onto the plane PF11 is a straight line L41, the angle between the straight line L41 and the line segment W21 on the plane PF11, that is, the angle indicated by the arrow K21, is the object azimuth angle ψ i_obj It becomes.

[0158] Furthermore, the angle between the straight lines L41 and L31, i.e., the angle indicated by the arrow K22, is the object elevation angle θ i_obj In other words, the object elevation angle θ i_obj is the angle between the plane PF11 and the straight line L31.

[0159] On the other hand, as shown in FIG. 9, if the straight line L31 projected onto the plane PF12 is a straight line L51, the angle between the straight line L51 and the line segment W22 on the plane PF12, that is, the angle indicated by the arrow K31, is the object rotation azimuth angle ψ_rot i_obj It becomes.

[0160] Furthermore, the angle between the straight line L51 and the straight line L31, that is, the angle indicated by the arrow K32, is the object rotation elevation angle θ_rot i_obj In other words, the object rotation elevation angle θ_rot i_obj is the angle between the plane PF12 and the straight line L31.

[0161] The object azimuth angle ψ i_obj , object elevation angle θ i_obj , Object rotation azimuth angle ψ_rot i_obj , and the object rotation elevation angle θ_rot i_obj Specifically, the relative direction information can be calculated, for example, as follows.

[0162] For example, a rotation matrix describing rotation in three-dimensional space is shown in the following equation (2).

[0163]

number

[0164] In addition, in equation (2), the coordinates (x, y, z) in the X1Y1Z1 space, which is a space of a three-dimensional Cartesian coordinate system with the X1, Y1, and Z1 axes as its axes, are rotated by the rotation matrix, and the rotated coordinates (x', y', z') are obtained.

[0165] That is, in the calculation shown in formula (2), the second matrix from the right on the right side is a rotation matrix that rotates the X1Y1Z1 space by angle φ around the Z1 axis in the X1Y1 plane to obtain the X2Y2Z1 space after rotation. In other words, the coordinates (x, y, z) are rotated by angle -φ on the X1Y1 plane by the second rotation matrix from the right on the right side.

[0166] Moreover, the third matrix from the right on the right side of equation (2) is a rotation matrix that rotates the X2Y2Z1 space by an angle θ around the X2 axis in the Y2Z1 plane to obtain the rotated X2Y3Z2 space.

[0167] Furthermore, the fourth matrix from the right on the right side of equation (2) is a rotation matrix that rotates an angle ψ around the Y3 axis in the X2Z2 plane in the X2Y3Z2 space to obtain the rotated X3Y3Z3 space.

[0168] In the relative direction calculation unit 32, the rotation matrix shown in equation (2) is used to generate relative direction information.

[0169] Specifically, the relative direction calculation unit 32 calculates the sound source position information (x o ,y o ,z o ) and listener direction information (ψ v ,θ v ,φ v ) to calculate the coordinates (xo ,y o ,z o ) coordinates after rotation (x o ',y o ',z o ').

[0170]

number

[0171] In the calculation of equation (3), φ=-φ v , θ=-θ v , and ψ = -ψ v Then, the rotation matrix is ​​used for calculation.

[0172] The coordinates thus obtained (x o ',y o ',z o ') are coordinates that indicate the position of the object in the listener's coordinate system. However, the origin of the listener's coordinate system here is not the listening position but the origin O of the xyz coordinate system of the target space.

[0173] Next, the relative direction calculation unit 32 calculates the listening position information (x v ,y v ,z v ) and listener direction information (ψ v ,θ v ,φ v ) is calculated using the following formula (4), and the coordinates (x v ,y v ,z v ) coordinates after rotation (x v ',y v ',z v ').

[0174]

number

[0175] In the calculation of equation (4), φ=-φ v , θ=-θ v , and ψ = -ψ vThen, the rotation matrix is ​​used for calculation.

[0176] The coordinates thus obtained (x v ',y v ',z v ') are coordinates that indicate the listening position in the listener coordinate system. However, the origin of the listener coordinate system here is not the listening position but the origin O of the xyz coordinate system of the target space.

[0177] In addition, the relative direction calculation unit 32 calculates the coordinates (x o ',y o ',z o ') and the coordinates (x v ',y v ',z v Based on this, the following equation (5) is calculated:

[0178]

number

[0179] By calculating equation (5), the coordinates (x o ",y o ",z o The coordinates (x o ",y o ",z o '') are coordinates that indicate the relative position of an object as seen by the listener.

[0180] The relative direction calculation unit 32 calculates the coordinates (x o ",y o ",z o Based on the above, the following equations (6) and (7) are calculated to obtain the object azimuth angle ψ i_obj and the object elevation angle θ i_obj get.

[0181]

number

[0182]

number

[0183] In equation (6), the x and y coordinates are x o '' and y o '' based on the object azimuth angle ψ i_obj is required.

[0184] In more detail, when calculating equation (6), y o '' sign and x o Based on the result of the 0 judgment for '', a case judgment process is performed, and depending on the result of the case judgment, an exception is thrown to determine the object azimuth angle ψ i_obj is calculated, but a detailed description thereof will be omitted here.

[0185] In addition, in equation (7), the coordinate (x o ",y o ",z o '') based on the object elevation angle θ i_obj In more detail, when calculating equation (7), z o '' sign and (x o '' 2 +y o '' 2 ) is judged to be 0, and exception handling is performed depending on the result of the judgment. i_obj is calculated, but a detailed description thereof will be omitted here.

[0186] From the above calculation, the object azimuth angle ψ i_obj and the object elevation angle θ i_obj is calculated, the relative orientation calculation unit 32 performs a similar calculation to obtain the object rotation azimuth angle ψ_rot i_obj and the object rotation elevation angle θ_rot i_obj Request.

[0187] That is, the relative direction calculation unit 32 calculates the listening position information (x v ,y v ,z v ) and sound source direction information (ψ o ,θ o ,φ o ) to calculate the coordinates (x v ,y v ,z v ) coordinates after rotation (x v ',y v ',z v ').

[0188]

number

[0189] In the calculation of equation (8), φ=-φ o , θ=-θ o , and ψ = -ψ o Then, the rotation matrix is ​​used for calculation.

[0190] The coordinates thus obtained (x v ',y v ',z v ') is a coordinate that indicates the listening position (the position of the listener) in the object coordinate system. However, the origin of the object coordinate system here is not the position of the object but the origin O of the xyz coordinate system of the target space.

[0191] Next, the relative direction calculation unit 32 calculates the sound source position information (x o ,y o ,z o ) and sound source direction information (ψ o ,θ o ,φ o ) to calculate the coordinates (x o ,y o ,z o ) coordinates after rotation (x o ',y o ',z o ').

[0192]

number

[0193] In the calculation of equation (9), φ=-φ o , θ=-θ o , and ψ = -ψ o Then, the rotation matrix is ​​used for calculation.

[0194] The coordinates thus obtained (x o ',y o ',z o ') are coordinates that indicate the position of the object in the object coordinate system. However, the origin of the object coordinate system here is not the object's position but the origin O of the xyz coordinate system of the target space.

[0195] In addition, the relative direction calculation unit 32 calculates the coordinates (x v ',y v ',z v ') and the coordinates (x o ',y o ',z o Based on this, the following equation (10) is calculated.

[0196]

number

[0197] By calculating equation (10), the coordinates (x v ",y v ",z v The coordinates (x v ",y v ",z v '') are coordinates that indicate the relative position of the listening position from the object's perspective.

[0198] The relative direction calculation unit 32 calculates the coordinates (x v ",yv ",z v '') to calculate the following equations (11) and (12), and the object rotation azimuth angle ψ_rot i_obj and the object rotation elevation angle θ_rot i_obj get.

[0199]

number

[0200]

number

[0201] In equation (11), the same calculation as in equation (6) is performed, and the object rotation azimuth angle ψ_rot i_obj In addition, in equation (12), the same calculation as in equation (7) is performed, and the object rotation elevation angle θ_rot i_obj is required.

[0202] The relative direction calculation unit 32 performs the above-described processing for each frame of audio data for a plurality of objects.

[0203] This allows us to calculate the object azimuth angle ψ for each object for each frame. i_obj , object elevation angle θ i_obj , Object rotation azimuth angle ψ_rot i_obj , and the object rotation elevation angle θ_rot i_obj The relative direction information obtained is

[0204] By using the relative orientation information obtained in this way, the sound image of each object can be localized in accordance with the listening position, the listener's orientation, and the movement and rotation of the object, resulting in a greater sense of realism.

[0205] (Directional Pattern Database Section) The directional characteristic database unit 23 records directional characteristic data for each type of object, that is, for each type of sound source.

[0206] This directivity data is a function that uses the azimuth angle and elevation angle as viewed from the object as arguments to obtain the gain and spherical harmonic coefficient of the propagation direction indicated by the azimuth angle and elevation angle.

[0207] In addition, the directional characteristic data may be data in table format rather than a function, i.e., a table in which the azimuth angle and elevation angle as seen from the object correspond to the gain and spherical harmonic coefficient of the propagation direction indicated by those azimuth angle and elevation angle.

[0208] (Directional Rendering) The directional rendering unit 33 performs rendering processing based on the audio data of each object, the directional characteristic data, relative distance information, and relative orientation information obtained for each object, as well as the listening position information and listener orientation information, and generates a playback signal corresponding to the target device, the playback unit 12.

[0209] <Explanation of Content Playback Processing> Next, the operation of the signal processing device 11 will be described.

[0210] That is, the content reproduction process performed by the signal processing device 11 will be described below with reference to the flowchart of FIG.

[0211] In this description, it is assumed that the content to be played back is free viewpoint content, and that directional pattern data for each sound source type is acquired and recorded in the directional pattern database unit 23 in advance.

[0212] In step S11, the acquisition unit 21 acquires metadata and audio data for one frame of each object constituting the content from the transmission device. In other words, metadata and audio data for each predetermined time interval are acquired.

[0213] The acquisition unit 21 supplies the sound source type information included in the metadata of each acquired object to the directional characteristic database unit 23, and also supplies the audio data of each acquired object to the directivity rendering unit 33.

[0214] The acquisition unit 21 also acquires the sound source position information (x o ,y o ,z o ) to the relative distance calculation unit 31 and the relative direction calculation unit 32, and also supplies the sound source direction information (ψ o ,θ o ,φ o ) is supplied to the relative direction calculation unit 32.

[0215] In step S12, the listening position designation unit 22 designates the listening position and the orientation of the listener.

[0216] That is, the listening position designation unit 22 determines the listening position and the direction of the listener in response to the listener's operation or the like, and outputs listening position information (x v ,y v ,z v ) and listener direction information (ψ v ,θ v ,φ v )

[0217] The listening position designation unit 22 uses the obtained listening position information (x v ,y v ,z v ) is supplied to a relative distance calculation unit 31, a relative direction calculation unit 32, and a directivity rendering unit 33, and the obtained listener direction information (ψ v ,θ v ,φ v ) is supplied to the relative orientation calculation unit 32 and the directivity rendering unit 33.

[0218] When the content is for a fixed viewpoint, for example, the listening position information is set to (0,0,0), and the listener direction information is also set to (0,0,0).

[0219] In step S13, the relative distance calculation unit 31 receives the sound source position information (x o ,y o ,z o ), and the listening position information (x v ,y v ,z v ) and the relative distance d based on o and supplies relative distance information indicating the calculation result to the directional rendering unit 33. For example, in step S13, the above-mentioned formula (1) is calculated for each object, and the relative distance d o is calculated.

[0220] In step S14, the relative direction calculation unit 32 receives the sound source position information (x o ,y o ,z o ) and sound source direction information (ψ o ,θ o ,φ o ), and the listening position information (x v ,y v ,z v ) and listener direction information (ψ v ,θ v ,φ v ) and supplies relative direction information indicating the calculation result to the directivity rendering unit 33.

[0221] For example, the relative orientation calculation unit 32 calculates the above-mentioned formulas (3) to (7) for each object to obtain an object azimuth angle ψ i_obj and the object elevation angle θ i_obj Calculate.

[0222] In addition, for example, the relative orientation calculation unit 32 calculates the above-mentioned formulas (8) to (12) for each object to obtain an object rotation azimuth angle ψ_rot i_obj and the object rotation elevation angle θ_rot i_objCalculate.

[0223] The relative azimuth calculation unit 32 calculates the object azimuth angle ψ i_obj , object elevation angle θ i_obj , Object rotation azimuth angle ψ_rot i_obj , and the object rotation elevation angle θ_rot i_obj The information consisting of the above is supplied to the directivity rendering unit 33 as relative direction information.

[0224] In step S15, the directivity rendering unit 33 acquires the directivity data from the directivity database unit .

[0225] For example, when metadata is acquired for each object in step S11 and sound source type information included in the metadata is supplied to the directional pattern database unit 23, the directional pattern database unit 23 outputs directional pattern data for each object.

[0226] That is, for each sound source type information supplied from the acquisition unit 21, the directional characteristic database unit 23 reads out the directional characteristic data of the sound source type indicated by the sound source type information from among the multiple directional characteristic data recorded, and outputs the directional characteristic data to the directional rendering unit 33.

[0227] The directivity rendering unit 33 obtains the directivity data of each object by acquiring the directivity data output from the directivity database unit 23 for each object in this manner.

[0228] In step S16, the directivity rendering unit 33 converts the audio data supplied from the acquisition unit 21, the directional characteristic data supplied from the directional characteristic database unit 23, the relative distance information supplied from the relative distance calculation unit 31, the relative direction information supplied from the relative direction calculation unit 32, and the listening position information (x v ,y v ,z v ) and listener direction information (ψ v ,θ v,φ v ) to perform rendering processing.

[0229] In addition, the listening position information (x v ,y v ,z v ) and listener direction information (ψ v ,θ v ,φ v ) may be used in the rendering process as necessary, but is not necessarily used in the rendering process.

[0230] For example, the directional rendering unit 33 performs rendering processing such as VBAP, processing for wave field synthesis, and HRTF convolution processing to generate a playback signal for playing back the sound of an object (content) at the listening position.

[0231] Here, an example in which VBAP is performed as the rendering process will be described. In this case, therefore, it is assumed that the playback unit 12 is configured with a plurality of speakers.

[0232] For simplicity of explanation, the case where the content includes one object will be explained as an example.

[0233] First, the directivity rendering unit 33 calculates the relative distance d o Based on this, the following equation (13) is calculated, and the gain value gain for reproducing the distance attenuation is calculated. i_obj Calculate.

[0234]

number

[0235] In addition, in equation (13), power (d o ,2.0) is the relative distance d o Here, an example in which the distance square law is used will be described, but the calculation of the gain value that reproduces the distance attenuation is not limited to this, and any other method may be used.

[0236] Next, the directional rendering unit 33 calculates, for example, the object rotation azimuth angle ψ_rot i_obj and the object rotation elevation angle θ_rot i_obj The gain value dir_gain according to the directional characteristic of the object is calculated by the following equation (14) based on i_obj Calculate.

[0237]

number

[0238] In equation (14), dir(i, ψ_rot i_obj ,θ_rot i_obj ) indicates a gain function corresponding to the value i of the sound source type information supplied as directional characteristic data.

[0239] Therefore, in the calculation of equation (14), the directional rendering unit 33 calculates the object rotation azimuth angle ψ_rot i_obj , and the object rotation elevation angle θ_rot i_obj The gain function is substituted and the calculation is performed, and the gain value dir_gain is calculated as the result. i_obj get.

[0240] That is, in equation (14), the object rotation azimuth angle ψ_rot i_obj , and the object rotation elevation angle θ_rot i_obj and the directional characteristic data to obtain the gain value dir_gain i_obj is obtained.

[0241] The gain value dir_gain thus obtained i_obj This realizes gain correction for adding the transmission characteristics of sound propagating from an object to a listener, in other words, gain correction for reproducing sound propagation according to the directional characteristics of the object.

[0242] As described above, the argument (variable) of the gain function as the directional characteristic data includes the distance from the object, and the gain value dir_gain, which is the output of the gain function, is i_obj In this case, the gain function may be modified so that the gain correction reproduces not only the directional characteristics but also the distance attenuation. In this case, the gain function may be modified so that the gain correction is performed based on the relative distance d o will be used.

[0243] Furthermore, the directivity rendering unit 33 calculates the object azimuth angle ψ i_obj and the object elevation angle θ i_obj Based on this, the reproduction gain value VBAP_gain of the channel corresponding to each of the multiple speakers constituting the reproduction unit 12 is calculated by VBAP. i_spk Request.

[0244] Then, the directional rendering unit 33 converts the audio data obj_audio i_obj , distance attenuation gain value i_obj , directional characteristic gain value dir_gain i_obj , and the playback gain value VBAP_gain for the channel corresponding to the speaker i_spk The following equation (15) is calculated based on the i_spk Request.

[0245]

number

[0246] Here, the calculation of equation (15) is performed for each combination of the speaker constituting the playback unit 12 and the object constituting the content, and the playback signal speaker_signal i_spk is required.

[0247] This realizes gain correction for reproducing distance attenuation, gain correction for reproducing sound propagation according to directional characteristics, and VBAP processing for localizing a sound image at a desired position.

[0248] In contrast, the gain value dir_gain obtained from the directional characteristic data i_obj is a gain value that takes into account both the directional characteristic and distance attenuation, that is, the relative distance d o If included, the calculation of the following equation (16) is performed.

[0249] That is, the directional rendering unit 33 converts the audio data obj_audio i_obj , directional characteristic gain value dir_gain i_obj , and the playback gain value VBAP_gain i_spk The following equation (16) is calculated based on the i_spk Request.

[0250]

number

[0251] When the playback signal is obtained in the above manner, the directivity rendering unit 33 finally converts the playback signal speaker_signal obtained for the current frame into i_spk and the playback signal speaker_signal of the frame immediately before the current frame i_spk The two signals are overlap-added to obtain the final reproduced signal.

[0252] Although the above description has been given by taking an example in which VBAP is performed as the rendering process, a playback signal can also be obtained by similar processing in the case in which HRTF convolution processing is performed as the rendering process.

[0253] Here, we will explain the case where an HRTF database is used that contains HRTFs for each user according to the distance, azimuth, and elevation angle that indicate the relative positional relationship between an object and a user (listener), and a headphone playback signal is generated that takes into account the directional characteristics of the object.

[0254] In particular, it is assumed here that an HRTF database consisting of HRTFs from virtual speakers corresponding to the real speakers at the time of HRTF measurement is held in the directivity rendering unit 33, and the playback unit 12 is a headphone.

[0255] Note that, although a case will be described here in which an HRTF database is prepared for each user in consideration of differences in characteristics between individual users, a common HRTF database may also be used for all users.

[0256] In this example, the personal ID information for identifying an individual user is j, and the azimuth and elevation angles indicating the direction of sound arrival from the sound source (virtual speaker), i.e., the object to the user's ears, are respectively ψ L and ψ R and θ L and θ R Here, the azimuth angle ψ L and the elevation angle θ L are the azimuth and elevation angles indicating the direction of arrival at the user's left ear, and the azimuth angle ψ R and the elevation angle θ R are the azimuth and elevation angles indicating the direction of arrival to the user's right ear.

[0257] In addition, the HRTF, which is the transfer characteristic from the sound source to the user's left ear, is specifically defined as HRTF(j,ψ L ,θ L ), and the HRTF, which is the transfer characteristic from the sound source to the user's right ear, is specifically denoted as HRTF(j,ψ R ,θ R ) will be written as

[0258] Note that HRTFs for the left and right ears of the user may be prepared for each direction of arrival and distance to the sound source, and distance attenuation may be reproduced by convolution of the HRTFs.

[0259] Furthermore, the directional characteristic data may be a function indicating the transfer characteristic from the sound source to each direction, or may be a gain function as in the above-mentioned VBAP example. i_obj and the object rotation elevation angle θ_rot i_obj is used.

[0260] Alternatively, the object rotation azimuth angle and object rotation elevation angle may be calculated for each of the left and right ears, taking into account the convergence angle of the user's left and right ears relative to the object, i.e., the difference in the arrival angle of sound from the object to the user's ears due to the width of the user's face.

[0261] The convergence angle here is the angle between a line connecting the left ear of the user (listener) and the object and a line connecting the right ear of the user and the object.

[0262] In the following, of the object rotation azimuth angle and the object rotation elevation angle that constitute the relative orientation information, the object rotation azimuth angle ψ_rot i_obj_l and the object rotation elevation angle θ_rot i_obj_l It will be written as follows.

[0263] Similarly, hereinafter, of the object rotation azimuth angle and the object rotation elevation angle constituting the relative orientation information, the object rotation azimuth angle ψ_rot i_obj_r and the object rotation elevation angle θ_rot i_obj_r It will be written as follows.

[0264] First, the directional rendering unit 33 performs the calculation of the above-mentioned equation (13) to obtain a gain value GAIN for reproducing distance attenuation. i_obj Calculate.

[0265] In addition, the HRTF database includes HRTFs for each sound arrival direction and distance to the sound source. If distance attenuation can be reproduced by convolution of HRTFs, the gain value i_obj Alternatively, the reproduction of the distance attenuation may be realized by convolution of the transfer characteristic obtained from the directional characteristic data, instead of the convolution of the HRTF.

[0266] Next, the directivity rendering unit 33 acquires a transfer characteristic according to the directivity of the object based on, for example, the directivity data and the relative direction information.

[0267] For example, when a function for obtaining the transfer characteristic is supplied as directional characteristic data and the function has distance, azimuth angle, and elevation angle as arguments, the directivity rendering unit 33 calculates the following equation (17) based on the relative distance information, relative direction information, and directional characteristic data.

[0268]

number

[0269] That is, in equation (17), the directional rendering unit 33 uses the relative distance d o d i_obj Let us assume that.

[0270] The directivity rendering unit 33 then uses the function dir(i,d i_obj ,ψ_rot i_obj_l ,θ_rot i_obj_l ) at a relative distance d o , Object rotation azimuth angle ψ_rot i_obj_l , and the object rotation elevation angle θ_rot i_obj_l Substituting the above, the transfer characteristic of the left ear is dir_func i_obj_l get.

[0271] Similarly, the directivity rendering unit 33 converts the directivity data supplied as the directional characteristic data into a function dir(i,di_obj ,ψ_rot i_obj_r ,θ_rot i_obj_r ) at a relative distance d o , Object rotation azimuth angle ψ_rot i_obj_r , and the object rotation elevation angle θ_rot i_obj_r Substituting the above, the transfer characteristic of the right ear, dir_func i_obj_r get.

[0272] In this case, the transfer function dir_func i_obj_l or transfer function dir_func i_obj_r By convolution, the distance attenuation can also be reproduced.

[0273] Furthermore, the directional rendering unit 33 calculates the object azimuth angle ψ i_obj and the object elevation angle θ i_obj Based on this, the HRTF for the left ear (j, ψ L ,θ L ) and the right-ear HRTF(j,ψ R ,θ R ) is obtained. For example, L =ψ i_obj And θ L =θ i_obj HRTF(j,ψ L ,θ L ) is read from the HRTF database. Note that the object azimuth angle and the object elevation angle may also be calculated for each of the left and right ears.

[0274] Once the transfer characteristics and HRTFs of the left and right ears are obtained through the above processing, the transfer characteristics and HRTFs are combined with the object's audio data obj_audio i_obj Based on this, reproduction signals for the left and right ears that are to be supplied to headphones serving as the reproduction unit 12 are calculated.

[0275] Specifically, for example, the transfer function dir_func obtained from the directional characteristic data i_obj_l and the transfer function dir_func i_obj_rWhen both the directivity and the distance attenuation are taken into consideration, that is, when the transfer characteristic is calculated by the formula (17), the directivity rendering unit 33 calculates the reproduction signal HPout for the left ear by the following formula (18). L and the playback signal HPout for the right ear R Request.

[0276]

number

[0277] In addition, in equation (18), * represents a convolution process.

[0278] So here the audio data obj_audio i_obj For the transfer function dir_func i_obj_l and HRTF(j,ψ L ,θ L ) is convoluted to produce the playback signal HPout for the left ear. L Similarly, the audio data obj_audio i_obj For the transfer function dir_func i_obj_r and HRTF(j,ψ R ,θ R ) is convoluted to generate the playback signal HPout for the right ear R Even when distance attenuation is reproduced by HRTF, the reproduced signal can be obtained by a calculation similar to that of equation (18).

[0279] On the other hand, for example, when the transfer characteristic or HRTF obtained from the directional characteristic data does not take into account distance attenuation, the directivity rendering unit 33 obtains a reproduction signal by performing the calculation of the following equation (19).

[0280]

number

[0281] In equation (19), in addition to the convolution process performed in equation (18), the audio data obj_audioi_obj The gain value for reproducing distance attenuation is i_obj The left ear playback signal HPout is also L and the playback signal HPout for the right ear R This gain value is calculated. i_obj is obtained by the above equation (13).

[0282] The above process results in a playback signal HPout L and playback signal HPout R When the directivity rendering unit 33 obtains the reproduced signal HPout, it performs overlap addition with the reproduced signal of the previous frame to obtain the final reproduced signal HPout L and playback signal HPout R Let us assume that.

[0283] Furthermore, when processing for wave field synthesis is performed as the rendering process, i.e., when a sound field including the sound of an object is formed by wave field synthesis using multiple speakers as the playback unit 12, a playback signal is generated as follows.

[0284] Here, an example will be described in which a speaker drive signal to be supplied to a speaker constituting the reproduction unit 12 is generated as a reproduction signal using a spherical harmonic function.

[0285] The external sound field at a position outside a certain radius r from a given sound source, that is, the radius (distance) from the sound source is r' (where r'>r) and the azimuth and elevation angles indicating the direction as seen from the sound source are ψ and θ, i.e., the sound pressure p(r', ψ, θ), can be expressed by the following equation (20).

[0286]

number

[0287] In addition, in formula (20), Y n m (ψ,θ) is a spherical harmonic function, and n and m indicate the degree and order of the spherical harmonic function. n(1) (kr) is the first kind of spherical Hankel function, where k is the wave number.

[0288] Furthermore, in equation (20), X(k) represents the reproduced signal expressed in the frequency domain, and P nm (r) shows the spherical harmonic spectrum for a sphere of radius (distance) r, where the frequency domain signal X(k) corresponds to the audio data of an object.

[0289] For example, if the measurement microphone array for measuring the directional characteristics is a sphere with a radius of r, the measurement microphone array can be used to measure the sound pressure at a position of radius r of the sound propagating in all directions from a sound source at the center of the sphere (measurement microphone array). In particular, since the directional characteristics differ depending on the sound source, by measuring the sound from the sound source at each position, observed sounds containing directional characteristic information can be obtained.

[0290] Spherical harmonic spectrum P nm (r) can be expressed as the following equation (21) using the measured observed sound pressure p(r, ψ, θ) measured by such a measurement microphone array.

[0291]

number

[0292] In equation (21), ∂Ω represents the integration range, in particular the integral over radius r.

[0293] Such a spherical harmonic spectrum P nm (r) is data indicating the directional characteristics of the sound source. Therefore, for example, for each type of sound source, the spherical harmonic spectrum P nm If (r) is measured in advance, the function shown in the following equation (22) can be used to calculate the directional characteristic data dir(i_obj, d i_obj ) can be used.

[0294]

number

[0295] In equation (22), i_obj indicates the type of sound source, and d i_obj indicates the distance from the sound source, and this distance d i_obj is the relative distance d o The directional pattern data dir(i_obj,d i_obj ) is data that indicates the transmission characteristics in each direction determined by the azimuth angle ψ and the elevation angle θ, that is, in all directions, taking into account the amplitude and phase.

[0296] If there is no change in the relative positional relationship between the object and the listening position, a playback signal that also takes into account the directional characteristics can be obtained using the above equation (20).

[0297] However, even if the relative positional relationship between the object and the listening position changes, the directional characteristic data dir(i_obj, d i_obj ) Object rotation azimuth angle ψ_rot i_obj and the object rotation elevation angle θ_rot i_obj By performing a rotation operation based on the azimuth angle ψ, the elevation angle θ, and the distance d i_obj The point determined by (d i_obj ,ψ,θ) i_obj ,ψ,θ) can be obtained.

[0298]

number

[0299] In addition, when calculating equation (23), the distance d i_obj As the relative distance d o is assigned, and the audio data of the object is assigned to X(k), and the sound pressure p(d i_obj ,ψ,θ) is obtained. Then, the sound pressure p(d i_obj,ψ,θ) to obtain the sum of the points (d i_obj , ψ, θ), i.e., the reproduced signal, is obtained.

[0300] Therefore, when generating a reproduction signal for wave field synthesis, the calculation of equation (23) is performed for each wave number k for each object in the process of step S16, and a reproduction signal is generated based on the calculation result.

[0301] When a playback signal to be supplied to the playback unit 12 is obtained by the rendering process described above, the process proceeds from step S16 to step S17.

[0302] In step S17, the directional rendering unit 33 supplies the playback signal obtained by the rendering process to the playback unit 12, which outputs the sound. This causes the sound of the content, that is, the sound of the object, to be played back.

[0303] In step S18, the signal generator 24 determines whether or not to end the process of playing back the sound of the content. For example, when the process is performed for all frames and the playback of the content is completed, it is determined that the process is to be ended.

[0304] If it is determined in step S18 that the process is not yet to be ended, the process returns to step S11, and the above-mentioned process is repeated.

[0305] On the other hand, if it is determined in step S18 that the process is to be ended, the content playback process is ended.

[0306] In this way, the signal processing device 11 generates the relative distance information and the relative direction information, and performs rendering processing that takes into account the directional characteristics using the relative distance information and the relative direction information. In this way, sound propagation according to the directional characteristics of the object can be reproduced, and a higher sense of realism can be obtained.

[0307] Example of computer configuration The above-mentioned series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed in a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, for example, capable of executing various functions by installing various programs.

[0308] FIG. 11 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes according to a program.

[0309] In the computer, a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, and a RAM (Random Access Memory) 503 are interconnected via a bus 504.

[0310] An input / output interface 505 is further connected to the bus 504. An input unit 506, an output unit 507, a recording unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.

[0311] The input unit 506 includes a keyboard, a mouse, a microphone, an image sensor, etc. The output unit 507 includes a display, a speaker, etc. The recording unit 508 includes a hard disk, a non-volatile memory, etc. The communication unit 509 includes a network interface, etc. The drive 510 drives a removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0312] In a computer configured as described above, the CPU 501 loads, for example, a program recorded in the recording unit 508 into the RAM 503 via the input / output interface 505 and the bus 504, and executes the program, thereby performing the series of processes described above.

[0313] The program executed by the computer (CPU 501) can be provided by being recorded on a removable recording medium 511 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0314] In the computer, the program can be installed in the recording unit 508 via the input / output interface 505 by mounting the removable recording medium 511 in the drive 510. The program can also be received by the communication unit 509 via a wired or wireless transmission medium and installed in the recording unit 508. Alternatively, the program can be installed in the ROM 502 or the recording unit 508 in advance.

[0315] In addition, the program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or it may be a program in which processing is performed in parallel or at the required timing, such as when called.

[0316] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0317] For example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0318] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.

[0319] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices.

[0320] Furthermore, the present technology can also be configured as follows.

[0321] (1) an acquisition unit that acquires metadata including position information indicating a position of an audio object and orientation information indicating a direction of the audio object, and audio data of the audio object; a signal generating unit that generates a playback signal for playing back a sound of the audio object at the listening position based on listening position information indicating a listening position, listener direction information indicating a direction of the listener at the listening position, the position information, the direction information, and the audio data; A signal processing device comprising: (2) The acquisition unit acquires the metadata for each predetermined time interval. A signal processing device as described in (1). (3) The signal generating unit generates the playback signal based on directional characteristic data indicating directional characteristics of the audio object, the listening position information, the listener direction information, the position information, the direction information, and the audio data. A signal processing device according to (1) or (2). (4) The signal generating unit generates the reproduction signal based on the directional characteristic data determined for the type of the audio object. A signal processing device according to (3). (5) The orientation information is information including an azimuth angle indicating a direction of the audio object. A signal processing device according to (3) or (4). (6) The orientation information includes an azimuth angle and an elevation angle that indicate a direction of the audio object. A signal processing device according to (3) or (4). (7) The orientation information includes an azimuth angle and an elevation angle that indicate a direction of the audio object, and a tilt angle that indicates a rotation of the audio object. A signal processing device according to (3) or (4). (8) The listening position information is information indicating a predetermined fixed listening position, and the listener direction information is information indicating a predetermined fixed orientation of the listener. A signal processing device according to any one of (3) to (7). (9) The position information is information consisting of an azimuth angle and an elevation angle indicating the direction of the audio object as viewed from the listening position, and a radius indicating the distance from the listening position to the audio object. A signal processing device according to (8). (10) The listening position information is information indicating an arbitrary listening position, and the listener direction information is information indicating an arbitrary direction of the listener. A signal processing device according to any one of (3) to (7). (11) The position information is a coordinate in a Cartesian coordinate system indicating a position of the audio object. A signal processing device according to (10). (12) The signal generating unit is The directional characteristic data; Relative distance information indicating a relative distance between the audio object and the listening position, the relative distance information being obtained from the listening position information and the position information; Relative orientation information indicating a relative direction between the audio object and the listener, the relative orientation information being obtained from the listening position information, the listener orientation information, the position information, and the orientation information; The audio data; generating the reproduction signal based on A signal processing device according to any one of (3) to (11). (13) The relative orientation information includes an azimuth angle and an elevation angle that indicate a relative direction between the audio object and the listener. A signal processing device according to (12). (14) The relative direction information includes information indicating a direction of the listener from the audio object and information indicating a direction of the audio object from the listener. A signal processing device according to (12) or (13). (15) The signal generating unit generates the reproduction signal based on information indicating a transfer characteristic of a direction of the listener as seen from the audio object, the transfer characteristic being obtained from the directional characteristic data and information indicating a direction of the listener as seen from the audio object. A signal processing device according to (14). (16) A signal processing device, obtaining metadata including position information indicating a position of an audio object and orientation information indicating a direction of the audio object, and audio data of the audio object; A playback signal for playing back the sound of the audio object at the listening position is generated based on listening position information indicating a listening position, listener direction information indicating a direction of the listener at the listening position, the position information, the direction information, and the audio data. Signal processing methods. (17) obtaining metadata including position information indicating a position of an audio object and orientation information indicating a direction of the audio object, and audio data of the audio object; A playback signal for playing back the sound of the audio object at the listening position is generated based on listening position information indicating a listening position, listener direction information indicating a direction of the listener at the listening position, the position information, the direction information, and the audio data. A program that causes a computer to execute a process including steps. [Explanation of symbols]

[0322] 11 signal processing device, 21 acquisition unit, 22 listening position designation unit, 23 directional characteristic database unit, 24 signal generation unit, 31 relative distance calculation unit, 32 relative direction calculation unit, 33 directional rendering unit

Claims

1. an acquisition unit that acquires metadata including position information indicating a position of an audio object and orientation information indicating a direction of the audio object, and audio data of the audio object; a signal generating unit that generates a playback signal for playing back a sound of the audio object at the listening position based on listening position information indicating a listening position, listener direction information indicating a direction of the listener at the listening position, the position information, the direction information, and the audio data; A signal processing device comprising:

2. The acquisition unit acquires the metadata for each predetermined time interval. The signal processing device according to claim 1 .

3. The signal generating unit generates the playback signal based on directional characteristic data indicating directional characteristics of the audio object, the listening position information, the listener direction information, the position information, the direction information, and the audio data. The signal processing device according to claim 1 .

4. The signal generating unit generates the reproduction signal based on the directional characteristic data determined for the type of the audio object. The signal processing device according to claim 3 .

5. The orientation information is information including an azimuth angle indicating a direction of the audio object. The signal processing device according to claim 3 .

6. The orientation information includes an azimuth angle and an elevation angle that indicate a direction of the audio object. The signal processing device according to claim 3 .

7. The orientation information includes an azimuth angle and an elevation angle that indicate a direction of the audio object, and a tilt angle that indicates a rotation of the audio object. The signal processing device according to claim 3 .

8. The listening position information is information indicating a predetermined fixed listening position, and the listener direction information is information indicating a predetermined fixed orientation of the listener. The signal processing device according to claim 3 .

9. The position information is information consisting of an azimuth angle and an elevation angle indicating the direction of the audio object as viewed from the listening position, and a radius indicating the distance from the listening position to the audio object. The signal processing device according to claim 8.

10. The listening position information is information indicating an arbitrary listening position, and the listener direction information is information indicating an arbitrary direction of the listener. The signal processing device according to claim 3 .

11. The position information is a coordinate in a Cartesian coordinate system indicating a position of the audio object. The signal processing device according to claim 10.

12. The signal generating unit is The directional characteristic data; Relative distance information indicating a relative distance between the audio object and the listening position, the relative distance information being obtained from the listening position information and the position information; Relative orientation information indicating a relative direction between the audio object and the listener, the relative orientation information being obtained from the listening position information, the listener orientation information, the position information, and the orientation information; The audio data; generating the reproduction signal based on The signal processing device according to claim 3 .

13. The relative orientation information includes an azimuth angle and an elevation angle that indicate a relative direction between the audio object and the listener. The signal processing device according to claim 12.

14. The relative direction information includes information indicating a direction of the listener from the audio object and information indicating a direction of the audio object from the listener. The signal processing device according to claim 12.

15. The signal generating unit generates the reproduction signal based on information indicating a transfer characteristic of a direction of the listener as seen from the audio object, the transfer characteristic being obtained from the directional characteristic data and information indicating a direction of the listener as seen from the audio object. The signal processing device according to claim 14.

16. A signal processing device, obtaining metadata including position information indicating a position of an audio object and orientation information indicating a direction of the audio object, and audio data of the audio object; A playback signal for playing back the sound of the audio object at the listening position is generated based on listening position information indicating a listening position, listener direction information indicating a direction of the listener at the listening position, the position information, the direction information, and the audio data. Signal processing methods.

17. obtaining metadata including position information indicating a position of an audio object and orientation information indicating a direction of the audio object, and audio data of the audio object; A playback signal for playing back the sound of the audio object at the listening position is generated based on listening position information indicating a listening position, listener direction information indicating a direction of the listener at the listening position, the position information, the direction information, and the audio data. A program that causes a computer to execute a process including steps.

Images