Signal processing device, method and program

Abstract

The present technology relates to a signal processing device and method and a program capable of obtaining a greater sense of reality. The signal processing device is provided with: an acquisition unit for acquiring metadata and audio data related to an audio object, the metadata including position information indicating the position of the audio object and direction information indicating the direction of the audio object; and a signal generation unit for generating a reproduction signal for reproducing the sound of the audio object at the listening position based on listening position information indicating the listening position, listener direction information indicating the direction of the listener at the listening position, position information, direction information and audio data. The present technology can be applied to a transmission / reproduction system.

Description

Technical Field

[0001] The present technology relates to a signal processing device, a signal processing method, and a program, and more particularly to a signal processing device, a signal processing method, and a program capable of providing a higher sense of reality. Background Art

[0002] For example, to reproduce a sound field from a free viewpoint such as a bird's-eye view or a walk-through, it is important to record target sounds such as human voices, movement sounds of players such as a kicking sound in sports, or instrumental sounds in music with as high a signal-to-noise ratio (SNR) as possible.

[0003] Furthermore, at the same time, for each sound source of the target sound, it is necessary to reproduce the sound with accurate localization, and make the sound image localization and the like follow the movement of the viewpoint or the sound source.

[0004] Incidentally, a technology capable of providing a higher sense of reality in free viewpoint or fixed viewpoint content is desired, and a large number of such technologies have been proposed.

[0005] For example, as a technology for reproducing a sound field from a free viewpoint, a technology for performing gain correction and frequency characteristic correction according to the distance from a changed listening position to an audio object when a user can freely specify a listening position has been proposed (for example, see Patent Document 1).

[0006] Reference List

[0007] Patent Documents

[0008] Patent document 1: WO 2015 / 107926 A. Summary of the invention

[0009] Problems to be solved by the present invention

[0010] However, the above techniques cannot provide a high enough sense of realism in some cases.

[0011] For example, in the real world, a sound source is not a point sound source, and sound waves propagate from a sound-emitting body having a size with a specific directional characteristic including reflection and diffraction caused by the sound-emitting body.

[0012] However, many attempts have been made to record the sound field in the target space, and even when recording is performed for each sound source (i.e., for each audio object), sufficiently high realism cannot be obtained in some cases because the direction of each audio object is not considered on the reproduction side.

[0013] The present technology has been made in view of such circumstances, and an object of the present technology is to provide a higher sense of reality.

[0014] Solution to the problem

[0015] A signal processing device according to one aspect of the present technology includes: an acquisition unit that acquires metadata and audio data of an audio object, the metadata including position information indicating the position of the audio object and direction information indicating the direction of the audio object; and a signal generation unit that generates a reproduction signal for reproducing the sound of the audio object at the listening position based on listening position information indicating the listening position, listener direction information indicating the direction of the listener at the listening position, position information, direction information and audio data.

[0016] A signal processing method or program according to one aspect of the present technology includes: the steps of acquiring metadata and audio data of an audio object, the metadata including position information indicating the position of the audio object and direction information indicating the direction of the audio object; and the step of generating a reproduction signal for reproducing the sound of the audio object at the listening position based on the listening position information indicating the listening position, the listener direction information indicating the direction of the listener at the listening position, the position information, the direction information and the audio data.

[0017] In one aspect of the present technology, metadata and audio data of an audio object are obtained, the metadata including position information indicating the position of the audio object and direction information indicating the direction of the audio object; and a reproduction signal for reproducing the sound of the audio object at the listening position is generated based on the listening position information indicating the listening position, the listener direction information indicating the direction of the listener at the listening position, the position information, the direction information and the audio data. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is an illustration of the directions of objects included in the content.

[0019] Figure 2 It is a diagram illustrating the directional characteristics of an object.

[0020] Figure 3 An example of the syntax of metadata is shown.

[0021] Figure 4 An example of the syntax of the directional characteristic data is shown.

[0022] Figure 5 A configuration example of a signal processing device is shown.

[0023] Figure 6 This is a diagram for explaining relative direction information.

[0024] Figure 7 This is a diagram for explaining relative direction information.

[0025] Figure 8 This is a diagram for explaining relative direction information.

[0026] Fig. 9 This is a diagram for explaining relative direction information.

[0027] Fig.10 is a flowchart showing a content reproduction process.

[0028] Fig.11 A configuration example of a computer is shown. DETAILED DESCRIPTION

[0029] Hereinafter, embodiments to which the present technology is applied will be described with reference to the drawings.

[0030] <First Embodiment>

[0031] <This technology>

[0032] The present technology relates to a transmission reproduction system capable of providing higher realism by appropriately transmitting directional characteristic data indicating the directional characteristics of an audio object serving as a sound source and reflecting the directional characteristics of the audio object in content reproduction on a content reproduction side based on the directional characteristic data.

[0033] Content for reproducing the sound of an audio object (hereinafter, also simply referred to as an object) serving as a sound source is, for example, fixed viewpoint content or free viewpoint content.

[0034] In fixed viewpoint content, the viewpoint position of the listener, that is, the listening position (listening point) is set to a predetermined fixed position, while in free viewpoint content, the user as the listener can freely specify the listening position (viewpoint position) in real time.

[0035] In the real world, each sound source has unique directional characteristics. That is, even sounds emitted from the same sound source have different sound transmission characteristics depending on the direction from which the sound source is viewed.

[0036] Therefore, in the case where an object serving as a sound source in the content or a listener at a listening position freely moves or rotates, the way in which the listener hears the sound of the object also changes according to the directional characteristics of the object.

[0037] In the reproduction of content, a process of reproducing distance attenuation according to the distance from the listening position to the object is generally performed. Meanwhile, the present technology reproduces content in consideration of not only distance attenuation but also directional characteristics of the object, thereby providing a higher sense of reality.

[0038] That is, in the present technology, when a listener or an object moves or rotates freely, not only the distance between the listener and the object but also, for example, the relative direction between the listener and the object is taken into consideration, and the transfer characteristics according to the distance attenuation and directional characteristics are dynamically added to the sound of the content of each object.

[0039] For example, the transfer characteristics are added by gain correction according to distance attenuation and directional characteristics, wave field synthesis processing based on wavefront amplitude and phase propagation characteristics taking distance attenuation and directional characteristics into consideration, and the like.

[0040] The present technology uses the directional characteristic data to add the transfer characteristic according to the directional characteristic. In the case where the directional characteristic data is prepared corresponding to each target sound source (ie, each type of object), a higher sense of reality can be provided.

[0041] For example, the directional characteristic data for each type of object can be obtained by recording sound in advance using a microphone array or the like or by performing a simulation and calculating the transfer characteristics for each direction and each distance when sound emitted from the object propagates through space.

[0042] Directional characteristic data of each type of object is transmitted in advance to a device on the reproduction side together with or separately from audio data of content.

[0043] Then, when reproducing the content, the device on the reproducing side uses the directional characteristic data to add the transfer characteristic according to the distance from the object and the directional characteristic to the audio data of the object, that is, to the reproduction signal for reproducing the sound of the content.

[0044] This makes it possible to reproduce content with a higher sense of realism.

[0045] In this technology, for each type of sound source (object), a transfer characteristic according to the relative positional relationship between the listener and the object (i.e., according to the relative distance or direction between the listener and the object) is added. Therefore, even in the case where the object and the listening position are equidistant, the way the listener hears the sound of the object changes depending on the direction from which the listener hears the sound. This makes it possible to reproduce a more realistic sound field.

[0046] Examples of content for which this technology is applicable include the following:

[0047] - Content that reproduces the field where team sports are played;

[0048] - Content that reproduces a space with multiple performers (such as a musical, opera or play);

[0049] - Recreate content in any space, whether in a live performance venue or theme park;

[0050] - Reproductions of performances by orchestras, military bands, etc.; and

[0051] -Content such as games.

[0052] Note that, for example, in performance content of a marching band or the like, the performers may be stationary or moving.

[0053] Next, hereinafter, the present technology will be described in more detail.

[0054] For example, an example in which the content reproduces a sound field in which an arbitrary position on a soccer field is set as a listening position will be described.

[0055] In this case, for example, Figure 1 As shown, there are players of each team and referees on the field, and these players and referees are sound sources, that is, audio objects.

[0056] exist Figure 1 In the example, Figure 1 Each circle in represents a player or a referee, that is, an object, and the direction of the line segment attached to each circle represents the direction in which the player or referee represented by the circle is facing, that is, the direction of the object such as the player or referee.

[0057] Here, these objects face different directions at different locations, and the position and orientation of the objects change over time. That is, each object moves or rotates over time.

[0058] For example, the object OB11 is a referee, and the position of the object OB11 is set as the viewpoint position (listening position) and Figure 1 The video and audio obtained with the upward direction in (that is, the direction of the object OB11 is set as the line of sight direction) are presented to the listener as content as an example.

[0059] exist Figure 1 In the example of , each object is located on a two-dimensional plane, but in reality, each player and referee as an object is different in the height of the mouth, the height of the foot as the position where the kicking sound is generated, etc. In addition, the posture of the object is constantly changing.

[0060] That is, in practice, each object and viewpoint (listening position) are both located in a three-dimensional space, and at the same time, these objects and listeners (users) at the viewpoint face various directions in various postures.

[0061] The following is a classification of cases where the directional characteristics according to the direction of the object can be reflected in the content.

[0062] (Case 1)

[0063] The object or the listening position is located on a two-dimensional plane, and only the azimuth (yaw) indicating the direction of the object is considered, without considering the pitch or roll angles.

[0064] (Case 2)

[0065] The object or the listening position is located in a three-dimensional space, and the azimuth and elevation angles indicating the direction of the object are considered, while the tilt angle indicating the rotation of the object is not considered.

[0066] (Case 3)

[0067] The object or the listening position is located in a three-dimensional space, and a case including an azimuth angle and an elevation angle indicating the direction of the object and an Euler angle indicating the tilt angle of the rotation of the object is considered.

[0068] The present technology is applicable to any of the above-mentioned cases 1 to 3, and in each case, content is reproduced taking into account the listening position, the position of the object, and the direction and rotation (tilt) of the object (ie, the rotation angle of the object) as appropriate.

[0069] <Transmission device>

[0070] A transmission reproduction system that transmits and reproduces such content includes, for example, a transmission device that transmits content data and a signal processing device that serves as a reproduction device that reproduces the content based on the content data transmitted from the transmission device. Note that one or more signal processing devices may be used as the reproduction device.

[0071] The transmission device on the transmission side of the transmission reproduction system transmits, for example, audio data for reproducing the sound of each of one or more objects included in the content and metadata (audio data) of each object as data of the content.

[0072] Here, the metadata includes sound source type information, sound source position information, and sound source direction information.

[0073] The sound source type information is ID information indicating the type of an object serving as a sound source.

[0074] For example, the sound source type information may be information unique to a sound source such as a player or a musical instrument (indicating the type (type) of the object itself used as the sound source), or it may be information indicating the type of sound emitted from the object (such as a player's voice, a kicking sound, a clapping sound, or other sports sounds).

[0075] In addition, the sound source type information may be information indicating the type of the object itself and the type of sound emitted from the object.

[0076] Furthermore, directional characteristic data is prepared for each type indicated by the sound source type information, and a reproduction signal is generated on the reproduction side based on the directional characteristic data determined for the sound source type information. Therefore, it can also be said that the sound source type information is ID information indicating directional characteristic data.

[0077] In the transmission device, the sound source type information is, for example, manually assigned to each object included in the content and included in metadata of the object.

[0078] Furthermore, the sound source position information included in the metadata indicates the position of an object serving as a sound source.

[0079] Here, the sound source position information is, for example, latitude and longitude indicating an absolute position on the earth's surface measured (acquired) by a position measurement module such as a global positioning system (GPS) module, coordinates obtained by converting latitude and longitude into distance, or the like.

[0080] In addition, the sound source position information may be any information as long as the information indicates the position of an object, such as coordinates in a coordinate system with a predetermined position in a target space (target area) where content is to be recorded as a reference position.

[0081] In addition, when the sound source position information is a coordinate (coordinate information), the coordinate can be a coordinate in any coordinate system, such as a coordinate in a polar coordinate system including azimuth, elevation and radius, a coordinate in an xyz coordinate system, that is, a coordinate in a three-dimensional orthogonal coordinate system, or a coordinate in a two-dimensional orthogonal coordinate system.

[0082] Furthermore, the sound source direction information included in the metadata indicates the absolute direction that the object at the position indicated by the sound source position information faces, that is, the front direction of the object.

[0083] Note that the sound source direction information may include not only information indicating the direction of the object but also information indicating the rotation (tilt) of the object. Hereinafter, the sound source direction information includes information indicating the direction of the object and information indicating the rotation of the object.

[0084] Specifically, for example, the sound source direction information includes an azimuth angle ψ indicating the direction of the object in the coordinate system used as the coordinates of the sound source position information. o and the elevation angle θ o , and a tilt angle indicating the rotation (tilt) of the object in the coordinate system used as the coordinates of the sound source position information

[0085] In other words, it can be said that the sound source direction information indicates the Euler angles, including the absolute direction indicating the object and the azimuth angle ψ of the rotation o (yaw), elevation angle θ o Pitch and tilt angle For example, the sound source direction information can be obtained from a geomagnetic sensor attached to the object, video data in which the object is used as a target, or the like.

[0086] The transmission device generates the sound source position information and the sound source direction information for each object, for each frame of audio data, or for each discrete unit time, such as for a predetermined number of frames, that is, at predetermined time intervals.

[0087] Then, metadata including the sound source type information, the sound source position information, and the sound source direction information is transmitted to the signal processing device for each unit time (such as for each frame) together with the audio data of the object.

[0088] Furthermore, the transmission device transmits the directional characteristic data to the signal processing device on the reproduction side in advance or sequentially for each sound source type indicated by the sound source type information. Note that the signal processing device may acquire the directional characteristic data from a device other than the transmission device or the like.

[0089] The directional characteristic data indicates the directional characteristics of an object of the sound source type indicated by the sound source type information, that is, transfer characteristics in various directions viewed from the object.

[0090] For example, Figure 2 As shown, each sound source has a directional characteristic specific to the sound source.

[0091] For example, in Figure 2 In the example of , the whistle as the sound source has a directivity in which the sound is strongly propagated in the front (front) direction, that is, has a sharp front directivity as shown by arrow Q11.

[0092] Furthermore, for example, the sound of footsteps emitted from a peak or the like serving as a sound source has a directional characteristic (non-directionality) in which the sound propagates in all directions with approximately the same intensity, as indicated by an arrow Q12.

[0093] Furthermore, for example, the sound emitted from the mouth of the player serving as the sound source has a directional characteristic in which the sound strongly propagates to the front and the side, that is, has a relatively strong front directivity as shown by an arrow Q13.

[0094] Directional characteristic data indicating the directional characteristics of such a sound source can be obtained by acquiring the propagation characteristics (transfer characteristics) of the sound to the surrounding environment for each sound source type using, for example, a microphone array in an anechoic chamber, etc. In addition, directional characteristic data can also be obtained by, for example, performing simulation on 3D data simulating the shape of the sound source.

[0095] Specifically, the directional characteristic data is, for example, a gain function dir(i, ψ, θ) defined as a function of an azimuth angle ψ and an elevation angle θ indicating a direction viewed from the sound source, determined for a value i of an ID indicating a sound source type.

[0096] Furthermore, a gain function dir(i, d, ψ, θ) having not only the azimuth angle ψ and the elevation angle θ but also the distance d of the distance-dispersed sound source as a parameter may be used as the directional characteristic data.

[0097] In this case, when each parameter is substituted into the gain function dir(i, d, ψ, θ), a gain value indicating the sound transfer characteristic (propagation characteristic) is obtained as an output of the gain function dir(i, d, ψ, θ).

[0098] The gain value indicates the characteristics (transfer characteristics) of the sound emitted from the sound source type with ID value i, which propagates in the directions of azimuth angle ψ and elevation angle θ observed from the sound source and reaches a position at a distance d from the sound source (hereinafter referred to as position P).

[0099] Therefore, when the audio data of the sound source type with an ID value i is gain-corrected according to the gain value, the sound emitted from the sound source type with an ID value i and that should be actually heard at the position P can be reproduced.

[0100] Specifically, in this example, using a gain value used as an output of the gain function dir(i, d, ψ, θ), gain correction for adding a transfer characteristic indicated by a directional characteristic taking into account the distance from the sound source (ie, distance attenuation) can be achieved.

[0101] Note that the directional characteristic data may be, for example, a gain function indicating transfer characteristics that also consider reverberation characteristics, etc. In addition, the directional characteristic data may be, for example, Ambisonics format data, that is, data including spherical harmonic coefficients (spherical harmonic spectrum) in each direction.

[0102] The transmission means sends the directional characteristic data prepared for each sound source type as described above to the signal processing means on the reproduction side.

[0103] Here, a specific example of transmitting metadata and directional characteristic data will be described.

[0104] For example, metadata is prepared for each frame of the audio data of the object having a predetermined time length, and metadata is prepared for each frame by using Figure 3 The bitstream syntax shown sends metadata to the playback side. Note that Figure 3 In the , uimsbf means unsigned integer MSB first, and tcimsbf means two's complement integer MSB first.

[0105] exist Figure 3In the example of , the metadata includes sound source type information “Object_type_index” of each object included in the content, sound source position information “Object_position[3]”, and sound source direction information “Obiect_direction[3]”.

[0106] Specifically, in this example, the sound source position information Object_position[3] is set as the coordinates (x o ,y o , z o ). Coordinate (x o ,y o , z o ) indicates the absolute position of the object in the xyz coordinate system (that is, in the target space).

[0107] In addition, the sound source direction information Object_direction[3] includes an azimuth angle ψ indicating the absolute direction of the object in the target space. o , elevation angle θ o and tilt angle

[0108] For example, in free viewpoint content, the viewpoint (listening position) changes over time during the reproduction of the content. Therefore, it is advantageous to generate a reproduction signal when the position of an object is represented by coordinates indicating an absolute position rather than relative coordinates based on the listening position.

[0109] At the same time, for example, in the case of fixed viewpoint content, coordinates of a polar coordinate system including an azimuth and elevation indicating the direction of an object observed from a listening position and a radius indicating the distance from the listening position to the object are preferentially set as sound source position information indicating the position of the object.

[0110] Note that metadata configuration is not limited to Figure 3 In addition, the metadata only needs to be transmitted at predetermined time intervals, and it is not always necessary to transmit the metadata for each frame.

[0111] Furthermore, the directional characteristic data of each sound source type may be stored in metadata and then transmitted, or may be transmitted by using e.g. Figure 4 The bitstream syntax shown is pre-transmitted separately from the metadata and audio data.

[0112] exist Figure 4In the example, a gain function "Object_directivity[distance][azimuth][elevation]" having the distance "distance" from the sound source and the azimuth "azimuth" and elevation "elevation" indicating the direction observed from the sound source as parameters is sent as directional characteristic data corresponding to the value of predetermined sound source type information.

[0113] Note that the directional characteristic data may be data in a format where the sampling intervals of the azimuth and elevation angles used as parameters are not equiangularly spaced, or may be data in a higher order high-fidelity stereo (HOA) format, i.e., high-fidelity stereo format (spherical harmonic coefficients).

[0114] For example, the directional characteristic data of a general sound source type is preferentially transmitted to the reproduction side in advance.

[0115] Meanwhile, directional characteristic data of a sound source having non-general directional characteristics (such as no pre-defined object) may be included in Figure 3 and sent as metadata.

[0116] As described above, metadata, audio data, and directional characteristic data are transmitted from the transmission device to the signal processing device on the reproduction side.

[0117] <Configuration Example of Signal Processing Device>

[0118] Next, a signal processing device as a device on the reproduction side will be described.

[0119] For example, the signal processing device on the reproduction side is configured as follows Figure 5 shown.

[0120] Figure 5 The signal processing device 11 generates a reproduction signal for reproducing the sound of the content (object) at the listening position based on the directional characteristic data acquired in advance from the transmission device or the like or shared in advance, and outputs the reproduction signal to the reproduction unit 12.

[0121] For example, the signal processing device 11 generates a reproduction signal by performing a process of vector-based amplitude translation (VBAP) or wave field synthesis, a head-related transfer function (HRTF) convolution process, or the like using the directional characteristic data.

[0122] The reproduction unit 12 includes, for example, a headphone, an earphone, a speaker array including two or more speakers, or the like, and reproduces the sound of the content based on the reproduction signal supplied from the signal processing device 11 .

[0123] Furthermore, the signal processing device 11 includes an acquisition unit 21 , a listening position specifying unit 22 , a direction characteristic database unit 23 , and a signal generating unit 24 .

[0124] The acquisition unit 21 acquires the directional characteristic data, metadata, and audio data by, for example, receiving data transmitted from a transmission device or reading data from a transmission device connected by wire or the like.

[0125] Note that the timing of acquiring the directional characteristic data and the timing of acquiring the metadata and audio data may be the same as or different from each other.

[0126] The acquisition unit 21 supplies the acquired directional characteristic data and metadata to the directional characteristic database unit 23 , and also supplies the acquired metadata and audio data to the signal generation unit 24 .

[0127] The listening position specifying unit 22 specifies the listening position in the target space and the direction of the listener (user) at the listening position, and supplies the signal generating unit 24 with listening position information indicating the listening position and listener direction information indicating the direction of the listener as a result of the specification.

[0128] The direction characteristic database unit 23 records the direction characteristic data of each of the plurality of sound source types supplied from the acquisition unit 21 .

[0129] Furthermore, in the case where the sound source type information included in the metadata is supplied from the acquisition unit 21 , the directional characteristic database unit 23 supplies directional characteristic data of the sound source type indicated by the supplied sound source type information among the recorded plurality of directional characteristic data to the signal generation unit 24 .

[0130] The signal generating unit 24 generates a reproduction signal based on the metadata and audio data supplied from the acquiring unit 21 , the listening position information and the listener direction information supplied from the listening position specifying unit 22 , and the direction characteristic data supplied from the direction characteristic database unit 23 , and supplies the reproduction signal to the reproducing unit 12 .

[0131] The signal generating unit 24 includes a relative distance calculating unit 31 , a relative direction calculating unit 32 , and a directivity rendering unit 33 .

[0132] The relative distance calculation unit 31 calculates the relative distance between the listening position (listener) and the object based on the sound source position information included in the metadata provided from the acquisition unit 21 and the listening position information provided from the listening position specification unit 22, and provides the relative distance information indicating the calculation result to the directivity rendering unit 33.

[0133] The relative direction calculation unit 32 calculates the relative direction between the listener and the object based on the sound source position information and sound source direction information included in the metadata provided by the acquisition unit 21 and the listening position information and listener direction information provided by the listening position designation unit 22, and provides the relative direction information indicating the calculation result to the directional rendering unit 33.

[0134] The directional rendering unit 33 performs rendering processing based on the audio data provided from the acquisition unit 21, the directional characteristic data provided from the directional characteristic database unit 23, the relative distance information provided from the relative distance calculation unit 31, the relative direction information provided from the relative direction calculation unit 32, and the listening position information and listener direction information provided from the listening position designation unit 22.

[0135] The directivity rendering unit 33 supplies a reproduction signal obtained by the rendering process to the reproduction unit 12, and causes the reproduction unit 12 to reproduce the sound of the content. For example, the directivity rendering unit 33 performs processing for VBAP or wave field synthesis, HRTF convolution processing, or the like as the rendering process.

[0136] <Each unit of the signal processing device>

[0137] (Listening position designation unit)

[0138] Next, each unit of the signal processing device 11 will be described in more detail.

[0139] The listening position specifying unit 22 specifies the listening position and direction of the listener in response to a user operation or the like.

[0140] For example, in the case of free viewpoint content, a user who is viewing the content, ie, a listener, operates a graphical user interface (GUI) or the like in a currently executed service, application, or the like, thereby designating an arbitrary listening position or direction of the listener.

[0141] In this case, the listening position specifying unit 22 sets the listening position and direction of the listener specified by the user as the listening position (viewpoint position) serving as the viewpoint of the content and the direction the listener faces (ie, the direction of the listener).

[0142] Furthermore, for example, when the user specifies a desired player from among a plurality of predetermined players or the like, the position and direction of the player can be set as the listening position and direction of the listener.

[0143] Furthermore, the listening position specifying unit 22 may execute some automatic routing program or the like or acquire information indicating the position and direction of the user from a head mounted display including the reproduction unit 12, thereby specifying an arbitrary listening position and direction of the listener without receiving a user operation.

[0144] As described above, in the free viewpoint content, the listening position and direction of the listener are set to an arbitrary position and an arbitrary direction which can be changed over time.

[0145] Meanwhile, in the fixed viewpoint content, the listening position specifying unit 22 specifies a predetermined fixed position and a fixed direction as the listening position and direction of the listener.

[0146] A specific example of the listening position information indicating the listening position is, for example, a coordinate (xyz) indicating the listening position in an xyz coordinate system indicating an absolute position on the surface of the earth or an xyz coordinate system indicating an absolute position in a target space. v ,y v , z v ).

[0147] In addition, for example, the listener direction information may include an azimuth angle ψ indicating the absolute direction of the listener in the xyz coordinate system. v and the elevation angle θ v and the absolute rotation (tilt) angle of the listener in the xyz coordinate system (That is, it can be information of Euler angles).

[0148] Specifically, in this case, in the fixed viewpoint content, for example, only the listening position information (x v ,y v , z v )=(0,0,0) and listener direction information

[0149] Note that in the following, it will be assumed that the listening position information is a coordinate in the xyz coordinate system (x v ,y v , z v ) and the listener direction information is the Euler angle The description continues in the following case.

[0150] Similarly, in the following, it will be assumed that the sound source position information is the coordinates (x o ,y o , z o ) and the sound source direction information is the Euler angle The description continues in the following case.

[0151] (Relative distance calculation unit)

[0152] The relative distance calculation unit 31 calculates the distance from the listening position to the object as the relative distance d of each object included in the content. o .

[0153] Specifically, the relative distance calculation unit 31 calculates the distance based on the listening position information (x v ,y v , z v ) and the sound source location information (x o ,y o , z o ) Calculate the following expression (1) to obtain the relative distance d o , and outputs the relative distance d obtained o Relative distance information.

[0154] [Mathematics 1]

[0155] d o =sqrt((x o -x v ) 2 +(y o -y v ) 2 +(z o -z v ) 2 ) ···(1)

[0156] (Relative direction calculation unit)

[0157] Furthermore, the relative direction calculation unit 32 obtains relative direction information indicating the relative direction between the listener and the object.

[0158] For example, the relative direction information includes the object azimuth ψ i_obj 、Object elevation angle θ i_obj 、Object rotation azimuth ψ _roti_obj and the object is rotated by an elevation angle θ _roti_obj .

[0159] Here, the object azimuth ψ i_obj and the object elevation angle θ i_obj are the azimuth and elevation angles respectively indicating the relative directions of an object as viewed from the listener.

[0160] To listen to the location information (x v ,y v , z v ) is the origin and the xyz coordinate system is rotated to the listener direction information The three-dimensional orthogonal coordinate system obtained by the indicated angles will be referred to as a listener coordinate system. In the listener coordinate system, the direction of the listener, that is, the front direction of the listener is set as the +y direction.

[0161] At this time, the azimuth and elevation angles indicating the direction of the object in the listener coordinate system are the object azimuth angle ψ i_obj and the object elevation angle θ i_obj .

[0162] Similarly, the object is rotated by an angle ψ_ roti_obj and the object is rotated by an elevation angle θ _roti_obj are the azimuth and elevation angles respectively indicating the relative direction (listening position) of the listener as viewed from the object. In other words, it can be said that the object is rotated by the azimuth angle ψ _roti_obj and the object is rotated by an elevation angle θ _roti_obj Is information indicating how much the front orientation of the object is rotated relative to the listener.

[0163] The sound source location information (x o ,y o , z o ) is the origin and the xyz coordinate system is rotated to the direction of the sound source. The three-dimensional orthogonal coordinate system obtained by the indicated angle will be referred to as an object coordinate system. In the object coordinate system, the direction of the object, that is, the front direction of the object is set to the a+y direction.

[0164] At this time, the azimuth and elevation angles indicating the direction of the listener (listening position) in the object coordinate system are the object rotation azimuth angle ψ_ roti_obj and the object is rotated by an elevation angle θ _roti_obj .

[0165] These objects are rotated by an azimuth angle ψ _roti_obj and the object is rotated by an elevation angle θ _roti_obj are the azimuth and elevation angles used to reference directional characteristic data during the rendering process.

[0166] Note that in the following description, the clockwise direction of the front direction (+y direction) of the azimuth in each three-dimensional orthogonal coordinate system such as the xyz coordinate system in the target space, the listener coordinate system, and the object coordinate system is set as a positive direction.

[0167] For example, in the xyz coordinate system, after projecting a target point such as an object onto the xy plane, the angle indicating the position (direction) of the projected target point based on the +y direction in the xy plane, that is, the angle between the direction of the projected target point and the +y direction is set as the azimuth. At this time, the clockwise direction from the +y direction is the positive direction.

[0168] Furthermore, in the listener coordinate system or the object coordinate system, the direction of the listener or the object, that is, the front direction of the listener or the object is the +y direction.

[0169] The upward direction of the elevation angle in each three-dimensional orthogonal coordinate system such as the xyz coordinate system in the target space, the listener coordinate system, and the object coordinate system is set as a positive direction.

[0170] For example, in an xyz coordinate system, the angle between the xy plane and a straight line passing through the origin of the xyz coordinate system and a target point such as an object is the elevation angle.

[0171] In addition, when a target point such as an object is projected onto the xy plane and a plane including the origin, the target point, and the projected target point of the xyz coordinate system is set as plane A, the a+z direction from the xy plane is set as the positive direction of the elevation angle on plane A.

[0172] Note that, for example, in the case of a listener coordinate system or an object coordinate system, an object or a listening position is used as a target point.

[0173] In addition, after the elevation rotation, when the inclination angle in each three-dimensional orthogonal coordinate system such as the xyz coordinate system in the target space, the listener coordinate system, and the object coordinate system rotates in the upper right direction when the +y direction is the positive direction, this rotation is set to a positive rotation.

[0174] Note that here, the azimuth, elevation, and tilt angles indicating the listening position, the direction of the object, etc. in the three-dimensional orthogonal coordinate system are defined as described above. However, the present technology is not limited to this and there is no loss of generality even if these angles are defined in another manner by using quaternions, rotation matrices, etc.

[0175] Here, the relative distance d will be described. o 、Object azimuth ψ i_obj 、Object elevation angle θ i_obj 、Object rotation angle ψ_rot i_obj and the object is rotated by an elevation angle θ_rot i_obj Specific examples.

[0176] First, a case where only the azimuth angle is considered without considering the elevation angle and the tilt angle in the sound source direction information and the listener direction information, that is, a two-dimensional case will be described.

[0177] For example, Figure 6 As shown, the position of a point P21 in the xy coordinate system with the origin O as a reference is set as the listening position, and the object is located at the position of a point P22.

[0178] Furthermore, the direction of the line segment W11 passing through the point P21, more specifically, the direction from the point P21 toward the end point of the line segment W11 opposite to the point P21 is set as the direction of the listener.

[0179] Similarly, the direction of the line segment W12 passing through the point P22 is set as the direction of the object. Furthermore, a straight line passing through the point P21 and the point P22 is defined as a straight line L11.

[0180] In this case, the distance between point P21 and point P22 is set to a relative distance do .

[0181] In addition, the angle between the line segment W11 and the straight line L11, that is, the angle indicated by the arrow K11 is the object azimuth angle ψ i_obj Similarly, the angle between the line segment W12 and the straight line L11, that is, the angle indicated by the arrow K12, is the object rotation azimuth angle ψ_rot i_obj .

[0182] In addition, in the case of a three-dimensional target space, the relative distance d o 、Object azimuth ψ i_obj 、Object elevation angle θ i_obj 、Object rotation angle ψ_rot i_obj and the object is rotated by an elevation angle θ_r oti_obj like Figures 7 to 9 Note that Figures 7 to 9 Corresponding parts in FIG. 1 are denoted by the same reference numerals, and description thereof will be appropriately omitted.

[0183] For example, Figure 7 As shown, the positions of points P31 and P32 in the xyz coordinate system with the origin O as a reference are set as the listening position and the position of the object, respectively, and a straight line passing through points P31 and P32 is set as straight line L31.

[0184] In addition, the listener direction information is obtained by rotating the xy plane of the xyz coordinate system The angle indicated by the above formula is the same as the angle indicated by the above formula, and then the origin O is translated to the listening position information (x v ,y v , z v ) is set as plane PF11. Plane PF11 is the xy plane of the listener coordinate system.

[0185] Similarly, by rotating the xy plane of the xyz coordinate system, the sound source direction information The angle indicated by the above formula is the same as the angle indicated by the above formula, and then the origin O is translated to the sound source position information (x o ,y o , z o ) is set as plane PF12. Plane PF12 is the xy plane of the object coordinate system.

[0186] Furthermore, the direction of the line segment W21 passing through the point P31, more specifically, the direction of the end point of the line segment W21 from the point P31 toward the opposite end point of the point P31 is set as the listener direction information The direction of the listener indicated.

[0187] Similarly, the direction of the line segment W22 passing through the point P32 is set as the sound source direction information The direction of the indicated object.

[0188] In this case, the distance between point P31 and point P32 is set to a relative distance d o .

[0189] In addition, if Figure 8 As shown, when the straight line obtained by projecting the straight line L31 onto the plane PF11 is set as the straight line L41, the angle between the straight line L41 and the line segment W21 on the plane PF11, that is, the angle indicated by the arrow K21 is the object azimuth angle ψ i_obj .

[0190] In addition, the angle between the straight line L41 and the straight line L31, that is, the angle indicated by the arrow K22 is the object elevation angle θ i_obj In other words, the object elevation angle θ i_obj It is the angle between plane PF11 and straight line L31.

[0191] At the same time, if Fig. 9 As shown, when the straight line obtained by projecting the straight line L31 onto the plane PF12 is set as the straight line L51, the angle between the straight line L51 and the line segment W22 on the plane PF12, that is, the angle indicated by the arrow K31 is the object rotation azimuth angle ψ_rot i_obj .

[0192] In addition, the angle between the straight line L51 and the straight line L31, that is, the angle indicated by the arrow K32 is the object rotation elevation angle θ_rot i_obj In other words, the object is rotated by an angle θ_roti _obj It is the angle between plane PF12 and line L31.

[0193] Specifically, for example, the above-mentioned object azimuth angle ψ can be calculated as follows i_obj 、Object elevation angle θ i_obj 、Object rotation angle ψ_rot i_obj and the object is rotated by an elevation angle θ_rot i_ob j, that is, relative direction information.

[0194] For example, a rotation matrix describing rotation in a three-dimensional space is expressed by the following expression (2).

[0195] [Mathematics 2]

[0196]

[0197] Note that in expression (2), as having a predetermined X 1 , Y 1 and Z 1The X axis of the three-dimensional orthogonal coordinate system 1 Y 1 Z 1 The coordinates (x, y, z) in the space are rotated by the rotation matrix, and the rotated coordinates (x', y', z') are obtained.

[0198] That is, in the calculation shown in expression (2), the second matrix from the right is used to calculate 1 Y 1 X 1 Y 1 Z 1 Space around Z 1 Axis rotation angle To get the rotated X 2 Y 2 Z 1 The rotation matrix of the space. In other words, the coordinate (x, y, z) in X 1 Y 1 Rotate the plane by the second rotation matrix from the right

[0199] In addition, the third matrix from the right on the right side of expression (2) is used to calculate the matrix in Y 2 Z 1 X 2 Y 2 Z 1 Space around X 2 The axis is rotated by angle θ to obtain the rotated X 2 Y 3 Z 2 The rotation matrix in space.

[0200] In addition, the fourth matrix from the right on the right side of expression (2) is used to calculate the 2 Z 2 X 2 Y 3 Z 2 Space around Y 3 The axis is rotated by an angle ψ to obtain the rotated X 3 Y 3 Z 3 The rotation matrix in space.

[0201] The relative direction calculation unit 32 generates relative direction information by using the rotation matrix shown in Expression (2).

[0202] Specifically, the relative direction calculation unit 32 calculates the relative direction based on the sound source position information (x o ,y o , z o ) and listener direction information The following expression (3) is calculated to obtain the coordinates (x o ,y o , z o )'s rotated coordinates (x o ',y o ', z o ').

[0203] [Mathematics 3]

[0204]

[0205] In the calculation of expression (3), set θ = -θv and ψ = -ψv, and calculate the rotation matrix.

[0206] The coordinates (x o ',y o ', z o ') indicates the position of the object in the listener coordinate system. However, the origin of the listener coordinate system here is not the listening position, but the origin O of the xyz coordinate system in the target space.

[0207] Next, the relative direction calculation unit 32 calculates the relative direction based on the listening position information (x v ,y v , z v ) and listener direction information The following expression (4) is calculated to obtain the coordinates (x v ,y v , z v )'s rotated coordinates (x v ',y v ', z v ').

[0208] [Mathematics 4]

[0209]

[0210] In the calculation of expression (4), set θ = -θv and ψ = -ψv, and calculate the rotation matrix.

[0211] The coordinates (x v ',y v ', z v ') indicates the listening position in the listener coordinate system. However, the origin of the listener coordinate system here is not the listening position, but the origin O of the xyz coordinate system in the target space.

[0212] Furthermore, the relative direction calculation unit 32 calculates the coordinates (xo ',y o ', z o ') and the coordinates (x v ',y v ', z v ') Calculate the following expression (5).

[0213] [Mathematics 5]

[0214]

[0215] Expression (5) is calculated to obtain the coordinates (x o ”,y o ”, z o ”). Coordinates (x o ”,y o ”, z o ”) indicates the relative position of an object as viewed from the listener.

[0216] The relative direction calculation unit 32 calculates the relative direction based on the coordinates (x o ″,y o ″,z o ″) Calculate the following expressions (6) and (7) to obtain the object azimuth angle ψ i_obj and the object elevation angle θ i_obj .

[0217] [Mathematics 6]

[0218] ψ i_obj =arctan(y o ” / x o ”) ···(6)

[0219] [Mathematics 7]

[0220] θ i_obj =arctan(Z o ” / sqrt(x o ” 2 +y o ” 2 )) ···(7)

[0221] In expression (6), x based on the x-coordinate and y-coordinate o ” and y o "Get the object azimuth ψ i_obj .

[0222] Note that, more specifically, in the calculation of Expression (6), based on y o " and x o"The result of zero determination of "is executed according to the situation and the abnormality processing is executed based on the result of the situation to calculate the object azimuth ψ i_obj However, a detailed description thereof will be omitted here.

[0223] In addition, in expression (7), based on the coordinates (x o ”,y o ”, z o ”) to obtain the object elevation angle θ i_obj Note that, more specifically, in the calculation of Expression (7), based on z o "The symbol and (x o ” 2 +y 0 ” 2 ) is determined as a result of zero determination, and a case-by-case proof process is performed based on the case-by-case proof result to calculate the object elevation angle θ i_obj However, a detailed description thereof will be omitted here.

[0224] The object azimuth angle ψ is obtained by the above calculation. i_obj and the object elevation angle θ i_obj In the case of i_obj and the object is rotated by an elevation angle θ_rot i_obj .

[0225] That is, the relative direction calculation unit 32 calculates the relative direction based on the listening position information (x v ,y v , z v ) and sound source direction information The following expression (8) is calculated to obtain the coordinates (x v ,y v , z v )'s rotated coordinates (x v ',y v ', z v ').

[0226] [Mathematics 8]

[0227]

[0228] In the calculation of expression (8), set θ=-θ o and ψ=-ψ o , and calculate the rotation matrix.

[0229] The coordinates (x v ',y v ', z v') indicates the listening position (the position of the listener) in the object coordinate system. However, the origin of the object coordinate system here is not the position of the object, but the origin O of the xyz coordinate system in the target space.

[0230] Next, the relative direction calculation unit 32 calculates the relative direction based on the sound source position information (x o ,y o , z o ) and sound source direction information The following expression (9) is calculated to obtain the coordinates (x o ,y o , z o )'s rotated coordinates (x o ',y o ', z o ').

[0231] [Mathematics 9]

[0232]

[0233] In the calculation of expression (9), set θ=-θ o and ψ=-ψ o , and calculate the rotation matrix.

[0234] The coordinates (x o ',y o ', z o ') indicates the position of the object in the object coordinate system. However, the origin of the object coordinate system here is not the position of the object, but the origin O of the xyz coordinate system in the target space.

[0235] Furthermore, the relative direction calculation unit 32 calculates the coordinates (x v ',y v ', z v ') and the coordinates (x o ',y o ', z o ') Calculate the following expression (10).

[0236] [Mathematics 10]

[0237]

[0238] Expression (10) is calculated to obtain the coordinates (x v ”,y v ”, z v ”). Coordinates (x v ”,yv ”, z v ”) indicates the relative position of the listening position as viewed from the subject.

[0239] The relative direction calculation unit 32 calculates the relative direction based on the coordinates (x v ″,y v ″,z v ″) Calculate the following expressions (11) and (12) to obtain the object rotation azimuth angle ψ_rot i_obj and the object is rotated by an elevation angle θ_rot i_obj .

[0240] [Mathematics 11]

[0241] ψ_rot i_obj =arctan(y v ” / x v ”) ···(11)

[0242] [Mathematics 12]

[0243] θ_rot i_obj =arctan(z v ” / sqrt(x v ” 2 +y v ” 2 )) ···(12)

[0244] Expression (11) is calculated in a similar manner to Expression (6) to obtain the object rotation azimuth angle ψ_rot i_obj In addition, expression (12) is calculated in a similar manner to expression (7) to obtain the object rotation elevation angle θ_rot i_obj .

[0245] The relative direction calculation unit 32 performs the above-described processing on each frame of the audio data of the plurality of objects.

[0246] Therefore, the object azimuth angle ψ of each object in each frame can be obtained. i_obj 、Object elevation angle θ i_obj 、Object rotation angle ψ_rot i_obj and the object is rotated by an elevation angle θ_rot i_obj relative direction information.

[0247] Using the relative direction information obtained as described above makes it possible to localize the sound image of each object according to the listening position, the direction of the listener, and the movement and rotation of the object, thereby providing a higher sense of realism.

[0248] (Directional characteristics database unit)

[0249] The directional characteristic database unit 23 records directional characteristic data for each type of object, that is, for each sound source type.

[0250] The directional characteristic data is, for example, a function that uses the azimuth and elevation angles observed from the object as parameters and obtains the gain in the propagation direction and the spherical harmonic coefficient indicated by the azimuth and elevation angles.

[0251] Note that, instead of a function, the directional characteristic data may be data in a table form, that is, for example, a table associating the azimuth and elevation angles observed from the object with the gain in the propagation direction and the spherical harmonic coefficients indicated by the azimuth and elevation angles.

[0252] (Directional Rendering Unit)

[0253] The directionality rendering unit 33 performs rendering processing based on the audio data of each object, the directional characteristic data obtained for each object, the relative distance information and relative direction information, the listening position information and the listener direction information, and generates a reproduction signal for the corresponding reproduction unit 12 serving as a target device.

[0254] <Description of Content Reproduction Processing>

[0255] Next, the operation of the signal processing device 11 will be described.

[0256] That is, the following will refer to Fig.10 The flowchart describes the content reproduction process performed by the signal processing device 11.

[0257] Note that here, description is made assuming that the content to be reproduced is free viewpoint content and that directional characteristic data of each sound source type is acquired in advance and recorded in the directional characteristic database unit 23 .

[0258] In step S11, the acquisition unit 21 acquires metadata and audio data of one frame of each object included in the content from the transmission device. In other words, the metadata and audio data are acquired at predetermined time intervals.

[0259] The acquisition unit 21 supplies the sound source type information included in the acquired metadata of each object to the direction characteristic database unit 23 , and supplies the acquired audio data of each object to the directionality rendering unit 33 .

[0260] In addition, the acquisition unit 21 converts the sound source position information (x o ,y o , z o ) is provided to the relative distance calculation unit 31 and the relative direction calculation unit 32, and the sound source direction information included in the metadata of each object is obtained. Provided to the relative direction calculation unit 32.

[0261] In step S12 , the listening position specifying unit 22 specifies the listening position and direction of the listener.

[0262] That is, the listening position specifying unit 22 determines the listening position and direction of the listener in response to the listener's operation or the like, and generates listening position information (x v ,y v , z v ) and listener direction information

[0263] The listening position specifying unit 22 converts the obtained listening position information (x v ,y v , z v ) is provided to the relative distance calculation unit 31, the relative direction calculation unit 32 and the directional rendering unit 33, and the obtained listener direction information is Provided to the relative direction calculation unit 32 and the directional rendering unit 33.

[0264] Note that in the case of fixed viewpoint content, for example, the listening position information is set to (0, 0, 0), and the listener direction information is also set to (0, 0, 0).

[0265] In step S13, the relative distance calculation unit 31 calculates the relative distance based on the sound source position information (x o ,y o , z o ) and the listening position information (x v ,y v , z v ) Calculate the relative distance d o , and the relative distance information indicating the calculation result is provided to the directional rendering unit 33. For example, in step S13, the above expression (1) is calculated for each object, and the relative distance d is calculated for each object. o .

[0266] In step S14, the relative direction calculation unit 32 calculates the relative direction based on the sound source position information (x o ,y o , z o ) and sound source direction information and the listening position information (x v ,y v , z v ) and listener direction information The relative direction between the listener and the object is calculated, and relative direction information indicating the calculation result is supplied to the directivity rendering unit 33 .

[0267] For example, the relative direction calculation unit 32 calculates the above-mentioned expressions (3) to (7) for each object, thereby obtaining the object azimuth angle ψ of each object. i_obj and the object elevation angle θ i_obj .

[0268] Furthermore, for example, the relative direction calculation unit 32 calculates the above-mentioned expressions (8) to (12) for each object, thereby obtaining the object rotation azimuth angle ψ_rot of each object. i_obj and the object is rotated by an elevation angle θ_rot i_obj .

[0269] The relative direction calculation unit 32 includes the object azimuth angle ψ obtained for each object. i_obj 、Object elevation angle θ i_obj 、Object rotation angle ψ_rot i_obj and the object is rotated by an elevation angle θ_rot i_obj The information is provided to the directional rendering unit 33 as relative direction information.

[0270] In step S15 , the directionality rendering unit 33 acquires the direction characteristic data from the direction characteristic database unit 23 .

[0271] For example, in a case where metadata is acquired for each object and sound source type information included in the metadata is supplied to the directional characteristic database unit 23 in step S11 , the directional characteristic database unit 23 outputs directional characteristic data for each object.

[0272] That is, the directional characteristic database unit 23 reads directional characteristic data of the sound source type indicated by the sound source type information from the recorded plurality of directional characteristic data for each piece of sound source type information supplied from the acquisition unit 21 , and outputs the directional characteristic data to the directivity rendering unit 33 .

[0273] Directivity rendering unit 33 acquires the direction characteristic data output for each object from direction characteristic database unit 23 as described above, thereby obtaining the direction characteristic data for each object.

[0274] In step S16, the directionality rendering unit 33 generates a signal based on the audio data supplied from the acquisition unit 21, the directionality data supplied from the directionality database unit 23, the relative distance information supplied from the relative distance calculation unit 31, the relative direction information supplied from the relative direction calculation unit 32, and the listening position information (x ) supplied from the listening position specifying unit 22. v ,y v , z v) and listener direction information Perform rendering processing.

[0275] Note that listening to the position information (x v ,y v , z v ) and listener direction information It only needs to be used for rendering processing as needed and does not have to be used for rendering processing.

[0276] For example, the directivity rendering unit 33 performs processing for VBAP or wave field synthesis, HRTF convolution processing, or the like as rendering processing, thereby generating a reproduction signal for reproducing the sound of an object (content) at the listening position.

[0277] Here, an example of performing VBAP as a rendering process will be described. Therefore, in this case, the reproduction unit 12 includes a plurality of speakers.

[0278] Furthermore, for simplicity of description, an example of a single object included in content will be described.

[0279] First, the directional rendering unit 33 calculates the relative distance d indicated by the relative distance information. o The following expression (13) is calculated to obtain the gain value gain for reproducing the distance attenuation: i_obj .

[0280] [Mathematics 13]

[0281] gain i_obj =1.0 / power(d o , 2.0) …(13)

[0282] Note that power(d o , 2.0) indicates the distance used to calculate the relative distance d o Here, an example using the inverse square law will be described. However, the calculation of the gain value for reproducing the distance attenuation is not limited, and any other method may be used.

[0283] Next, the directional rendering unit 33, for example, rotates the azimuth angle ψ_rot based on the object included in the relative direction information. i _ ob j and the object's rotation angle θ_rot i_obj The following expression (14) is calculated to obtain the gain value dir_gain according to the directional characteristics of the object. i_obj .

[0284] [Mathematics 14]

[0285] dir_gaini_obj =dir(i,ψ_rot i_obj ,θ_rot i_obj ) …(14)

[0286] In expression (14), dir(i, ψ_rot i_obj ,θ_rot i_obj ) represents a gain function of a value i corresponding to the sound source type information provided as the directional characteristic data.

[0287] Therefore, the directional rendering unit 33 rotates the object by an azimuth angle ψ_rot i_obj and the object is rotated by an elevation angle θ_rot i_obj Substitute the gain function to calculate expression (14) to obtain the gain value dir_gain i_obj As the calculation result.

[0288] That is, in expression (14), the rotation angle ψ_rot from the object i_obj 、The object's rotation angle θ_rot i_obj And directional characteristic data to obtain the gain value dir_gain i_obj .

[0289] The gain value dir_gain obtained as described above i_obj Gain correction for adding the transfer characteristic of the sound propagating from the object toward the listener, in other words, gain correction for reproducing the sound propagation according to the directional characteristic of the object is implemented.

[0290] Note that the distance to the object may be included as a parameter (variable) of the gain function used as the directional characteristic data as described above, thereby obtaining the distance by using the gain value dir_gain outputted by the gain function. i_obj Gain correction that reproduces not only the directional characteristics but also the distance attenuation is achieved. In this case, the relative distance d indicated by the relative distance information o The distance used as a parameter as the gain function.

[0291] In addition, the directional rendering unit 33 performs the directional rendering based on the object azimuth angle ψ included in the relative direction information. i_obj and the object elevation angle θ i_obj VBAP is performed to obtain a reproduction gain value VBAP_gain of a channel corresponding to each of a plurality of speakers included in the reproduction unit 12. i_spk .

[0292] Then, the directional rendering unit 33 generates a directional rendering signal based on the audio data obj_audio of the object. i_obj , gain value of distance attenuation gain i_obj, Directional characteristic gain value dir_gain i_obj And the reproduction gain value VBAP_gain of the channel corresponding to the speaker i_spk The following expression (15) is calculated to obtain the reproduced signal speaker_signal supplied to the speaker: i_spk .

[0293] [Mathematics 15]

[0294] speaker_signal i_spk

[0295] =obj_audio i_obj ×VBAP_gain i_spk ×gain i_obj ×dir_gain i_obj

[0296] ···(15)

[0297] Here, Expression (15) is calculated for each combination of a speaker included in the reproduction unit 12 and an object included in the content, and a reproduction signal speaker_signal is obtained for each of the plurality of speakers included in the reproduction unit 12. i_spk .

[0298] Thus, gain correction for reproducing distance attenuation, gain correction for reproducing sound propagation according to directional characteristics, and VBAP processing for localizing a sound image at a desired position are realized.

[0299] At the same time, the gain value dir_gain obtained from the directional characteristic data i_obj is the case where both the directional characteristics and the distance attenuation are considered, that is, at the relative distance d indicated by the relative distance information o When included as a parameter of the gain function, the following expression (16) is calculated.

[0300] That is, the directional rendering unit 33 generates a directivity image based on the audio data obj_audio of the object. i_obj , Directional characteristic gain value dir_gain i_obj and reproduced gain value VBAP_gain i_spk The following expression (16) is calculated to obtain the reproduced signal speakersignal i_spk .

[0301] [Mathematics 16]

[0302] speaker_signal i_spk

[0303] =obj_audio i_obj ×VBAP_gain i_spk ×dir_gain i_obj ···(16)

[0304] In the case where the reproduction signal is obtained as described above, the directivity rendering unit 33 finally converts the reproduction signal speaker_signal obtained for the current frame into i_spk The reproduced signal speaker_signal of the previous frame of the current frame i_spk Overlap addition is performed, thereby obtaining a final reproduced signal.

[0305] Note that an example in which VBAP is performed as the rendering process has been described herein, but in the case of performing HRTF convolution processing as the rendering process, a reproduction signal can be obtained by performing similar processing.

[0306] Here, a case will be described in which a reproduction signal of a headphone considering directional characteristics of an object is generated by using an HRTF database including HRTF for each user according to a distance, azimuth, and elevation indicating a relative positional relationship between the object and the user (listener).

[0307] Specifically, here, the directivity rendering unit 33 holds an HRTF database including HRTFs from virtual speakers corresponding to real speakers used when measuring HRTFs, and the reproduction unit 12 is a headphone.

[0308] Note that here, a case where an HRTF database is prepared for each user in consideration of differences in personal characteristics of each user will be described. However, an HRTF database common to all users may be used.

[0309] In this example, personal ID information for identifying a single user is set to j, and the azimuth and elevation angles indicating the arrival direction of the sound from the sound source (virtual speaker), i.e., the object, to the user's ear are denoted by ψ L and ψ R and θ L and θ R Here, the azimuth angle ψ L and the elevation angle θ L are the azimuth and elevation angles indicating the direction to the user's left ear, and the azimuth angle ψ R and the elevation angle θ R are the azimuth and elevation angles indicating the direction to the user's right ear.

[0310] In addition, the HRTF used as the transfer characteristic from the sound source to the user's left ear will be specifically composed of HRTF(j, ψ L ,θ L), and the HRTF used as the transfer characteristic from the sound source to the user's left ear will be specifically represented by HRTF(j, ψ R ,θ R )express.

[0311] Note that HRTFs to each of the left and right ears of the user may be prepared for each arrival direction and distance from the sound source, and distance attenuation may also be reproduced by HRTF convolution.

[0312] Furthermore, the directional characteristic data may be a function indicating the transfer characteristic from the sound source to each direction, or may be a gain function as in the above-mentioned example of VBAP, and in either case, the object rotation azimuth angle ψ_rot i_obj and the object is rotated by an elevation angle θ_rot i_obj Used as a function argument.

[0313] In addition, the convergence angle of the user's left and right ears relative to the object (i.e., the difference in sound arrival angles between the object and each ear of the user caused by the user's facial width) can be considered to obtain the object rotation azimuth and object rotation elevation for each of the left and right ears.

[0314] The convergence angle here is the angle between a straight line connecting the left ear of the user (listener) and the object and a straight line connecting the right ear of the user and the object.

[0315] Hereinafter, among the object rotation azimuth and the object rotation elevation included in the relative direction information, the object rotation azimuth and the object rotation elevation obtained for the left ear of the user are denoted by ψ_rot i_obj_1 and θ_rot i_obj_1 Specific expression.

[0316] Similarly, in the following, among the object rotation azimuth and the object rotation elevation included in the relative direction information, the object rotation azimuth and the object rotation elevation obtained for the right ear of the user are denoted by ψ_rot i_obj_r and θ_rot i_obj_r Specific expression.

[0317] First, the directional rendering unit 33 calculates the above expression (13), thereby obtaining a gain value gain for reproducing distance attenuation. i_obj .

[0318] Note that in the case where HRTF is prepared for each sound arrival direction and distance from the sound source as an HRTF database and distance attenuation can be reproduced by HRTF convolution, the gain value gain is not calculated. i_obj Furthermore, distance attenuation can be reproduced by convolution of the transfer characteristics obtained from the directional characteristic data instead of HRTF convolution.

[0319] Next, the directional rendering unit 33 acquires, for example, a transfer characteristic according to the directional characteristic of the object based on the directional characteristic data and the relative direction information.

[0320] For example, when a function for obtaining transfer characteristics is provided as directional characteristic data and the function uses distance, azimuth and elevation as parameters, the directivity rendering unit 33 calculates the following expression (17) based on relative distance information, relative direction information and directional characteristic data.

[0321] [Mathematics 17]

[0322] dir_func i_obj_l =dir(i, d i_obj , ψ_rot i_obj_l ,θ_rot i_obj_l )

[0323] dir_func i_obj_r =dir(i, d i_obj , ψ_rot i_obj_r ,θ_rot i_obj_r )

[0324] ···(17)

[0325] That is, in Expression (17), the directional rendering unit 33 converts the relative distance d indicated by the relative distance information into o Set to d i_obj .

[0326] Then, the directional rendering unit 33 converts the relative distance d o 、Object rotation angle ψ_rot i_obj_1 and the object is rotated by an elevation angle θ_rot i_obj_1 Substitute the left ear function dir(i, d i_obj , ψ_rot i_obj_1 ,θ_rot i_obj_1 ), thereby obtaining the transfer characteristic dir_func of the left ear i_obj_1 .

[0327] Similarly, the directional rendering unit 33 converts the relative distance d o 、Object rotation angle ψ_rot i_obj_r and the object is rotated by an elevation angle θ_rot i_obj_r Substitute the right ear function dir(i, d i_obj , ψ_rot i_obj_r ,θ_rot i_obj_r ), thereby obtaining the transfer characteristic dir_func of the right ear i_obj_r .

[0328] In this case, distance decay is also passed through the property dir_func i_obj_1 and dir_func i_obj_r The convolution reproduction of .

[0329] In addition, the directional rendering unit 33 is based on the object azimuth angle ψ i_obj and the object elevation angle θ i_obj Obtain the left ear HRTF (j, ψ L ,θ L ) and the HRTF of the right ear (j, ψ R ,θ R ). Here, for example, the setting ψ is read from the HRTF database. L =ψ i_obj and θ L =θ i_obj HRTF(j,ψ L ,θ L ). Note that the object azimuth and object elevation can also be obtained for each of the left ear and the right ear.

[0330] In the case where the transfer characteristics and HRTF of the left and right ears are obtained through the above-mentioned processing, based on the transfer characteristics, HRTF and the audio data obj_audio of the object i_obj Reproduction signals for the left and right ears to be supplied to the headphones serving as the reproduction unit 12 are obtained.

[0331] Specifically, for example, the transfer characteristic dir_func is obtained from the directional characteristic data in consideration of both the directional characteristic and the distance attenuation. i_obj_l and dir_func i_obj_r In the case where the transfer characteristic is obtained from Expression (17), the directivity rendering unit 33 calculates the following Expression (18) to obtain the reproduced signal HPout for the left ear: L and the reproduced signal HPout for the right ear R .

[0332] [Mathematics 18]

[0333] HPout L =obj_audio i_obj *dir_func i_obj_l *HRTF(j,ψ L ,θ L )

[0334] HPout R =obj_audio i_obj *dir_func i_obj_r *HRTF(j,ψ R,θ R )

[0335] ···(18)

[0336] Note that in expression (18), * represents convolution processing.

[0337] So here, passing the property dir_func i_obj_1 and HRTF(j,ψ L ,θ L ) is convolved into the audio data obj_audio i_obj To obtain the left ear reproduction signal HPout L Similarly, passing the property dir_func i_obj_r and HRTF(j,ψ R ,θ R ) is convolved into the audio data obj_audio i_obj To obtain the reproduced signal HPout for the right ear R Furthermore, in the case where the distance attenuation is reproduced by the HRTF, the reproduced signal is also obtained by calculation similar to that of Expression (18).

[0338] Meanwhile, for example, in the case of obtaining the transfer characteristic obtained from the directional characteristic data and the HRTF without considering the distance attenuation, the directivity rendering unit 33 calculates the following Expression (19) to obtain a reproduction signal.

[0339] [Mathematics 19]

[0340] HPout L =obj_audio i_obj *dir_func iobj_l *HRTF(j,ψ L ,θ L )*gain i_obj

[0341] HPout R =obj_audio j_obj *dir_func i_obj_r *HRTF(j,ψ R ,θ R )*gain i_obj

[0342] ···(19)

[0343] In expression (19), the audio data obj_audio i_obj Not only is it subjected to the convolution processing performed in Expression (18), but it is also subjected to convolution with the gain value gain for reproducing the distance attenuation. i_objTherefore, the left ear reproduction signal HPout is obtained. L and the reproduced signal HPout for the right ear R . Gain value gain i_obj Obtained from the above expression (13).

[0344] After the above processing, the reproduced signal HPout is obtained. L and HPout R In the case of , the directional rendering unit 33 performs overlap addition of the reproduced signal and the reproduced signal of the previous frame, thereby obtaining the final reproduced signal HPout L and HPout R .

[0345] Furthermore, in the case of performing wave field synthesis processing as rendering processing, that is, in the case of forming a sound field including the sound of an object by wave field synthesis using a plurality of speakers as the reproduction unit 12 , a reproduction signal is generated as follows.

[0346] Here, an example in which a speaker driving signal to be supplied to a speaker included in the reproducing unit 12 is generated as a reproduction signal by using a spherical harmonic function will be described.

[0347] The external sound field (i.e., the sound pressure p(r', ψ, θ)) at a position outside a certain radius r from a predetermined sound source (i.e., a position where the radius (distance) from the sound source is r' (where r'>r) and the azimuth and elevation angles indicating the direction observed from the sound source are ψ and θ) can be expressed by the following expression (20).

[0348] [Mathematics 20]

[0349]

[0350] Note that in expression (20), Y n m (ψ, θ) represents a spherical harmonic function, and n and m represent the degree and order of the spherical harmonic function. n (1) (kr) is the Hankel function of the first kind, and k represents the wave number.

[0351] Furthermore, in Expression (20), X(k) represents a reproduced signal expressed in the frequency domain, and P nm (r) represents the spherical harmonic spectrum of a sphere with a radius (distance) of r. Here, the signal X(k) in the frequency domain corresponds to the audio data of the object.

[0352] For example, in the case where a measurement microphone array for measuring directional characteristics has a spherical shape with a radius of r, the sound pressure at a position with a radius of r of the sound propagating in all directions from a sound source existing at the center of the sphere (the measurement microphone array) can be measured by using the measurement microphone array. Specifically, since the directional characteristics vary depending on the sound source, the observed sound including the directional characteristic information is obtained by measuring the sound from the sound source at each position.

[0353] Spherical Harmonic Spectrum P n m (r) The sound pressure p(r, ψ, θ) can be expressed by the following expression (21) by using such measurement observation sound pressure p(r, ψ, θ) measured by the measurement microphone array.

[0354] [Mathematics 21]

[0355]

[0356] Note that in expression (21), denotes the integration range and in particular denotes the integral over the radius r.

[0357] Such a spherical harmonic spectrum P n m (r) is data indicating the directional characteristics of the sound source. Therefore, for example, the spherical harmonic spectrum P of each combination of the nth and mth orders in a predetermined domain is measured in advance for each sound source type. n m In the case of (r), the function shown in the following expression (22) can be used as the directional characteristic data dir(i_obj, d i_obj ).

[0358] [Mathematics 22]

[0359]

[0360] Note that in expression (22), i_obj represents the sound source type, d i_obj represents the distance from the sound source, and the distance d i_obj Relative distance d o Corresponding to the nth and mth order of such a set of directional characteristic data dir(i_obj, d i_obj ) is data indicating transfer characteristics in each direction (ie, in all directions) determined based on the azimuth angle ψ and the elevation angle θ in consideration of the amplitude and the phase.

[0361] In the case where the relative positional relationship between the object and the listening position does not change, a reproduction signal that also takes the directional characteristics into consideration can be obtained from the above expression (20).

[0362] However, even in the case where the relative positional relationship between the object and the listening position changes, the distance d i_obj Determined point (d i_obj ,ψ,θ) at the sound pressure p(d i_obj , ψ, θ) can be obtained by rotating the object based on the azimuth angle ψ_rot i_obj and the object is rotated by an elevation angle θ_rot i_obj Direction characteristic data dir(i_obj, d i_obj ) is obtained by performing a rotation operation, as shown in the following expression (23).

[0363] [Mathematics 23]

[0364]

[0365] Note that in the calculation of expression (23), the relative distance d o Substitute the distance d i_obj The audio data of the object is substituted into X(k), and thus the sound pressure p(d i_obj , ψ, θ). Then, the sound pressure p (d i_obj , ψ, θ) to obtain the sum of i_obj , ψ, θ), that is, the reproduced signal.

[0366] Therefore, in order to generate a reproduction signal for wave field synthesis, as processing in step S16, Expression (23) is calculated for each wave number k of each object, and a reproduction signal is generated based on the calculation result.

[0367] In the case where a reproduction signal to be supplied to the reproduction unit 12 is obtained by the above-described rendering process, the process proceeds from step S16 to step S17.

[0368] In step S17, the directivity rendering unit 33 supplies the reproduction signal obtained by the rendering process to the reproduction unit 12 and causes the reproduction unit 12 to output the sound. Thus, the sound of the content, that is, the sound of the object is reproduced.

[0369] In step S18, the signal generation unit 24 determines whether to terminate the process of reproducing the sound of the content. For example, in the case where the process is performed on all frames and the reproduction of the content ends, it is determined that the process is to be terminated.

[0370] In the case where it is determined in step S18 that the processing has not been terminated, the processing returns to step S11 and the above-mentioned processing is repeatedly executed.

[0371] Meanwhile, in the case where it is determined in step S18 that the processing is to be terminated, the content reproduction processing is terminated.

[0372] As described above, the signal processing device 11 generates relative distance information and relative direction information, and performs rendering processing taking directional characteristics into consideration by using the relative distance information and relative direction information. This makes it possible to reproduce the sound propagation according to the directional characteristics of the object, thereby providing a higher sense of reality.

[0373] <Computer Configuration Example>

[0374] Incidentally, the above series of processing can be performed by hardware or software. In the case where a series of processing is performed by software, the program forming the software is installed in a computer. Here, the computer includes, for example, a computer built into dedicated hardware, a general-purpose personal computer that can perform various functions by installing various programs, etc.

[0375] Fig.11 : is a block diagram showing a configuration example of hardware of a computer that executes the above-described series of processes by a program.

[0376] A central processing unit (CPU) 501 , a read only memory (ROM) 502 , and a random access memory (RAM) 503 are connected to each other via a bus 504 in the computer.

[0377] The bus 504 is further connected to an input / output interface 505. The input / output interface 505 is connected to an input unit 506, an output unit 507, a recording unit 508, a communication unit 509, and a drive 510.

[0378] The input unit 506 includes a keyboard, a mouse, a microphone, an imaging element, etc. The output unit 507 includes a display, a speaker, etc. The recording unit 508 includes a hard disk, a nonvolatile memory, etc. The communication unit 509 includes a network interface, etc. The drive 510 drives a removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0379] In the computer configured as described above, for example, the CPU 501 loads the program recorded in the recording unit 508 to the RAM 503 through the input / output interface 505 and the bus 504 and executes the program to perform the above-described series of processing.

[0380] The program executed by the computer (CPU 501) can be provided by being recorded on the removable recording medium 511 as a package medium, etc. In addition, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0381] In the computer, by attaching a removable recording medium 511 to a drive 510, the program can be installed in a recording unit 508 via an input / output interface 505. In addition, the program can be received by a communication unit 509 through a wired or wireless transmission medium and installed in the recording unit 508. In addition, the program can be pre-installed in the ROM 502 or the recording unit 508.

[0382] Note that the program executed by the computer may be a program that executes processing in time series in the order described in this specification, or may be a program that executes processing in parallel or at necessary timing such as when a call is executed.

[0383] Furthermore, the embodiments of the present technology are not limited to the above embodiments, and various modifications can be made without departing from the gist of the present technology.

[0384] For example, the present technology may have a configuration of cloud computing in which a single function is shared and jointly processed by a plurality of devices via a network.

[0385] Furthermore, each step described in the above flowchart may be executed by a single device, or may be executed in a shared manner by a plurality of devices.

[0386] Furthermore, in the case where a single step includes a plurality of processes, the plurality of processes included in the single step may be executed by a single device or may be executed by being shared by a plurality of devices.

[0387] Furthermore, the present technology may also have the following configurations.

[0388] (1) A signal processing device comprising:

[0389] an acquisition unit that acquires metadata and audio data of the audio object, the metadata including position information indicating a position of the audio object and direction information indicating a direction of the audio object; and

[0390] A signal generating unit generates a reproduction signal for reproducing the sound of an audio object at the listening position based on the listening position information indicating the listening position, the listener direction information indicating the direction of the listener at the listening position, the position information, the direction information, and the audio data.

[0391] (2) The signal processing device according to (1), wherein:

[0392] The acquisition unit acquires metadata at predetermined time intervals.

[0393] (3) The signal processing device according to (1) or (2), wherein:

[0394] The signal generating unit generates a reproduction signal based on directional characteristic data indicating directional characteristics of an audio object, listening position information, listener direction information, position information, direction information, and audio data.

[0395] (4) A signal processing device according to (3), wherein:

[0396] The signal generating unit generates a reproduction signal based on the directional characteristic data determined for the type of the audio object.

[0397] (5) The signal processing device according to (3) or (4), wherein:

[0398] The direction information includes an azimuth indicating a direction of the audio object.

[0399] (6) The signal processing device according to (3) or (4), wherein:

[0400] The direction information includes an azimuth and an elevation indicating a direction of an audio object.

[0401] (7) The signal processing device according to (3) or (4), wherein:

[0402] The direction information includes an azimuth and an elevation indicating a direction of the audio object and a tilt angle indicating a rotation of the audio object.

[0403] (8) The signal processing device according to any one of (3) to (7), wherein:

[0404] The listening position information indicates a predetermined and fixed listening position, and the listener direction information indicates a predetermined and fixed direction of a listener.

[0405] (9) The signal processing device according to (8), wherein:

[0406] The position information includes an azimuth and an elevation indicating a direction of the audio object viewed from a listening position and a radius indicating a distance from the listening position to the audio object.

[0407] (10) The signal processing device according to any one of (3) to (7), wherein:

[0408] The listening position information indicates an arbitrarily determined listening position, and the listener direction information indicates the direction of an arbitrarily determined listener.

[0409] (11) The signal processing device according to (10), wherein:

[0410] The position information is coordinates of an orthogonal coordinate system indicating the position of the audio object.

[0411] (12) The signal processing device according to any one of (3) to (11), wherein:

[0412] The signal generation unit generates a reproduction signal based on:

[0413] Directional characteristic data,

[0414] relative distance information obtained based on the listening position information and the position information and indicating a relative distance between the audio object and the listening position,

[0415] relative direction information obtained based on the listening position information, the listener direction information, the position information, and the direction information and indicating a relative direction between the audio object and the listener, and

[0416] Audio data.

[0417] (13) A signal processing device according to (12), wherein:

[0418] The relative direction information includes an azimuth and an elevation indicating a relative direction between the audio object and the listener.

[0419] (14) The signal processing device according to (12) or (13), wherein:

[0420] The relative direction information includes information indicating the direction of the listener viewed from the audio object and information indicating the direction of the audio object viewed from the listener.

[0421] (15) A signal processing device according to (14), wherein:

[0422] The signal generating unit generates a reproduction signal based on information indicating a transfer characteristic of a direction of a listener viewed from the audio object, the information being obtained based on the directional characteristic data and the information indicating the direction of the listener viewed from the audio object.

[0423] (16) A signal processing method, comprising:

[0424] Enable the signal processing device:

[0425] acquiring metadata and audio data of the audio object, the metadata including position information indicating a position of the audio object and direction information indicating a direction of the audio object; and

[0426] A reproduction signal for reproducing the sound of an audio object at the listening position is generated based on the listening position information indicating the listening position, the listener direction information indicating the direction of the listener at the listening position, the position information, the direction information, and the audio data.

[0427] (17) A program for causing a computer to execute a process comprising the following steps:

[0428] a step of acquiring metadata and audio data of an audio object, the metadata comprising position information indicating a position of the audio object and direction information indicating a direction of the audio object; and

[0429] A step of generating a reproduction signal for reproducing the sound of the audio object at the listening position based on the listening position information indicating the listening position, the listener direction information indicating the direction of the listener at the listening position, the position information, the direction information, and the audio data.

[0430] Reference Mark List

[0431] 11 Signal processing device

[0432] 21 Get Unit

[0433] 22 Listening position designation unit

[0434] 23 Directional characteristics database unit

[0435] 24 Signal generation unit

[0436] 31 Relative distance calculation unit

[0437] 32 Relative direction calculation unit

[0438] 33 Directional Rendering Units.

Claims

1. A signal processing device, comprising: an acquiring unit, acquiring metadata and audio data of an audio object, wherein the metadata includes position information indicating a position of the audio object and direction information indicating a direction of the audio object; as well as a signal generating unit that generates a reproduction signal for reproducing the sound of the audio object at the listening position based on listening position information indicating a listening position, listener direction information indicating a direction of a listener at the listening position, the position information, the direction information, and the audio data, wherein the signal generating unit generates the reproduction signal based on directional characteristic data indicating directional characteristics of the audio object, the listening position information, the listener direction information, the position information, the direction information, and the audio data, And wherein the directional characteristic data indicates a directional characteristic of an audio object of the type indicated by the type of the audio object, and the directional characteristic of the audio object indicates a transfer characteristic in each direction observed from the audio object.

2. The signal processing device according to claim 1, wherein: The acquiring unit acquires the metadata at predetermined time intervals.

3. The signal processing device according to claim 1, wherein: The direction information includes an azimuth indicating a direction of the audio object.

4. The signal processing device according to claim 1, wherein: The direction information includes an azimuth and an elevation indicating a direction of the audio object.

5. The signal processing device according to claim 1, wherein: The direction information includes an azimuth and an elevation indicating a direction of the audio object and a tilt angle indicating a rotation of the audio object.

6. The signal processing device according to claim 1, wherein: The listening position information indicates the predetermined and fixed listening position, and the listener direction information indicates the predetermined and fixed direction of the listener.

7. The signal processing device according to claim 6, wherein: The position information includes an azimuth and an elevation indicating a direction of the audio object viewed from the listening position and a radius indicating a distance from the listening position to the audio object.

8. The signal processing device according to claim 1, wherein: The listening position information indicates the arbitrarily determined listening position, and the listener direction information indicates the arbitrarily determined direction of the listener.

9. The signal processing device according to claim 8, wherein: The position information is coordinates of an orthogonal coordinate system indicating the position of the audio object.

10. The signal processing device according to claim 1, wherein: The signal generating unit generates the reproduction signal based on: The directional characteristic data, relative distance information obtained based on the listening position information and the position information and indicating a relative distance between the audio object and the listening position, relative direction information obtained based on the listening position information, the listener direction information, the position information, and the direction information and indicating a relative direction between the audio object and the listener, and the audio data.

11. The signal processing device according to claim 10, wherein: The relative direction information includes an azimuth and an elevation indicating the relative direction between the audio object and the listener.

12. The signal processing device according to claim 10, wherein: The relative direction information includes information indicating a direction of the listener viewed from the audio object and information indicating a direction of the audio object viewed from the listener.

13. The signal processing device according to claim 12, wherein: The signal generating unit generates the reproduction signal based on information indicating a transfer characteristic of a direction of the listener viewed from the audio object, the information being obtained based on the directional characteristic data and information indicating the direction of the listener viewed from the audio object.

14. A signal processing method, comprising: Enable the signal processing device: Acquire metadata and audio data of an audio object, the metadata including position information indicating a position of the audio object and direction information indicating a direction of the audio object; and generating a reproduction signal for reproducing the sound of the audio object at the listening position based on listening position information indicating a listening position, listener direction information indicating a direction of a listener at the listening position, the position information, the direction information, and the audio data, wherein the reproduction signal is generated based on directional characteristic data indicating directional characteristics of the audio object, the listening position information, the listener direction information, the position information, the directional information and the audio data, And wherein the directional characteristic data indicates a directional characteristic of an audio object of the type indicated by the type of the audio object, and the directional characteristic of the audio object indicates a transfer characteristic in each direction observed from the audio object.

15. A computer-readable storage medium storing a program, which, when executed, causes a computer to perform a process comprising the following steps: a step of acquiring metadata and audio data of an audio object, the metadata comprising position information indicating a position of the audio object and direction information indicating a direction of the audio object; and a step of generating a reproduction signal for reproducing the sound of the audio object at the listening position based on listening position information indicating a listening position, listener direction information indicating a direction of a listener at the listening position, the position information, the direction information, and the audio data, in, generating the reproduction signal based on directional characteristic data indicating directional characteristics of the audio object, the listening position information, the listener direction information, the position information, the directional information, and the audio data, And wherein the directional characteristic data indicates a directional characteristic of an audio object of the type indicated by the type of the audio object, and the directional characteristic of the audio object indicates a transfer characteristic in each direction observed from the audio object.

Images