Signal processing device, method, and program

By combining polar and absolute coordinate systems for encoding position information in free-viewpoint audio, the method optimizes transmission efficiency and reduces computational overhead in free-viewpoint audio systems, ensuring high-quality audio reproduction.

JP2024166232AActive Publication Date: 2024-11-28SONY GROUP CORP
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2024152849
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2019-12-17
Filing Date
2024-09-05
Publication Date
2024-11-28
Estimated Expiration
2040-12-03

AI Technical Summary

Technical Problem

In free-viewpoint audio systems, where both the audio objects and the listener can move, encoding position information using polar coordinates centered on the listener leads to inefficient transmission due to the need to update position information for all audio objects as the listener moves, reducing transmission efficiency.

Method used

A signal processing method that combines polar and absolute coordinate systems to encode and transmit position information, where polar coordinates are used for objects less dependent on position and absolute coordinates for objects with fixed relationships to the listener, optimizing transmission by reducing redundant data transmission.

Benefits of technology

This approach improves transmission efficiency by minimizing redundant data transmission and reducing processing overhead, allowing for high-quality audio reproduction with reduced computational and bandwidth requirements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024166232000001_ABST
    Figure 2024166232000001_ABST
Patent Text Reader

Abstract

To improve transmission efficiency and efficiency in a data processing amount.SOLUTION: A signal processing device includes: an acquisition unit that acquires polar coordinate position information indicating a position of a first object expressed by polar coordinates, audio data of the first object, absolute coordinate position information indicating a position of a second object expressed by absolute coordinates, and audio data of the second object; a coordinate conversion unit that converts the absolute coordinate position information into polar coordinate position information indicating a position of the second object; and a rendering processing unit that performs rendering processing on the basis of the polar coordinate position information and the audio data of the first object and the polar coordinate position information and the audio data of the second object. The present technology can be applied to a content reproduction system.SELECTED DRAWING: Figure 5
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present technology relates to a signal processing device, method, and program, and in particular to a signal processing device, method, and program that are capable of improving transmission efficiency. [Background technology]

[0002] The MPEG (Moving Picture Experts Group)-H coding standard, which was established as a standard for 3D Audio for conventional fixed viewpoints, is based on the idea that the position of the listener is the origin and audio objects move within the space around it (see, for example, non-patent document 1).

[0003] Therefore, from a fixed viewpoint, the position information of each audio object as seen by a listener at the origin is described by polar coordinates using the horizontal angle, vertical angle, and distance from the listener to each audio object.

[0004] By utilizing this MPEG-H encoding standard, in fixed viewpoint content, it is possible to localize the sound images of each audio object to the position of that audio object in space, thereby achieving highly realistic audio playback. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] ISO / IEC 23008-3 Information technology - High efficiency coding and media delivery in heterogeneous environments - Part 3: 3D audio Summary of the Invention [Problem to be solved by the invention]

[0006] On the other hand, free viewpoint content is also known in which the listener can be located at any position in space. In the free viewpoint content, not only the audio object moves in the space, but the listener can also move. In other words, the free viewpoint is different from the fixed viewpoint in that the listener can move.

[0007] In such free viewpoint audio, both the audio object and the listener move.

[0008] Therefore, when encoding the position information of each audio object in space, if the position of the audio object is expressed using polar coordinates centered on the listener, which are used for encoding from a fixed viewpoint, the position information may not be transmitted efficiently.

[0009] For example, in the case of a fixed viewpoint, if an audio object is stationary, the relative positional relationship between the listener and the audio object does not change, so that when the audio object moves, position information is encoded and transmitted.

[0010] However, in the case of free viewpoint, even if the audio objects are stationary, if the listener moves, the position information of all the audio objects must be coded and transmitted, resulting in a decrease in transmission efficiency.

[0011] Therefore, it is considered to be advantageous in terms of the transmission efficiency of position information to express the position of each audio object at a free viewpoint using absolute coordinates.

[0012] However, in some cases, for example, sounds that are less dependent on absolute positions in space and surround the listener, such as background noise and reverberation, it may be desirable to reproduce the sounds from the listener's perspective.

[0013] In addition to background noise and reverberation, audio objects such as intentional sound effects intended for the listener may also be used.

[0014] The present technology has been made in consideration of such circumstances, and is intended to make it possible to improve transmission efficiency. [Means for solving the problem]

[0015] A signal processing device according to a first aspect of the present technology includes an acquisition unit that acquires polar coordinate position information indicating a position of a first object expressed in polar coordinates, audio data of the first object, absolute coordinate position information indicating a position of a second object expressed in absolute coordinates, and audio data of the second object, a coordinate conversion unit that converts the absolute coordinate position information into polar coordinate position information indicating the position of the second object, and a rendering processing unit that performs rendering processing based on the polar coordinate position information and the audio data of the first object, and the polar coordinate position information and the audio data of the second object.

[0016] A signal processing method or program according to a first aspect of the present technology includes the steps of acquiring polar coordinate position information indicating a position of a first object expressed in polar coordinates, audio data of the first object, absolute coordinate position information indicating a position of a second object expressed in absolute coordinates, and audio data of the second object, converting the absolute coordinate position information into polar coordinate position information indicating the position of the second object, and performing a rendering process based on the polar coordinate position information and the audio data of the first object and the polar coordinate position information and the audio data of the second object.

[0017] In a first aspect of the present technology, polar coordinate position information indicating a position of a first object expressed in polar coordinates, audio data of the first object, absolute coordinate position information indicating a position of a second object expressed in absolute coordinates, and audio data of the second object are obtained, the absolute coordinate position information is converted into polar coordinate position information indicating the position of the second object, and a rendering process is performed based on the polar coordinate position information and the audio data of the first object and the polar coordinate position information and the audio data of the second object.

[0018] A signal processing device according to a second aspect of the present technology includes a polar coordinate position information encoding unit that encodes polar coordinate position information indicating a position of a first object expressed in polar coordinates, an absolute coordinate position information encoding unit that encodes absolute coordinate position information indicating a position of a second object expressed in absolute coordinates, an audio encoding unit that encodes audio data of the first object and audio data of the second object, and a bitstream generation unit that generates a bitstream including the encoded polar coordinate position information, the encoded absolute coordinate position information, the encoded audio data of the first object, and the encoded audio data of the second object.

[0019] A signal processing method or program according to a second aspect of the present technology includes the steps of encoding polar coordinate position information indicating a position of a first object expressed in polar coordinates, encoding absolute coordinate position information indicating a position of a second object expressed in absolute coordinates, encoding audio data of the first object, and audio data of the second object, and generating a bitstream including the encoded polar coordinate position information, the encoded absolute coordinate position information, the encoded audio data of the first object, and the encoded audio data of the second object.

[0020] In a second aspect of the present technology, polar coordinate position information indicating a position of a first object expressed in polar coordinates is encoded, absolute coordinate position information indicating a position of a second object expressed in absolute coordinates is encoded, audio data of the first object, and audio data of the second object are encoded, and a bitstream including the encoded polar coordinate position information, the encoded absolute coordinate position information, the encoded audio data of the first object, and the encoded audio data of the second object is generated. [Brief description of the drawings]

[0021] [Figure 1] FIG. 2 is a diagram illustrating an object and a coordinate system. [Diagram 2] FIG. 1 is a diagram illustrating an example of a bitstream format. [Diagram 3] FIG. 13 is a diagram illustrating an example of a bit stream configuration. [Figure 4] FIG. 2 illustrates an example of a server configuration. [Diagram 5] FIG. 2 illustrates an example of the configuration of a client. [Figure 6] 11 is a flowchart illustrating a transmission process and a reception process. [Figure 7] FIG. 2 illustrates an example of a server configuration. [Figure 8] 11 is a flowchart illustrating a transmission process and a reception process. [Figure 9] FIG. 2 illustrates an example of a server configuration. [Figure 10] 11 is a flowchart illustrating a transmission process and a reception process. [Figure 11] FIG. 2 illustrates an example of the configuration of a client. [Figure 12] 11 is a flowchart illustrating a transmission process and a reception process. [Figure 13] FIG. 2 illustrates an example of a server configuration. [Figure 14] FIG. 2 illustrates an example of the configuration of a client. [Figure 15]11 is a flowchart illustrating a transmission process and a reception process. [Figure 16] FIG. 2 illustrates an example of a server configuration. [Figure 17] 11 is a flowchart illustrating a transmission process and a reception process. [Figure 18] FIG. 1 illustrates an example of the configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Hereinafter, embodiments to which the present technology is applied will be described with reference to the drawings.

[0023] First embodiment About this technology This technology makes it possible to improve transmission efficiency when encoding and transmitting position information of an audio object (hereinafter also simply referred to as an object) by combining polar coordinate position information expressed in polar coordinates with absolute coordinate position information expressed in absolute coordinates.

[0024] In this technology, on the server side, audio data for playing the sound of one or more objects and polar coordinate position information or absolute coordinate position information indicating the position of each object are encoded and transmitted to the client.

[0025] Furthermore, the client reproduces free viewpoint audio content consisting of the sound of each object based on the audio data of each object received from the server and the polar coordinate position information and absolute coordinate position information of each object.

[0026] For example, when absolute coordinate position information expressing the position of an object in space in absolute coordinates is encoded and transmitted to a client, the server obtains listener position information expressing the position of the listener in space in absolute coordinates from the client and generates absolute coordinate position information.

[0027] At this time, the server may generate absolute coordinate position information that indicates the position of the object with an accuracy according to the positional relationship between the listener and the object, for example, the distance from the listener to the object.

[0028] Specifically, for example, the shorter the distance from the listener to the object, the higher the precision of the absolute coordinate position information, that is, the more accurate the generated absolute coordinate position information indicating the position.

[0029] This is because the quantization precision (quantization step width) used during encoding causes a shift in the position of an object, and the greater the distance from the listener to the object, the greater the amount of positional shift (tolerable error) that does not cause the listener to perceive a shift in the position of the sound image.

[0030] Therefore, if absolute coordinate position information with an appropriate accuracy according to the positional relationship between the listener and the object is generated and transmitted, the amount of information (number of bits) of the absolute coordinate position information can be reduced without making the listener perceive a shift in the sound image position.

[0031] Note that, although absolute coordinate position information of the required accuracy may be generated each time absolute coordinate position information is transmitted, encoded absolute coordinate position information of the highest accuracy may be prepared in advance, and absolute coordinate position information of the required accuracy may be generated from that absolute coordinate position information.

[0032] Specifically, for example, the absolute coordinate position information with the highest precision is prepared in advance by quantizing the absolute coordinates indicating the position of an object in space with a predetermined quantization precision. This absolute coordinate position information with the highest precision is encoded absolute coordinate position information.

[0033] The server obtains absolute coordinate position information obtained by quantizing the absolute coordinates of the object with any quantization accuracy by extracting a part of the highest accuracy absolute coordinate position information according to the listener's conditions specified by the client, such as the listener's position information, i.e., encoded absolute coordinate position information indicating the object's position with any accuracy can be obtained.

[0034] On the other hand, when polar coordinate position information expressing the position of an object in space using polar coordinates is encoded and transmitted to a client, the server generates polar coordinate position information based on position information such as absolute coordinates indicating the position of the object in a previously prepared space and listener position information.

[0035] For example, as shown in Figure 1, there are mainly two types of objects in three-dimensional space.

[0036] That is, for example, in the example indicated by the arrow Q11 in FIG. 1, an object OB11 and an object OB12 exist around a listener U11 in a three-dimensional space.

[0037] Here, the object OB11 is an audio object that is highly dependent on the placement position in the space, such as a musical instrument. In other words, the object OB11 is an object that should be located at an absolute position in the space during audio playback. Direct sound objects such as musical instruments are also called dry objects.

[0038] In the following, an object such as object OB11 that is highly dependent on its placement position in space will also be referred to as an absolute coordinate object.

[0039] In contrast, the object OB12 is an audio object that has low spatial dependency, that is, low dependency on the placement position, such as a large object in the background or a fixed object corresponding to background noise or reverberation components.

[0040] In other words, for example, the object OB12 is an object from which sound always reaches the listener U11 from a fixed direction relative to the listener U11 during audio reproduction, regardless of the position or movement of the listener U11 in space.

[0041] In the following, an object such as the object OB12 that is less dependent on its placement position in space will also be referred to as a polar coordinate object.

[0042] At a free viewpoint, as shown by arrow Q12, for an object such as object OB11, the degree of dependence on the placement position in space is high, so it is considered advantageous in terms of transmission efficiency to transmit absolute coordinate position information.

[0043] This is because, for example, when transmitting absolute coordinate position information of object OB11, once the absolute coordinate position information is transmitted, there is no need to transmit the absolute coordinate position information as long as object OB11 remains stationary even if the position of listener U11 changes.

[0044] On the other hand, background sound objects surrounding the listener U11, such as the object OB12, are less dependent on the position in space, and it is preferable to consider them as being positioned with the listener U11 at the center.

[0045] As described above, when transmitting absolute coordinate position information of an object with a precision according to the distance from the listener, mapping to an absolute coordinate position corresponding to an arbitrary position of the listener in order to maintain a positional relationship with the listener at the center must be performed in real time, which causes inconvenience in terms of control and calculation processing. That is, control and calculation processing such as determining quantization precision based on the distance from the listener are required.

[0046] Furthermore, when the size of a space is large, it may be necessary to place more objects with low position dependency, such as background noise, that are large enough to encompass that area, which may increase the number of objects to be transmitted, thereby increasing the amount of transmitted information.

[0047] Therefore, in this technology, for objects such as object OB12 whose position is less dependent on the object's placement, the position is not represented by absolute coordinates, but rather polar coordinate position information is transmitted that represents the position in a polar coordinate system centered on listener U11, as shown by arrow Q13.

[0048] In this case, polar coordinate position information is generated that is made up of an azimuth angle and an elevation angle that indicate the horizontal and vertical positions of the object OB12 as seen by the listener U11, and a radius that indicates the distance from the listener U11 to the object OB12.

[0049] If polar coordinate position information is transmitted as position information of an object that is less dependent on the placement position, mapping to an absolute coordinate position becomes unnecessary, and the amount of data processing (arithmetic processing) can be reduced (processing efficiency can be improved). Furthermore, for some objects, the polar coordinate position information does not change even if the position of the listener U11 changes, so the number of transmissions of the polar coordinate position information can be reduced and transmission efficiency can be improved.

[0050] In this way, by combining absolute coordinate position information and polar coordinate position information according to the nature (role) of the object, it is possible to transmit position information efficiently.

[0051] Polar coordinate objects can also be used for sound effects centered on the listener, similar to the above-mentioned background noise and reverberation. Even in such cases, expressing the object's position in polar coordinates makes it possible to transmit position information efficiently.

[0052] Also, for polar coordinate objects, gain information may be encoded and transmitted to the client along with the polar coordinate position information.

[0053] In such a case, the polar coordinate object can be classified into the following categories C1 to C3, and by performing such categorization, the amount of information can be controlled efficiently. Here, the angles indicating the position are the azimuth angle and the elevation angle.

[0054] Category C1: Both the angle and gain information indicating the position are fixed Category C2: The angle indicating the position is fixed, but the gain information is variable Category C3: Both the angle and gain information indicating the position are variable

[0055] For example, a polar coordinate object such as background noise is classified as category C1, a polar coordinate object such as reverberation whose gain changes in conjunction with the listener's position is classified as category C2, and a polar coordinate object such as sound effects is classified as category C3.

[0056] For example, for polar coordinate objects in categories C1 and C2, predetermined fixed coordinate values ​​(fixed values) are used as the polar coordinate position information, so once the polar coordinate position information is transmitted to the client, there is no need to transmit the polar coordinate position information thereafter.

[0057] Therefore, not only can the number of times polar coordinate position information is transmitted be reduced and transmission efficiency be improved, but also the amount of code in the bit stream can be reduced.

[0058] In particular, for polar coordinate objects in category C1, not only the polar coordinate position information but also the gain information is set to a fixed value, so that the transmission efficiency can be improved and the amount of code can be reduced by the amount of gain information.

[0059] Also, for example, for a polar coordinate object of category C2, the server may calculate a gain amount according to listener position information acquired from a client, and encode gain information indicating the gain amount and transmit it to the client.

[0060] FIG. 2 shows an example of a bit stream format for transmitting the above-described object position information.

[0061] In FIG. 2, "NumOfObjects" indicates the total number of absolute coordinate objects and polar coordinate objects, that is, the total number of objects.

[0062] Furthermore, "PosCodingMode[i]" indicates the position coding mode of the i-th object, i.e., the type of object. Depending on the value of the position coding mode, the position information and gain information of the object are stored in the bitstream.

[0063] Here, the value "0" of the position encoding mode indicates an absolute coordinate object, and the value "1" of the position encoding mode indicates a polar coordinate object of category C1, for which fixed polar coordinate position information and gain information prepared in advance are transmitted.

[0064] Furthermore, the value "2" of the position encoding mode indicates that the object is a polar coordinate object of category C2, and for this polar coordinate object, pre-prepared fixed polar coordinate position information and variable gain information are transmitted.

[0065] A position encoding mode value of "3" indicates a polar object of category C3, for which variable polar position and gain information is transmitted.

[0066] In this example, the polar coordinate position information and the absolute coordinate position information are stored in different areas and transmitted. In particular, the absolute coordinate position information is stored in an extension area of ​​the bit stream as shown in FIG.

[0067] That is, in this example, for an object whose position encoding mode value is 0, the quantization bit rate “ChildCubeDivIndex[i]”, the x-coordinate value “QposX[i]” constituting the absolute coordinate position information, the y-coordinate value “QposY[i]” constituting the absolute coordinate position information, and the z-coordinate value “QposZ[i]” constituting the absolute coordinate position information are encoded and stored in the extended area, etc.

[0068] The transmission of the polar coordinate position information and the absolute coordinate position information is not limited to the example described with reference to FIG. 2, and may be performed in any manner.

[0069] For example, the polar coordinate position information may be encoded using an existing encoding method such as MPEG-H. In such a case, for example, as shown in Fig. 3, audio data of an object is encoded for both the polar coordinate object and the absolute coordinate object.

[0070] Then, the encoded audio data obtained by encoding the audio data of the polar coordinate object is stored in a CPE (Channel Pair Element) or SCE (Single Channel Element) of the bit stream as data with position information.

[0071] Furthermore, the polar coordinate position information of the polar coordinate object is encoded and stored in the metadata area of ​​the bit stream or the like.

[0072] On the other hand, the encoded audio data obtained by encoding the audio data of the absolute coordinate object is stored in the CPE or SCE of the bitstream as data without position information.

[0073] Furthermore, the absolute coordinate position information of the absolute coordinate object may be stored in the format shown in FIG. 2 in, for example, “mpegh3daExtElement()”, which is an extension area of ​​the MPEG-H encoding standard, or may be transmitted in a format other than MPEG-H.

[0074] <Server configuration example> Next, a content playback system to which the present technology is applied will be described.

[0075] For example, a content reproduction system is made up of the above-mentioned server and client, and in the content reproduction system, objects that are absolute coordinate objects and objects that are polar coordinate objects are previously determined.

[0076] The server constituting the content reproduction system is configured, for example, as shown in FIG.

[0077] The server 11 shown in FIG. 4 includes a listener position information receiving section 21, an absolute coordinate position information encoding section 22, a polar coordinate position information encoding section 23, an audio encoding section 24, a bit stream generating section 25, and a transmitting section .

[0078] The listener position information receiving unit 21 receives listener position information indicating the position of the listener (user) in space, transmitted from a client via a communication network, and supplies the information to an absolute coordinate position information encoding unit 22 and a polar coordinate position information encoding unit 23. Here, the listener position information is, for example, absolute coordinates indicating the absolute position of the listener in space.

[0079] The absolute coordinate position information encoding unit 22 generates and encodes absolute coordinate position information indicating the absolute position of the absolute coordinate object in space based on the listener position information supplied from the listener position information receiving unit 21, and supplies the encoded information to a bit stream generation unit 25.

[0080] For example, the absolute coordinate position information encoding unit 22 quantizes position information indicating the absolute position of the absolute coordinate object with a quantization accuracy (quantization step width) determined by the distance from the listener to the absolute coordinate object, thereby generating encoded absolute coordinate position information with an accuracy according to the positional relationship with the listener.

[0081] In addition, there are also cases where encoded absolute coordinate position information with the highest precision, obtained by quantizing the absolute coordinates of an absolute coordinate object with a predetermined quantization precision, is prepared in advance.

[0082] In such a case, the absolute coordinate position information encoding unit 22 acquires the highest-precision absolute coordinate position information of the absolute coordinate object, and extracts information of a bit length determined for the distance from the listener to the absolute coordinate object from the highest-precision absolute coordinate position information, thereby obtaining encoded absolute coordinate position information that indicates the position of the absolute coordinate object with an accuracy determined for the distance from the listener.

[0083] Alternatively, the absolute coordinate position information encoding unit 22 may acquire or generate gain information of the absolute coordinate object, encode the gain information, and supply it to the bit stream generating unit 25 .

[0084] The polar coordinate position information encoding unit 23 generates polar coordinate position information indicating the relative position of the polar coordinate object as seen by the listener, as necessary, and encodes the polar coordinate position information.

[0085] For example, for the polar coordinate objects of the above-mentioned categories C1 and C2, polar coordinate position information is prepared in advance, so the polar coordinate position information encoding unit 23 acquires and encodes the prepared polar coordinate position information.

[0086] Furthermore, for example, for a polar coordinate object in category C3, position information indicating the absolute position of the polar coordinate object in space is prepared in advance.

[0087] Then, the polar coordinate position information encoding unit 23 acquires position information indicating the absolute position of the polar coordinate object, and generates and encodes polar coordinate position information based on the position information and the listener position information supplied from the listener position information receiving unit 21.

[0088] Furthermore, the polar coordinate position information encoding unit 23 generates gain information of a polar coordinate object or obtains gain information of a previously prepared polar coordinate object as appropriate based on the category of the polar coordinate object and the listener position information, and encodes the gain information.

[0089] The polar coordinate position information encoding unit 23 supplies the encoded polar coordinate position information and gain information to a bit stream generating unit 25 .

[0090] In the following, the encoded absolute coordinate position information will also be referred to as encoded absolute coordinate position information, and the encoded polar coordinate position information will also be referred to as encoded polar coordinate position information.

[0091] The audio encoding unit 24 acquires audio data of absolute coordinate objects, audio data of polar coordinate objects, and channel-based audio data, encodes the acquired audio data, and supplies the resulting encoded audio data to a bitstream generation unit 25.

[0092] Here, the channel-based audio data refers to the audio data for each channel of a multi-channel configuration.

[0093] For example, channel-based audio data may be audio data such as fixed background noise or background sound that does not change how it sounds regardless of the position of the listener. Also, audio data for reproducing sound effects that have a wide range of effects and are difficult to represent with one or more objects, such as a roar that spreads throughout the entire space, may be channel-based audio data.

[0094] On the other hand, audio data for absolute coordinate objects and polar coordinate objects is object-based audio data for playing back the sound of the object.

[0095] In the following, a case will be described in which the free viewpoint content played back on the client side is made up of sounds based on channel-based audio data, sounds of each absolute coordinate object, and sounds of each polar coordinate object.

[0096] However, if the sound of each absolute coordinate object and the sound of each polar coordinate object are reproduced as the sound of the content, the channel-based audio data is not necessarily required.

[0097] As an example, when there is audio data of a polar coordinate object as audio data such as background noise, it is possible to consider not having channel-based audio data as content data.

[0098] Conversely, when channel-based audio data is present as audio data such as background noise, it may be possible to eliminate objects such as background noise.

[0099] The bit stream generation unit 25 multiplexes the encoded absolute coordinate position information from the absolute coordinate position information encoding unit 22, the encoded polar coordinate position information and gain information from the polar coordinate position information encoding unit 23, and the encoded audio data from the audio encoding unit 24. The bit stream generation unit 25 supplies the bit stream generated by the multiplexing to the transmission unit 26.

[0100] The transmitting unit 26 transmits the bit stream supplied from the bit stream generating unit 25 to the client via a communication network.

[0101] <Client configuration example> Moreover, a client that receives a bit stream from the server 11 is configured, for example, as shown in FIG.

[0102] The client 51 shown in FIG. 5 has a listener position information input unit 61, a listener position information transmission unit 62, a reception separation unit 63, an object separation unit 64, a polar coordinate position information decoding unit 65, an absolute coordinate position information decoding unit 66, a coordinate conversion unit 67, an audio decoding unit 68, a renderer 69, a format conversion unit 70, and a mixer 71.

[0103] The listener position information input unit 61 is composed of, for example, a sensor worn by the listener, a mouse, a keyboard, a touch panel, etc., and supplies the listener position information input (specified) by the listener's movements, operations, etc. to the listener position information transmission unit 62 and the coordinate conversion unit 67.

[0104] The listener position information transmitting section 62 transmits the listener position information supplied from the listener position information input section 61 to the server 11 via the communication network.

[0105] The receiving and separating unit 63 receives the bit stream transmitted from the server 11, and separates the encoded absolute coordinate position information, the encoded polar coordinate position information, the gain information, and the encoded audio data from the bit stream.

[0106] In other words, the receiving and separating unit 63 functions as an acquiring unit that acquires encoded absolute coordinate position information, encoded polar coordinate position information, gain information, and encoded audio data by receiving a bit stream based on the listener position information. In particular, the receiving and separating unit 63 acquires encoded absolute coordinate position information with an accuracy according to the positional relationship between the listener and the absolute coordinate object based on the listener position information.

[0107] The receiving / separating unit 63 supplies the encoded absolute coordinate position information, encoded polar coordinate position information, and gain information separated (extracted) from the bit stream to an object separating unit 64, and also supplies the encoded audio data to an audio decoding unit 68.

[0108] The object separator 64 separates the encoded absolute coordinate position information, the encoded polar coordinate position information, and the gain information supplied from the receiving separator 63 .

[0109] That is, the object separator 64 supplies the encoded polar coordinate position information and gain information to a polar coordinate position information decoder 65 , and also supplies the encoded absolute coordinate position information to an absolute coordinate position information decoder 66 .

[0110] The polar coordinate position information decoding unit 65 decodes the encoded polar coordinate position information and gain information supplied from the object separation unit 64 and supplies the decoded information to a renderer 69 .

[0111] The absolute coordinate position information decoding unit 66 decodes the encoded absolute coordinate position information supplied from the object separating unit 64 and supplies it to a coordinate conversion unit 67 .

[0112] The coordinate conversion unit 67 converts the absolute coordinate position information supplied from the absolute coordinate position information decoding unit 66 into polar coordinate position information based on the listener position information supplied from the listener position information input unit 61 , and supplies it to a renderer 69 .

[0113] The coordinate conversion unit 67 converts the absolute coordinate position information of the absolute coordinate object into polar coordinate position information, which is a polar coordinate indicating the relative position of the absolute coordinate object as viewed from the listener position indicated by the listener position information, by coordinate conversion.

[0114] In addition, the coordinate conversion may utilize not only the listener position information but also directional information indicating the direction of the listener's face obtained by the listener position information input unit 61. In such a case, polar coordinate position information is generated that indicates the relative position of the absolute coordinate object based on the listener's front direction.

[0115] The audio decoding unit 68 decodes the encoded audio data supplied from the receiving and separating unit 63, and supplies the resulting audio data for each object to a renderer 69, while also supplying the channel-based audio data to a format conversion unit 70.

[0116] Therefore, the renderer 69 is supplied with audio data for each absolute coordinate object and audio data for each polar coordinate object.

[0117] The renderer 69 performs rendering processing based on the polar coordinate position information and gain information supplied from the polar coordinate position information decoding unit 65, the polar coordinate position information supplied from the coordinate conversion unit 67, and the audio data of each object supplied from the audio decoding unit 68.

[0118] The renderer 69 performs rendering processing in the polar coordinate system defined by, for example, MPEG-H.

[0119] More specifically, for example, the renderer 69 performs a rendering process such as VBAP (Vector Based Amplitude Panning) to generate audio data for playing back the sound of the object.

[0120] This audio data is multi-channel audio data corresponding to the speaker configuration of the speaker system that is the final output destination, i.e., the audio data obtained by the rendering process is made up of audio data for channels corresponding to the multiple speakers that make up the speaker system.

[0121] By playing back sound based on such audio data, it is possible to localize the sound image of the object at the position in space indicated by the polar coordinate position information.

[0122] In the renderer 69, the audio data of the polar coordinate object is gain-corrected based on the gain information of the polar coordinate object, and the gain-corrected audio data is used to perform the rendering process.

[0123] The renderer 69 supplies the audio data obtained by the rendering process to a mixer 71 .

[0124] The format conversion unit 70 performs format conversion to convert the channel-based audio data supplied from the audio decoding unit 68 into audio data with a channel configuration corresponding to the speaker configuration of the speaker system for reproducing the sound of the content.

[0125] The format conversion unit 70 supplies the channel-based audio data obtained by the format conversion to a mixer 71 .

[0126] The mixer 71 performs a mixing process based on the audio data supplied from the renderer 69 and the channel-based audio data supplied from the format conversion unit 70, and outputs the resulting multi-channel audio data to the subsequent stage.

[0127] For example, in the mixing process, the multi-channel audio data and the channel-based audio data supplied from the renderer 69 are added (mixed) together to produce the final audio data for that channel.

[0128] <Description of Transmission Process and Reception Process> Next, a description will be given of the operation of the content reproduction system made up of the server 11 and the client 51. That is, hereinafter, a transmission process by the server 11 and a reception process by the client 51 will be described with reference to the flowchart of FIG.

[0129] When an instruction to start playing the content is given in the client 51, the client 51 starts a receiving process. When the receiving process starts, the listener position information input unit 61 supplies the listener position information input (specified) by an operation by the listener to the listener position information transmission unit 62 and the coordinate conversion unit 67.

[0130] Then, in step S 11 , the listener position information transmitting section 62 transmits the listener position information supplied from the listener position information input section 61 to the server 11 .

[0131] The listener position information may be transmitted periodically, such as for each frame, or may be transmitted only when the listener's position changes.

[0132] When the listener position information is transmitted in this manner, the server 11 performs a transmission process.

[0133] That is, in step S 41 , the listener position information receiving section 21 receives the listener position information transmitted from the client 51 , and supplies it to the absolute coordinate position information encoding section 22 and the polar coordinate position information encoding section 23 .

[0134] In step S42, the absolute coordinate position information encoding unit 22 generates absolute coordinate position information of the absolute coordinate object based on the listener position information supplied from the listener position information receiving unit 21. In addition, in step S43, the absolute coordinate position information encoding unit 22 encodes the absolute coordinate position information based on the listener position information, and supplies the obtained encoded absolute coordinate position information to the bit stream generating unit 25.

[0135] For example, the absolute coordinate position information encoding unit 22 acquires position information indicating the absolute position of an absolute coordinate object, and quantizes it with a quantization accuracy determined by the listener position information, thereby generating encoded absolute coordinate position information with an accuracy according to the positional relationship with the listener.

[0136] Furthermore, for example, when absolute coordinate position information encoded with the highest precision is prepared in advance, the absolute coordinate position information encoding unit 22 acquires the absolute coordinate position information with the highest precision.

[0137] The absolute coordinate position information encoding unit 22 then extracts information of a bit length determined for the distance from the listener to the absolute coordinate object from the acquired absolute coordinate position information of the highest precision, thereby generating encoded absolute coordinate position information of a predetermined quantization precision.

[0138] In this case, taking into consideration the allowable quantization error depending on the human perception angle and the distance to the object, for example, the farther an absolute coordinate object is from the listener, the lower the quantization accuracy of the encoded absolute coordinate position information generated, thereby making it possible to improve the transmission efficiency of the encoded absolute coordinate position information without impairing the sense of positioning of the sound image.

[0139] In step S44, the polar coordinate position information encoding unit 23 generates polar coordinate position information of a required polar coordinate object according to the listener position information supplied from the listener position information receiving unit 21. That is, the polar coordinate position information encoding unit 23 acquires position information of a polar coordinate object, and generates polar coordinate position information of the polar coordinate object based on the acquired position information and listener position information.

[0140] Here, since the polar coordinate position information of categories C1 and C2 is already obtained, only the polar coordinate position information of category C3 is generated.

[0141] Furthermore, the polar coordinate position information encoding unit 23 acquires gain information of the polar coordinate object in category C1, and generates gain information for the polar coordinate objects in categories C2 and C3 based on the position information of the polar coordinate object and the listener position information.

[0142] In step S 45 , the polar coordinate position information encoding unit 23 encodes the polar coordinate position information and gain information of each polar coordinate object, and supplies the encoded information to the bit stream generation unit 25 .

[0143] In step S46, the audio encoding unit 24 obtains the audio data of the absolute coordinate object, the audio data of the polar coordinate object, and the channel-based audio data, and encodes the audio data.

[0144] The audio encoding unit 24 supplies the encoded audio data obtained by the encoding to a bit stream generating unit 25 .

[0145] In step S47, the bit stream generation unit 25 multiplexes the encoded absolute coordinate position information from the absolute coordinate position information encoding unit 22, the encoded polar coordinate position information and gain information from the polar coordinate position information encoding unit 23, and the encoded audio data from the audio encoding unit 24 to generate a bit stream. The bit stream generation unit 25 supplies the bit stream generated by the multiplexing to the transmission unit 26.

[0146] Note that, if the same encoded absolute coordinate position information has already been transmitted, for example, when the position of the absolute coordinate object and the distance from the listener to the absolute coordinate object have not changed, the encoded absolute coordinate position information is not stored in the bitstream by transmitting the absolute coordinate object with the number of quantization bits set to 0. In other words, the absolute coordinate position information is neither encoded nor transmitted to the client 51.

[0147] Similarly, the encoded polar coordinate position information is encoded and transmitted to the client 51 only if the polar coordinate position information changes.

[0148] In this way, the transmission efficiency of the encoded absolute coordinate position information and the encoded polar coordinate position information can be improved.

[0149] In step S48, the transmitting unit 26 transmits the bit stream supplied from the bit stream generating unit 25 to the client 51, and the transmission process ends.

[0150] When the bit stream is transmitted, the client 51 performs the process of step S12.

[0151] That is, in step S 12 , the receiving and separating unit 63 receives the bit stream transmitted from the server 11 .

[0152] In step S13, the receiving / separating unit 63 separates the received bit stream into encoded absolute coordinate position information, encoded polar coordinate position information, gain information, and encoded audio data.

[0153] The receiving / separating unit 63 supplies the separated encoded absolute coordinate position information, encoded polar coordinate position information, and gain information to an object separating unit 64 , and also supplies the encoded audio data to an audio decoding unit 68 .

[0154] In addition, the object separator 64 supplies the encoded polar coordinate position information and gain information supplied from the reception separator 63 to a polar coordinate position information decoder 65, and supplies the encoded absolute coordinate position information to an absolute coordinate position information decoder 66.

[0155] In step S 14 , the polar coordinate position information decoding unit 65 decodes the encoded polar coordinate position information and gain information supplied from the object separating unit 64 , and supplies the decoded information to the renderer 69 .

[0156] In this embodiment, the server 11 calculates the gain information of the polar coordinate objects in the categories C2 and C3.

[0157] However, the polar coordinate position information decoding unit 65 may calculate gain information of the polar coordinate objects of categories C2 and C3 based on the listener position information and the polar coordinate position information. In this case, the category (type) of each polar coordinate object can be identified from the position encoding mode included in the bit stream.

[0158] In step S15 , the absolute coordinate position information decoding unit 66 decodes the encoded absolute coordinate position information supplied from the object separating unit 64 , and supplies the decoded information to the coordinate transforming unit 67 .

[0159] In step S16, the coordinate conversion unit 67 performs coordinate conversion on the absolute coordinate position information supplied from the absolute coordinate position information decoding unit 66, based on the listener position information supplied from the listener position information input unit 61. As a result, for each absolute coordinate object, polar coordinate position information is obtained that indicates the relative position of the absolute coordinate object as seen by the listener.

[0160] In addition, in the coordinate transformation, information indicating the direction of the listener's face (Yaw), the up-and-down movement of the face (Pitch), and the rotation of the face (Roll) may also be used.

[0161] The coordinate conversion unit 67 supplies the polar coordinate position information of each absolute coordinate object obtained by the coordinate conversion to a renderer 69 .

[0162] In step S 17 , the audio decoding unit 68 decodes the encoded audio data supplied from the reception separation unit 63 .

[0163] The audio decoding unit 68 supplies the audio data of each absolute coordinate object and the audio data of each polar coordinate object obtained by decoding to a renderer 69, and also supplies the channel-based audio data obtained by decoding to a format conversion unit 70.

[0164] Furthermore, the format conversion unit 70 performs format conversion on the channel-based audio data supplied from the audio decoding unit 68 , and supplies the resulting audio data to a mixer 71 .

[0165] In step S18, the renderer 69 performs rendering processing such as VBAP based on the polar coordinate position information supplied from the polar coordinate position information decoding unit 65, the polar coordinate position information supplied from the coordinate conversion unit 67, and the audio data supplied from the audio decoding unit 68.

[0166] At this time, the renderer 69 performs gain correction on the audio data of the polar coordinate object based on the gain information supplied from the polar coordinate position information decoding unit 65, and performs rendering processing using the gain-corrected audio data. The renderer 69 supplies the audio data obtained by the rendering processing to the mixer 71.

[0167] In step S19, the mixer 71 performs a mixing process based on the audio data supplied from the renderer 69 and the channel-based audio data supplied from the format conversion unit .

[0168] Then, the mixer 71 outputs the multi-channel audio data obtained by the mixing process to the subsequent stage, and the receiving process ends.

[0169] If the bit stream does not contain channel-based audio data, the mixing process is not performed, and the audio data obtained by the renderer 69 is output to the subsequent stage, and the receiving process ends.

[0170] In the content reproduction system, the above-described process is performed for each frame of the audio data of the content.

[0171] In this manner, the server 11 encodes the absolute coordinate position information or the polar coordinate position information depending on whether the object is an absolute coordinate object or a polar coordinate object, stores it in a bitstream together with the encoded audio data, and transmits it.

[0172] Furthermore, the client 51 extracts and decodes the encoded absolute coordinate position information and encoded polar coordinate position information from the bit stream, and performs rendering processing.

[0173] In this way, by generating absolute coordinate position information or polar coordinate position information that indicates the position of an object in a coordinate system that corresponds to the properties (characteristics) of the object and transmitting it to client 51, the amount of information and transmission frequency of the object's position information can be reduced, and transmission efficiency can be improved.

[0174] Second embodiment <Server configuration example> For example, a polar coordinate object in category C1 such as background noise may be transmitted to the client 51 as channel-based audio data rather than as object audio data.

[0175] In such a case, the content playback system is composed of, for example, the server 11 shown in Fig. 7 and the client 51 shown in Fig. 5. In Fig. 7, the parts corresponding to those in Fig. 4 are given the same reference numerals, and the description thereof will be omitted as appropriate.

[0176] The server 11 shown in FIG. 7 has a listener position information receiving unit 21, an absolute coordinate position information encoding unit 22, a polar coordinate position information encoding unit 23, a pre-rendering processing unit 101, an audio encoding unit 24, a bit stream generating unit 25, and a transmitting unit 26.

[0177] The configuration of the server 11 in FIG. 7 differs from that of the server 11 in FIG. 4 in that a pre-rendering processing unit 101 is newly provided, but in other respects it has the same configuration as the server 11 in FIG.

[0178] However, in the server 11 of FIG. 7, the listener position information receiving unit 21 acquires not only the listener position information from the client 51 but also directional information indicating the direction of the listener's face, and supplies this to the pre-rendering processing unit 101.

[0179] In this example, it is assumed that for the polar coordinate objects in category C1, position information indicating the absolute positions of the polar coordinate objects in space is prepared in advance.

[0180] The pre-rendering processor 101 obtains audio data and position information indicating the absolute position of the polar coordinate object of category C1.

[0181] Furthermore, the pre-rendering processing unit 101 performs pre-rendering based on the acquired position information and audio data and the listener position information and direction information supplied from the listener position information receiving unit 21, and supplies the resulting channel-based audio data to the audio encoding unit 24.

[0182] For example, in pre-rendering, first, polar coordinate position information indicating the relative position of the polar coordinate object with respect to the front direction of the listener is generated based on the position information of the polar coordinate object and the listener position and direction information.

[0183] Then, VBAP or the like is performed based on the polar coordinate position information of the polar coordinate object and the audio data, and channel-based audio data is generated. This channel-based audio data is audio data with a multi-channel configuration in which the sound image of the polar coordinate object is localized at the position in space indicated by the polar coordinate position information.

[0184] In addition, if there is other channel-based audio data that is prepared in advance and that constitutes the content, apart from the channel-based audio data generated by pre-rendering, the other channel-based audio data are added together to form the final channel-based audio data.

[0185] Object-based audio data has the advantage that it is possible to control the sound image localization and gain for any object.

[0186] In contrast, channel-based audio data has the advantage that it is not necessary to encode object position information and transmit it to the decoding side.

[0187] 7, it is not necessary to transmit the encoded polar coordinate position information of the polar coordinate object of category C1 to the client 51, and the amount of code in the bit stream can be reduced. Furthermore, since the rendering process of the polar coordinate object of category C1 is not required on the client 51 side, the amount of processing on the client 51 can be reduced accordingly.

[0188] <Description of Transmission Process and Reception Process> Next, the operation of the content reproduction system including the server 11 shown in FIG. 7 and the client 51 shown in FIG. 5 will be described.

[0189] That is, the transmission process by the server 11 and the reception process by the client 51 will be described below with reference to the flowchart in FIG.

[0190] When the receiving process is started in the client 51 , the listener position information input unit 61 acquires the listener position information and direction information, and supplies them to the listener position information transmission unit 62 and the coordinate conversion unit 67 .

[0191] Then, in step S 81 , the listener position information transmitting section 62 transmits the listener position information and direction information supplied from the listener position information input section 61 to the server 11 .

[0192] When the listener position information and direction information are transmitted in this manner, the server 11 performs a transmission process.

[0193] That is, in step S 111 , the listener position information receiving section 21 receives the listener position information and direction information transmitted from the client 51 .

[0194] Furthermore, the listener position information receiving unit 21 supplies the listener position information to the absolute coordinate position information encoding unit 22 and the polar coordinate position information encoding unit 23 , and also supplies the listener position information and direction information to the pre-rendering processing unit 101 .

[0195] After the process of step S111 is performed, the processes of steps S112 to S115 are performed. However, since these processes are similar to the processes of steps S42 to S45 in FIG. 6, the description thereof will be omitted.

[0196] However, in step S115, only the polar coordinate position information and gain information of the polar coordinate objects in categories C2 and C3 are encoded.

[0197] In step S 116 , the pre-rendering processing unit 101 performs pre-rendering based on the listener position information and direction information supplied from the listener position information receiving unit 21 , and supplies the obtained channel-based audio data to the audio encoding unit 24 .

[0198] That is, for example, the pre-rendering processor 101 acquires audio data and position information indicating the absolute position of a polar coordinate object of category C1.

[0199] The pre-rendering processor 101 then performs processing such as VBAP as pre-rendering based on the acquired position information and audio data, and the listener position information and direction information, to generate channel-based audio data.

[0200] After the pre-rendering is performed, the processes of steps S117 to S119 are performed and the transmission process ends. However, since these processes are similar to the processes of steps S46 to S48 in FIG. 6, the description thereof will be omitted.

[0201] However, in step S117, the audio encoding unit 24 encodes the audio data of the absolute coordinate objects, the audio data of the polar coordinate objects of categories C2 and C3, and the channel-based audio data supplied from the pre-rendering processing unit 101.

[0202] When the process of step S119 is performed and the bit stream is transmitted to the client 51, the client 51 performs the processes of steps S82 to S89, and the receiving process ends.

[0203] The processing from step S82 to step S89 is similar to the processing from step S12 to step S19 in Fig. 6, and therefore the description thereof will be omitted. However, in step S86, coordinate conversion is performed using not only the listener position information but also the face direction information (Yaw, Pitch, Roll).

[0204] In this manner, the server 11 performs pre-rendering for polar coordinate objects of a specific category, and transmits the channel-based audio data obtained as a result to the client 51. In this manner, it is possible to improve transmission efficiency.

[0205] Third embodiment <Server configuration example> Incidentally, background noise and reverberation change depending on the virtual space, such as a live concert venue, in which the sound of the content is reproduced.

[0206] Therefore, for example, a plurality of object groups may be prepared in advance for polar coordinate objects, which are objects such as background noise and reverberation, and the listener may be allowed to select a preferred object group from among these object groups.

[0207] In this case, an object group is prepared for each type of virtual space in which the content is played back. Furthermore, one object group is made up of one or more polar coordinate objects that make up the content, and polar coordinate position information, gain information, and audio data are prepared for each of the polar coordinate objects.

[0208] In this way, when a plurality of object groups are prepared in advance, the content playback system is made up of, for example, the server 11 shown in Fig. 9 and the client 51 shown in Fig. 5. Note that in Fig. 9, parts corresponding to those in Fig. 4 are given the same reference numerals, and their explanation will be omitted as appropriate.

[0209] The server 11 shown in FIG. 9 includes a listener position information receiving unit 21, an absolute coordinate position information encoding unit 22, a selection unit 131, a polar coordinate position information encoding unit 23, an audio encoding unit 24, a bit stream generation unit 25, and a transmission unit .

[0210] The configuration of the server 11 in FIG. 9 differs from that of the server 11 in FIG. 4 in that a selection unit 131 is newly provided, but in other respects it has the same configuration as the server 11 in FIG.

[0211] However, in the server 11 of FIG. 9, the listener position information receiving unit 21 obtains not only the listener position information from the client 51 but also group selection information indicating the object group selected by the listener, and supplies it to the selecting unit 131.

[0212] In this example, polar coordinate position information, gain information, and audio data of polar coordinate objects belonging to each of the object groups are prepared for each of the object groups.

[0213] The selection unit 131 selects, from among a plurality of object groups, an object group indicated by the group selection information supplied from the listener position information receiving unit 21 .

[0214] The selection unit 131 then acquires the polar coordinate position information, gain information, and audio data prepared in advance for the polar coordinate objects of the selected object group, and supplies these to the polar coordinate position information encoding unit 23 and the audio encoding unit 24 .

[0215] <Description of Transmission Process and Reception Process> Next, the operation of the content reproduction system including the server 11 shown in FIG. 9 and the client 51 shown in FIG. 5 will be described.

[0216] That is, the transmission process by the server 11 and the reception process by the client 51 will be described below with reference to the flowchart in FIG.

[0217] When the receiving process is started in the client 51, the listener position information input unit 61 acquires the listener position information and the group selection information, and supplies them to the listener position information transmission unit 62. The listener position information input unit 61 also supplies the listener position information to the coordinate conversion unit 67.

[0218] Then, in step S 141 , the listener position information transmitting section 62 transmits the listener position information and group selection information supplied from the listener position information input section 61 to the server 11 .

[0219] More specifically, only when an object group is designated by a listener, the group selection information is transmitted to the server 11. Moreover, the listener position information and the group selection information may be transmitted at the same time or at different times.

[0220] When the listener position information and group selection information are transmitted in this manner, the server 11 performs a transmission process.

[0221] That is, in step S 171 , the listener position information receiving section 21 receives the listener position information and group selection information transmitted from the client 51 .

[0222] The listener position information receiving section 21 supplies the listener position information to an absolute coordinate position information encoding section 22 and a polar coordinate position information encoding section 23 , and also supplies group selection information to a selection section 131 .

[0223] After the process of step S171 is performed, the processes of steps S172 and S173 are performed. However, since these processes are similar to the processes of steps S42 and S43 in FIG. 6, the description thereof will be omitted.

[0224] In step S 174 , the selection unit 131 selects an object group based on the group selection information supplied from the listener position information receiving unit 21 .

[0225] The selection unit 131 acquires polar coordinate position information and gain information for the polar coordinate objects of the selected object group, and supplies them to the polar coordinate position information encoding unit 23.

[0226] More specifically, the selection unit 131 acquires polar coordinate position information and gain information for polar coordinate objects in category C1, and acquires only polar coordinate position information for polar coordinate objects in category C2.

[0227] Furthermore, for a polar coordinate object in category C3, the selection unit 131 acquires position information indicating the absolute position of the polar coordinate object in space, and supplies this to the polar coordinate position information encoding unit 23.

[0228] Furthermore, the selection unit 131 obtains the audio data of all the polar coordinate objects of the selected object group, and supplies this to the audio encoding unit 24 .

[0229] After the process of step S174 is performed, the processes of steps S175 to S179 are performed and the transmission process ends. However, since these processes are similar to the processes of steps S44 to S48 in FIG. 6, the description thereof will be omitted.

[0230] When the process of step S179 is performed and the bit stream is transmitted to the client 51, the process of steps S142 to S149 is performed in the client 51, and the receiving process ends.

[0231] The processes in steps S142 to S149 are similar to those in steps S12 to S19 in FIG. 6, and therefore will not be described further.

[0232] In this manner, the server 11 selects an object group based on the group selection information received from the client 51, and transmits to the client 51 the encoded polar coordinate position information and encoded audio data of the polar coordinate objects of that object group.

[0233] In this way, the listener can select and play back the background noise or reverberation sound that suits his / her taste from among a number of different background noises or reverberation sounds, thereby improving the listener's satisfaction.

[0234] Fourth embodiment <Client configuration example> Incidentally, the client 51 may prepare audio data of polar coordinate objects for each of a plurality of object groups in advance.

[0235] In such a case, the content reproduction system is composed of, for example, the server 11 shown in FIG. 4 and the client 51 shown in FIG.

[0236] However, in the server 11, for a polar coordinate object of a particular category, only the encoded polar coordinate position information and gain information are included in the bitstream, and the encoded audio data corresponding to the encoded polar coordinate position information is not included in the bitstream.

[0237] Fig. 11 is a diagram showing an example of the configuration of a client 51. In Fig. 11, the same reference numerals are used to designate parts corresponding to those in Fig. 5, and the description thereof will be omitted as appropriate.

[0238] The client 51 shown in FIG. 11 has a listener position information input unit 61, a listener position information transmission unit 62, a reception separation unit 63, an object separation unit 64, a polar coordinate position information decoding unit 65, an absolute coordinate position information decoding unit 66, a coordinate conversion unit 67, a recording unit 161, a selection unit 162, an audio decoding unit 68, a renderer 69, a format conversion unit 70, and a mixer 71.

[0239] The client 51 shown in FIG. 11 differs from the client 51 in FIG. 5 in that a recording unit 161 and a selection unit 162 are newly provided, but in other respects it has the same configuration as the client 51 in FIG.

[0240] In the client 51 of FIG. 11, the listener position information input unit 61 generates group selection information indicating an object group selected by the listener in response to an operation by the listener or the like, and supplies the group selection information to a selection unit 162.

[0241] The recording unit 161 pre-records, for a plurality of object groups, audio data of polar coordinate objects of a specific category that belong to the object group, and supplies the recorded audio data to the selection unit 162.

[0242] The selection unit 162 selects an object group indicated by the group selection information supplied from the listener position information input unit 61 from among a plurality of object groups prepared in advance.

[0243] In addition, the selection unit 162 reads audio data of polar coordinate objects of a specific category of the selected object group from the recording unit 161 based on the position encoding mode of the object supplied from the object separation unit 64, and supplies the audio data to the renderer 69.

[0244] Which object among a plurality of objects is a polar coordinate object of a particular category can be specified by the position encoding mode.

[0245] Furthermore, the client 51 associates (associates) the audio data read from the recording unit 161 with the polar coordinate position information and gain information extracted from the bit stream for each polar coordinate object in the selected object group.

[0246] In the following, the explanation will be continued assuming that the specific category of the polar coordinate object whose audio data is recorded in the recording unit 161 is category C1.

[0247] The audio data of the polar coordinate object recorded in the recording unit 161 may be encoded.

[0248] In such a case, the selection unit 162 reads out from the recording unit 161 the encoded audio data of the polar coordinate object of the specific category C1 of the selected object group, and supplies it to the audio decoding unit 68.

[0249] Also, here, an example will be described in which audio data is prepared in advance for each object group on the client 51 side only for polar coordinate objects in a specific category C1 among the polar coordinate objects.

[0250] However, audio data may be prepared in advance for each object group on the client 51 side for polar coordinate objects of all categories.

[0251] <Description of Transmission Process and Reception Process> Next, the operation of the content reproduction system including the server 11 shown in FIG. 4 and the client 51 shown in FIG. 11 will be described.

[0252] That is, the transmission process by the server 11 and the reception process by the client 51 will be described below with reference to the flowchart in FIG.

[0253] The process of step S201 in the reception process is similar to the process of step S11 in FIG. 6, and therefore a description thereof will be omitted.

[0254] Furthermore, when an object group is designated (selected) by an operation or the like of the listener at any timing, the listener position information input unit 61 supplies the selection unit 162 with group selection information indicating the designated object group.

[0255] After the process of step S201 is performed, the server 11 performs the processes of steps S241 to S248 as a transmission process.

[0256] The processes from step S241 to step S248 are similar to those from step S41 to step S48 in FIG. 6, and therefore will not be described further.

[0257] However, in step S246, no encoding of audio data is performed for polar coordinate objects in a specific predetermined category C1.

[0258] Therefore, the bitstream transmitted in step S248 contains encoded polar coordinate position information and gain information for polar coordinate objects in category C1, but does not contain encoded audio data.

[0259] When the process of step S248 is performed and the transmission process by the server 11 is completed, the client 51 performs the processes of steps S202 to S207.

[0260] The processes in steps S202 to S207 are similar to those in steps S12 to S17 in FIG. 6, and therefore will not be described further.

[0261] However, in step S 203 , the object separator 64 acquires the positional coding mode of each object extracted from the bitstream from the reception separator 63 , and supplies it to the selector 162 .

[0262] Also, in step S204, the encoded polar coordinate position information and gain information of each polar coordinate object of all categories is decoded.

[0263] Furthermore, in step S207, the coded audio data of absolute coordinate objects, the coded audio data of polar coordinate objects of categories C2 and C3, and the channel-based coded audio data are decoded.

[0264] In step S 208 , the selection unit 162 selects an object group based on the group selection information supplied from the listener position information input unit 61 .

[0265] Furthermore, the selection unit 162 identifies polar coordinate objects whose category is C1, based on the position encoding mode of each object supplied from the object separation unit 64.

[0266] The selection unit 162 reads out the audio data of the selected object group for each polar coordinate object in the category C1 from the recording unit 161 and supplies the audio data to the renderer 69.

[0267] Then, the processes of steps S209 and S210 are carried out and the reception process ends. However, since these processes are similar to the processes of steps S18 and S19 in FIG. 6, the description thereof will be omitted.

[0268] However, in step S209, the renderer 69 performs the rendering process using not only the audio data supplied from the audio decoding unit 68 but also the audio data supplied from the selection unit 162.

[0269] In this manner, the client 51 selects an object group based on the group selection information, reads out the audio data of a polar coordinate object in a specific category of the selected object group, and performs rendering processing.

[0270] In this way, the content can be reproduced with background noise and reverberation that suit the listener's taste, and the listener's satisfaction can be improved.

[0271] Fifth embodiment <Server and client configuration example> Also, when the polar coordinate object is a reverberation object, it may be possible to switch between encoding and transmitting polar coordinate position information and audio data, or transmitting reverb parameters for generating reverberation instead, to the client 51. Such switching is particularly useful when, for example, there is a limit to the transmission capacity of the bit stream.

[0272] For example, if audio data is prepared in advance for a polar coordinate object of reverberation sound, it is possible to reproduce a more accurate (highly precise) reverberation sound, that is, a reverberation sound closer to the real thing, from the audio data.

[0273] On the other hand, audio data of a polar coordinate object of a reverberation sound can be generated by performing reverb processing based on reverb parameters without preparing audio data of the polar coordinate object of the reverberation sound in advance.

[0274] In this case, the reverberation sound cannot be reproduced as accurately as when audio data of a pre-prepared polar coordinate object of the reverberation sound is used, but since polar coordinate position information and audio data are not required, the amount of coding in the bitstream can be reduced.

[0275] Furthermore, when playing back content, it is preferable to reproduce reverberation sounds associated with sounds of absolute coordinate objects located close to the listener as accurately as possible, but even if reverberation sounds associated with sounds of absolute coordinate objects located far from the listener are not reproduced accurately, no discomfort is felt in the auditory sense.

[0276] Therefore, for example, when the listener is close to the absolute coordinate object, the encoded polar coordinate position information of the polar coordinate object corresponding to the absolute coordinate object and the encoded audio data may be transmitted to the client 51. Here, the polar coordinate object corresponding to the absolute coordinate object is, for example, an object of reverberation sound generated by reflecting the sound (direct sound) of the absolute coordinate object.

[0277] Conversely, when the listener is far from the absolute coordinate object, the reverb parameters of the polar coordinate object corresponding to the absolute coordinate object may be transmitted to the client 51 .

[0278] This makes it possible to reduce the amount of code in the bit stream without causing any audible discomfort.

[0279] In the case where reverb parameters are transmitted appropriately in this manner, the content reproduction system is made up of, for example, the server 11 shown in FIG. 13 and the client 51 shown in FIG.

[0280] 13 and 14, parts corresponding to those in FIGS. 4 and 5 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

[0281] The server 11 shown in FIG. 13 has a listener position information receiving unit 21, an absolute coordinate position information encoding unit 22, a selection unit 191, a reverb parameter encoding unit 192, a polar coordinate position information encoding unit 23, an audio encoding unit 24, a bit stream generation unit 25, and a transmission unit 26.

[0282] The configuration of the server 11 in FIG. 13 differs from that of the server 11 in FIG. 4 in that a selection unit 191 and a reverb parameter encoding unit 192 are newly provided, but in other respects it has the same configuration as the server 11 in FIG.

[0283] In the example of FIG. 13, polar coordinate position information, gain information, audio data, and reverb parameters are prepared in advance for one or each of a plurality of polar coordinate objects.

[0284] Of course, there may be a polar coordinate object in which no reverb parameters are provided and the encoded polar coordinate position information and encoded audio data are always stored in a bit stream and transmitted to the client 51 .

[0285] For simplicity of explanation, the following will be described assuming that the content includes one absolute coordinate object and one polar coordinate object.

[0286] In this case, the absolute coordinate object is particularly an object of a direct sound of a musical instrument or the like, and the polar coordinate object is an object of a reverberation sound of the musical instrument or the like.

[0287] Based on the listener position information supplied from the listener position information receiving unit 21, the selection unit 191 selects whether to transmit the polar coordinate position information of the polar coordinate object or the like, or to transmit the reverb parameters.

[0288] For example, the selection unit 191 performs the selection based on the positional relationship between the listener and the absolute coordinate object, which is specified from the listener position information and the absolute coordinate position information.

[0289] Specifically, for example, when the distance from the listener to an absolute coordinate object is equal to or less than a predetermined threshold, the selection unit 191 selects transmission of the polar coordinate position information, etc. of the polar coordinate object corresponding to that absolute coordinate object.

[0290] In this case, the selection unit 191 obtains the polar coordinate position information and gain information of the polar coordinate object and supplies them to the polar coordinate position information encoding unit 23, and also obtains the audio data of the polar coordinate object and supplies them to the audio encoding unit 24.

[0291] On the other hand, if the distance from the listener to the absolute coordinate object is greater than a predetermined threshold, the selection unit 191 acquires the reverb parameters of the polar coordinate object corresponding to the absolute coordinate object, and supplies them to the reverb parameter encoding unit 192 .

[0292] The selection of whether to transmit polar coordinate position information or reverb parameters may be made by the listener.

[0293] In such a case, the listener position information receiving unit 21 receives selection information transmitted from the client 51 at any timing, indicating the selection result of whether to transmit polar coordinate position information, etc., or reverb parameters, and supplies it to the selection unit 191.

[0294] Based on the selection information supplied from the listener position information receiving unit 21, the selection unit 191 acquires polar coordinate position information of a polar coordinate object and / or acquires reverb parameters.

[0295] In addition, for example, the selection unit 191 may select whether to transmit polar coordinate position information, etc. or reverb parameters depending on the state of the communication path (transmission path) between the server 11 and the client 51, i.e., for example, the congestion state of the communication path.

[0296] In the following description, the state in which the transmission of polar coordinate position information and the like is selected and the polar coordinate position information and the like is transmitted to the client 51 is also referred to as a position information selection state.

[0297] In addition, the state in which the transmission of reverb parameters has been selected and the reverb parameters are being transmitted to the client 51 is also referred to as a reverb selected state.

[0298] The reverb parameter encoding unit 192 encodes the reverb parameters supplied from the selection unit 191 and supplies the encoded reverb parameters to the bit stream generation unit 25 .

[0299] Furthermore, in the case where a selection is made between transmitting polar coordinate position information and the like or transmitting reverb parameters, the client 51 is configured as shown in FIG.

[0300] The client 51 shown in FIG. 14 has a listener position information input unit 61, a listener position information transmission unit 62, a reception separation unit 63, an object separation unit 64, a reverb parameter decoding unit 221, a polar coordinate position information decoding unit 65, an absolute coordinate position information decoding unit 66, a coordinate conversion unit 67, an audio decoding unit 68, a reverb processing unit 222, a renderer 69, a format conversion unit 70, and a mixer 71.

[0301] 14 differs from the client 51 in FIG. 5 in that a reverb parameter decoding section 221 and a reverb processing section 222 are newly provided, but in other respects it has the same configuration as the client 51 in FIG.

[0302] In the example shown in FIG. 14, if the bit stream contains encoded reverb parameters of a polar coordinate object, the object separator 64 supplies the encoded reverb parameters to the reverb parameter decoder 221 .

[0303] The reverb parameter decoding unit 221 decodes the encoded reverb parameters supplied from the object separation unit 64 , and supplies the decoded reverb parameters to a reverb processing unit 222 .

[0304] The reverb processing unit 222 performs reverb processing on the audio data of the absolute coordinate object supplied from the audio decoding unit 68 based on the reverb parameters supplied from the reverb parameter decoding unit 221 .

[0305] As a result, for example, from audio data of an absolute coordinate object of a direct sound of a musical instrument or the like, audio data of a polar coordinate object of the reverberation sound of the musical instrument or the like is generated.

[0306] The reverberation processor 222 supplies the audio data of the polar coordinate object obtained by the reverberation process to the renderer 69 .

[0307] The audio data of the polar coordinate object thus obtained is used for rendering processing in the renderer 69, and the polar coordinate position information used in this process may be, for example, information indicating a predetermined position or information indicating a position determined from absolute coordinate position information.

[0308] <Description of Transmission Process and Reception Process> Next, the operation of the content reproduction system including the server 11 shown in FIG. 13 and the client 51 shown in FIG. 14 will be described.

[0309] That is, the transmission process by the server 11 and the reception process by the client 51 will be described below with reference to the flowchart in FIG.

[0310] In this case as well, for simplicity of explanation, it is assumed that there is one absolute coordinate object and one polar coordinate object.

[0311] When the receiving process is started in the client 51, the process of step S271 is performed and the listener position information is transmitted to the server 11. However, since the process of step S271 is similar to the process of step S11 in FIG. 6, the description thereof will be omitted.

[0312] In addition, when the listener operates the listener position information input unit 61 to select between the position information selection state and the reverb selection state, selection information indicating the selection result is supplied from the listener position information input unit 61 to the listener position information transmission unit 62.

[0313] Then, the listener position information transmitting section 62 transmits the selection information supplied from the listener position information input section 61 to the server 11 at an arbitrary timing.

[0314] After the process of step S271 is performed, the process of steps S311 to S313 is performed in the server 11. Note that these processes are similar to the processes of steps S41 to S43 in Fig. 6, and therefore the description thereof will be omitted.

[0315] However, in step S311, the listener position information receiving unit 21 supplies the received listener position information to the absolute coordinate position information encoding unit 22, the polar coordinate position information encoding unit 23, and the selection unit 191. When the listener position information receiving unit 21 receives selection information transmitted from the client 51, it supplies the selection information to the selection unit 191.

[0316] In step S314, the selection unit 191 determines whether or not to transmit polar coordinate position information.

[0317] That is, the selection unit 191 selects whether to transmit polar coordinate position information or the like or reverb parameters based on the listener position information and selection information supplied from the listener position information receiving unit 21 .

[0318] If it is determined in step S314 that the polar coordinate position information is to be transmitted, the processes of steps S315 and S316 are then carried out.

[0319] That is, the selection unit 191 obtains position information indicating the absolute position of the polar coordinate object and supplies it to the polar coordinate position information encoding unit 23, and also obtains audio data of the polar coordinate object and supplies it to the audio encoding unit 24.

[0320] Then, in step S315, the polar coordinate position information encoding unit 23 generates polar coordinate position information of the polar coordinate object based on the position information supplied from the selection unit 191 and the listener position information supplied from the listener position information receiving unit 21.

[0321] Furthermore, the polar coordinate position information encoding unit 23 also generates gain information based on the polar coordinate position information and the listener position information, as necessary.

[0322] When the polar coordinate position information and the gain information are obtained in advance, the polar coordinate position information and the gain information are acquired by the selection unit 191 and supplied to the polar coordinate position information encoding unit 23 .

[0323] In step S316, the polar coordinate position information encoding unit 23 encodes the polar coordinate position information and the gain information, and supplies the encoded information to the bit stream generating unit 25.

[0324] On the other hand, if it is determined in step S314 that the polar coordinate position information is not to be transmitted, that is, if it is determined that the reverb parameters are to be transmitted, then the process proceeds to step S317.

[0325] In this case, the selection unit 191 acquires the reverb parameters of the polar coordinate object and supplies them to the reverb parameter encoding unit 192 .

[0326] In step S 317 , the reverb parameter encoding unit 192 encodes the reverb parameters supplied from the selection unit 191 , and supplies the encoded reverb parameters to the bit stream generation unit 25 .

[0327] Although the case where there is one polar coordinate object will be described here as an example, if there are multiple polar coordinate objects, the above-mentioned processes from step S314 to step S317 are performed for each of the polar coordinate objects.

[0328] After the process of step S316 or step S317 is performed, the process of step S318 is performed.

[0329] In step S318, the audio encoding unit 24 encodes the audio data, and supplies the resulting encoded audio data to the bit stream generation unit 25.

[0330] For example, when the processing of steps S315 and S316 is performed, the audio encoding unit 24 encodes the audio data of the acquired absolute coordinate object, the audio data of the polar coordinate object supplied from the selection unit 191, and the acquired channel-based audio data.

[0331] On the other hand, when the process of step S317 is performed, the audio encoding unit 24 encodes the acquired audio data of the absolute coordinate object and the acquired channel-based audio data.

[0332] In step S319, the bit stream generating unit 25 generates a bit stream and supplies it to the transmitting unit .

[0333] For example, when the processing of steps S315 and S316 is performed, the bit stream generation unit 25 multiplexes the encoded absolute coordinate position information from the absolute coordinate position information encoding unit 22, the encoded polar coordinate position information and gain information from the polar coordinate position information encoding unit 23, and the encoded audio data from the audio encoding unit 24, to generate a bit stream.

[0334] In this case, the bitstream contains encoded polar position information of the polar object, gain information, and encoded audio data.

[0335] On the other hand, when the processing of step S317 is performed, the bitstream generation unit 25 multiplexes the encoded absolute coordinate position information from the absolute coordinate position information encoding unit 22, the encoded reverb parameters from the reverb parameter encoding unit 192, and the encoded audio data from the audio encoding unit 24, to generate a bitstream.

[0336] In this case, the bitstream contains the reverb parameters of the polar object, but does not contain the encoded polar position information of the polar object or the encoded audio data.

[0337] It should be noted that when reverb is selected, for a polar coordinate object, no encoded audio data is stored, but reverb parameters and encoded polar coordinate position information may be stored in the bit stream.

[0338] After the process of step S319 is performed, in step S320 the transmitting unit 26 transmits the bit stream supplied from the bit stream generating unit 25 to the client 51, and the transmission process ends.

[0339] Then, in the client 51, the processes of steps S272 to S276 are carried out. However, since these processes are similar to the processes of steps S12, S13, and S15 to S17 in FIG. 6, the description thereof will be omitted.

[0340] However, if the bit stream does not contain encoded audio data of a polar coordinate object, the audio decoding unit 68 supplies the audio data of an absolute coordinate object obtained by decoding not only to the renderer 69 but also to the reverb processing unit 222.

[0341] That is, if the bit stream contains encoded reverb parameters and reverb is selected, the audio data of the absolute coordinate object is also supplied to the reverb processing unit 222 .

[0342] In step S277, the object separator 64 determines whether or not the received bit stream contains encoded polar coordinate position information.

[0343] If it is determined in step S277 that encoded polar coordinate position information is included, the object separation unit 64 supplies the encoded polar coordinate position information and gain information supplied from the reception separation unit 63 to the polar coordinate position information decoding unit 65, and then processing proceeds to step S278.

[0344] In step S 278 , the polar coordinate position information decoding unit 65 decodes the encoded polar coordinate position information and gain information supplied from the object separating unit 64 , and supplies the obtained polar coordinate position information and gain information to the renderer 69 .

[0345] On the other hand, if it is determined in step S277 that the encoded polar coordinate position information is not included, that is, if the bit stream includes encoded reverb parameters, then the process proceeds to step S279.

[0346] In this case, the object separator 64 supplies the encoded reverb parameters supplied from the reception separator 63 to the reverb parameter decoder 221 .

[0347] In step S 279 , the reverb parameter decoding unit 221 decodes the encoded reverb parameters supplied from the object separation unit 64 , and supplies the decoded reverb parameters to the reverb processing unit 222 .

[0348] In step S 280 , the reverb processing unit 222 performs reverb processing on the audio data of the absolute coordinate object supplied from the audio decoding unit 68 based on the reverb parameters supplied from the reverb parameter decoding unit 221 .

[0349] The reverberation processor 222 supplies the audio data of the polar coordinate object obtained by the reverberation process to the renderer 69 .

[0350] Although the case where there is one polar coordinate object will be described here as an example, if there are multiple polar coordinate objects, the above-mentioned processing of steps S277 to S280 is carried out for each of those polar coordinate objects.

[0351] After the process of step S278 or step S280 is performed, the process of step S281 is performed.

[0352] In step S 281 , the renderer 69 performs rendering processing such as VBAP, and supplies the resulting audio data to the mixer 71 .

[0353] For example, if it is determined in step S277 that encoded polar coordinate position information is included, that is, if the position information is selected, the renderer 69 performs rendering processing based on the polar coordinate position information from the polar coordinate position information decoding unit 65, the polar coordinate position information from the coordinate conversion unit 67, and the audio data of the absolute coordinate object and the polar coordinate object from the audio decoding unit 68.

[0354] On the other hand, if it is determined in step S277 that the encoded polar coordinate position information is not included, that is, if reverb is selected, the renderer 69 performs rendering processing based on the polar coordinate position information from the coordinate conversion unit 67, the audio data of the absolute coordinate object from the audio decoding unit 68, and the audio data of the polar coordinate object from the reverb processing unit 222. In this case, the polar coordinate position information of the polar coordinate object is, for example, predetermined or generated from the polar coordinate position information of the absolute coordinate object.

[0355] After the rendering process is performed, the process of step S282 is performed and the reception process ends. However, since the process of step S282 is similar to the process of step S19 in FIG. 6, a description thereof will be omitted.

[0356] In this manner, the server 11 selects position information or reverb depending on the listener position information or selection information, and transmits a bitstream that includes encoded polar coordinate position information, etc., or that includes reverb parameters.

[0357] In this way, the amount of code in the bit stream can be reduced without causing any discomfort to the ear, that is, while maintaining the acoustic effect.

[0358] <Modification 1 of the fifth embodiment> <About cross-fade processing> In a content playback system consisting of the server 11 shown in FIG. 13 and the client 51 shown in FIG. 14, if a location information selection state is switched to a reverb selection state or a reverb selection state is switched to a location information selection state in an instant, there is a possibility that abnormal sounds such as discontinuous noise will occur.

[0359] Therefore, at the timing of switching from the position information selection state to the reverb selection state, and at the timing of switching from the reverb selection state to the position information selection state, smoothing such as cross-fade processing may be performed to suppress the occurrence of discontinuous noise, etc.

[0360] Here, the period consisting of one or more frames of audio data of an object when switching from a position information selection state to a reverb selection state, or when switching from a reverb selection state to a position information selection state, is also referred to as a switching period.

[0361] In this example, during the switching period, cross-fade processing is performed based on the audio data of the polar coordinate object obtained by reverb processing and the audio data of the polar coordinate object obtained by decoding.

[0362] In this case, basically, the transmission process and reception process described with reference to FIG.

[0363] However, in the transmission process performed by the server 11 during the switching period, both the processes of steps S315 and S316 and the process of step S317 are performed.

[0364] Therefore, the bitstream obtained in step S319 contains, for the polar object, encoded polar coordinate position information, gain information, encoded audio data, and encoded reverb parameters.

[0365] Therefore, in the reception process performed by the client 51 during the switching period, both the process of step S278 and the processes of steps S289 and S280 are performed.

[0366] Therefore, during the switching period, the renderer 69 is supplied with audio data of the polar coordinate object obtained by decoding from the audio decoding unit 68, and is also supplied with audio data of the polar coordinate object obtained by reverb processing from the reverb processing unit 222.

[0367] Therefore, in step S281 performed during the switching period, the renderer 69 performs cross-fade processing based on the audio data of the polar coordinate object obtained by decoding and the audio data of the polar coordinate object obtained by reverb processing.

[0368] That is, for example, the renderer 69 performs weighted addition of the audio data obtained by decoding and the audio data obtained by reverb processing while changing the weighting over time so as to gradually switch from one to the other.

[0369] Then, the audio data of the polar coordinate object obtained by such cross-fade processing is used to perform rendering processing.

[0370] In this way, it is possible to suppress the occurrence of discontinuous noise and the like, and to realize high-quality content reproduction.

[0371] Sixth embodiment <Server configuration example> Furthermore, polar coordinate position information may be prepared for each of a plurality of object groups on the server 11 side, and audio data of polar coordinate objects may be prepared for each of a plurality of object groups on the client 51 side.

[0372] In such a case, the content reproduction system is composed of, for example, the server 11 shown in Fig. 16 and the client 51 shown in Fig. 11. In Fig. 16, the parts corresponding to those in Fig. 9 are given the same reference numerals, and the description thereof will be omitted as appropriate.

[0373] The server 11 shown in FIG. 16 includes a listener position information receiving unit 21, an absolute coordinate position information encoding unit 22, a selection unit 131, a polar coordinate position information encoding unit 23, an audio encoding unit 24, a bit stream generation unit 25, and a transmission unit 26.

[0374] The configuration of the server 11 shown in Figure 16 is basically the same as the configuration of the server 11 shown in Figure 9, but differs from the server 11 in Figure 9 in that the selection unit 131 of the server 11 in Figure 16 does not output audio data of the polar coordinate object to the audio encoding unit 24.

[0375] That is, in the example of FIG. 16, the selection unit 131 selects an object group indicated by the group selection information supplied from the listener position information receiving unit 21 from among a plurality of object groups.

[0376] The selection unit 131 then acquires the polar coordinate position information, gain information, and the like prepared in advance for the polar coordinate objects of the selected object group, and supplies the information to the polar coordinate position information encoding unit 23 .

[0377] In particular, since the server 11 does not prepare audio data of the polar coordinate objects for each object group, the selection unit 131 does not supply the audio data of the polar coordinate objects of the selected object group to the audio encoding unit 24.

[0378] <Description of Transmission Process and Reception Process> Next, the operation of the content reproduction system including the server 11 shown in FIG. 16 and the client 51 shown in FIG. 11 will be described.

[0379] That is, the transmission process by the server 11 and the reception process by the client 51 will be described below with reference to the flowchart in FIG.

[0380] When the receiving process by the client 51 is started, the process of step S351 is performed and the listener position information and the group selection information are transmitted to the server 11. However, since the process of step S351 is similar to the process of step S141 in FIG. 10, the description thereof will be omitted.

[0381] Furthermore, after the process of step S351 is performed, the server 11 performs the processes of steps S381 to S389 as a transmission process. However, since these processes are similar to the processes of steps S171 to S179 in FIG. 10, a description thereof will be omitted.

[0382] However, since the audio data of the polar coordinate objects of the selected object group is not acquired by the selection unit 131, no encoding of the audio data of the polar coordinate objects of the selected object group is performed in step S387. Therefore, the bit stream transmitted in step S389 does not include encoded audio data of the polar coordinate objects.

[0383] After the process of step S389 is performed, the client 51 performs the processes of steps S352 through S357. However, since these processes are similar to the processes of steps S142 through S147 in FIG. 10, the description thereof will be omitted.

[0384] However, in this example, the bitstream does not contain encoded audio data of polar coordinate objects, so in step S357 only audio data of absolute coordinate objects and channel-based audio data are obtained by decoding.

[0385] In step S 358 , the selection unit 162 selects an object group based on the group selection information supplied from the listener position information input unit 61 .

[0386] Furthermore, for each polar coordinate object, the selection unit 162 reads out the audio data of the selected object group from the recording unit 161 and supplies it to the renderer 69 .

[0387] After the audio data of the polar coordinate object of the selected object group is read out in this manner, the processes of steps S359 and S360 are performed, and the receiving process ends. Note that these processes are similar to the processes of steps S148 and S149 in Fig. 10, and therefore the description thereof will be omitted.

[0388] In addition, as described above, for all polar coordinate objects in a selected object group, polar coordinate position information and gain information are read and encoded on the server 11 side, and audio data is read and rendered on the client 51 side.

[0389] However, the present invention is not limited to this, and audio data may be read and rendered only for polar coordinate objects of a specific category of the selected object group on the client 51 side. In such a case, the selection unit 162 identifies polar coordinate objects of a specific category based on the position encoding mode of each object supplied from the object separation unit 64.

[0390] In this manner, the server 11 selects an object group based on the group selection information, and reads and encodes the polar coordinate position information and gain information of the polar coordinate objects in the selected object group.

[0391] Furthermore, the client 51 selects an object group based on the group selection information, reads out the audio data of the polar coordinate objects of the selected object group, and performs rendering processing.

[0392] In this way, the content can be reproduced with background noise and reverberation that suit the listener's taste, and the listener's satisfaction can be improved.

[0393] Example of computer configuration The above-mentioned series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed in a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, for example, capable of executing various functions by installing various programs.

[0394] FIG. 18 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes according to a program.

[0395] In the computer, a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, and a RAM (Random Access Memory) 503 are interconnected via a bus 504.

[0396] An input / output interface 505 is further connected to the bus 504. An input unit 506, an output unit 507, a recording unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.

[0397] The input unit 506 includes a keyboard, a mouse, a microphone, an image sensor, etc. The output unit 507 includes a display, a speaker, etc. The recording unit 508 includes a hard disk, a non-volatile memory, etc. The communication unit 509 includes a network interface, etc. The drive 510 drives a removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0398] In a computer configured as described above, the CPU 501 loads, for example, a program recorded in the recording unit 508 into the RAM 503 via the input / output interface 505 and the bus 504, and executes the program, thereby performing the series of processes described above.

[0399] The program executed by the computer (CPU 501) can be provided by being recorded on a removable recording medium 511 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0400] In the computer, the program can be installed in the recording unit 508 via the input / output interface 505 by mounting the removable recording medium 511 in the drive 510. The program can also be received by the communication unit 509 via a wired or wireless transmission medium and installed in the recording unit 508. Alternatively, the program can be installed in the ROM 502 or the recording unit 508 in advance.

[0401] In addition, the program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or it may be a program in which processing is performed in parallel or at the required timing, such as when called.

[0402] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0403] For example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0404] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.

[0405] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices.

[0406] Furthermore, the present technology can also be configured as follows.

[0407] (1) an acquisition unit that acquires polar coordinate position information indicating a position of a first object expressed in polar coordinates, audio data of the first object, absolute coordinate position information indicating a position of a second object expressed in absolute coordinates, and audio data of the second object; a coordinate conversion unit that converts the absolute coordinate position information into polar coordinate position information that indicates a position of the second object; a rendering processing unit that performs a rendering process based on the polar coordinate position information and the audio data of the first object and the polar coordinate position information and the audio data of the second object; A signal processing device comprising: (2) The coordinate conversion unit converts the absolute coordinate position information of the second object into the polar coordinate position information based on listener position information indicating an absolute position of a listener. A signal processing device as described in (1). (3) The acquisition unit acquires the absolute coordinate position information of the second object based on the listener position information. A signal processing device as described in (2). (4) The acquisition unit acquires the absolute coordinate position information with an accuracy according to a positional relationship between the listener and the second object based on the listener position information. A signal processing device according to (3). (5) The acquisition unit acquires the polar coordinate position information indicating a position of the first object as seen by the listener based on the listener position information. A signal processing device according to any one of (2) to (4). (6) The rendering processing unit performs the rendering process in a polar coordinate system defined by MPEG-H. A signal processing device according to any one of (1) to (5). (7) The first object is a reverberant or background noise object. A signal processing device according to any one of (1) to (6). (8) The acquisition unit further acquires gain information of the first object, The polar coordinate position information or the gain information of the first object is a predetermined fixed value. A signal processing device according to any one of (1) to (7). (9) The acquisition unit acquires the polar coordinate position information and the audio data of the first object selected by a listener. A signal processing device according to any one of (1) to (8). (10) The acquisition unit further acquires channel-based audio data; The audio processing device further includes a mixing processing unit that mixes the channel-based audio data with the audio data obtained by the rendering process. A signal processing device according to any one of (1) to (9). (11) The channel-based audio data is audio data for reproducing background noise. A signal processing device according to (10). (12) the acquiring unit acquires the polar coordinate position information and the audio data for the first object, or acquires reverb parameters; a reverb processing unit that, when the reverb parameters are acquired, performs reverb processing based on the audio data of the second object corresponding to the first object and the reverb parameters to generate the audio data of the first object. A signal processing device according to any one of (1) to (8). (13) A signal processing device, obtaining polar coordinate position information indicating a position of a first object expressed in polar coordinates, audio data of the first object, absolute coordinate position information indicating a position of a second object expressed in absolute coordinates, and audio data of the second object; converting the absolute coordinate position information into polar coordinate position information indicative of a position of the second object; A rendering process is performed based on the polar coordinate position information and the audio data of the first object and the polar coordinate position information and the audio data of the second object. Signal processing methods. (14) obtaining polar coordinate position information indicating a position of a first object expressed in polar coordinates, audio data of the first object, absolute coordinate position information indicating a position of a second object expressed in absolute coordinates, and audio data of the second object; converting the absolute coordinate position information into polar coordinate position information indicative of a position of the second object; A rendering process is performed based on the polar coordinate position information and the audio data of the first object and the polar coordinate position information and the audio data of the second object. A program that causes a computer to execute a process including steps. (15) a polar coordinate position information encoding unit that encodes polar coordinate position information indicating a position of a first object expressed in polar coordinates; an absolute coordinate position information encoding unit that encodes absolute coordinate position information indicating a position of a second object expressed in absolute coordinates; an audio encoding unit for encoding the audio data of the first object and the audio data of the second object; a bitstream generating unit that generates a bitstream including the encoded polar coordinate position information, the encoded absolute coordinate position information, the encoded audio data of the first object, and the encoded audio data of the second object; A signal processing device comprising: (16) The absolute coordinate position information encoding unit encodes the absolute coordinate position information with an accuracy corresponding to listener position information indicating an absolute position of a listener. A signal processing device according to (15). (17) The absolute coordinate position information encoding unit encodes the absolute coordinate position information with an accuracy according to a positional relationship between the listener and the second object. A signal processing device according to (16). (18) The polar coordinate position information encoding unit encodes the polar coordinate position information indicating the position of the first object as seen by the listener. A signal processing device according to (16) or (17). (19) A signal processing device, encoding polar coordinate position information indicating a position of the first object expressed in polar coordinates; encoding absolute coordinate position information indicating a position of a second object expressed in absolute coordinates; encoding the audio data of the first object and the audio data of the second object; generating a bitstream including the encoded polar coordinate position information, the encoded absolute coordinate position information, the encoded audio data of the first object, and the encoded audio data of the second object; Signal processing methods. (20) encoding polar coordinate position information indicating a position of the first object expressed in polar coordinates; encoding absolute coordinate position information indicating a position of a second object expressed in absolute coordinates; encoding the audio data of the first object and the audio data of the second object; generating a bitstream including the encoded polar coordinate position information, the encoded absolute coordinate position information, the encoded audio data of the first object, and the encoded audio data of the second object; A program that causes a computer to execute a process including steps. [Explanation of symbols]

[0408] 11 server, 22 absolute coordinate position information encoding unit, 23 polar coordinate position information encoding unit, 24 audio encoding unit, 25 bit stream generation unit, 26 transmission unit, 51 client, 65 polar coordinate position information decoding unit, 66 absolute coordinate position information decoding unit, 67 coordinate conversion unit, 68 audio decoding unit, 69 renderer, 71 mixer

Claims

1. an acquisition unit that acquires polar coordinate position information indicating a position of a first object expressed in polar coordinates, audio data of the first object, absolute coordinate position information indicating a position of a second object expressed in absolute coordinates, and audio data of the second object; a coordinate conversion unit that converts the absolute coordinate position information into polar coordinate position information that indicates a position of the second object; a rendering processing unit that performs a rendering process based on the polar coordinate position information and the audio data of the first object and the polar coordinate position information and the audio data of the second object; A signal processing device comprising:

2. The coordinate conversion unit converts the absolute coordinate position information of the second object into the polar coordinate position information based on listener position information indicating an absolute position of a listener. The signal processing device according to claim 1 .

3. The acquisition unit acquires the absolute coordinate position information of the second object based on the listener position information. The signal processing device according to claim 2 .

4. The acquisition unit acquires the absolute coordinate position information with an accuracy according to a positional relationship between the listener and the second object based on the listener position information. The signal processing device according to claim 3 .

5. The acquisition unit acquires the polar coordinate position information indicating a position of the first object as seen by the listener based on the listener position information. The signal processing device according to claim 2 .

6. The rendering processing unit performs the rendering process in a polar coordinate system defined by MPEG-H. The signal processing device according to claim 1 .

7. The first object is a reverberation or background noise object. The signal processing device according to claim 1 .

8. The acquisition unit further acquires gain information of the first object, The polar coordinate position information or the gain information of the first object is a predetermined fixed value. The signal processing device according to claim 1 .

9. The acquisition unit acquires the polar coordinate position information and the audio data of the first object selected by a listener. The signal processing device according to claim 1 .

10. The acquisition unit further acquires channel-based audio data; The audio processing device further includes a mixing processing unit that mixes the channel-based audio data with the audio data obtained by the rendering process. The signal processing device according to claim 1 .

11. The channel-based audio data is audio data for reproducing background noise. The signal processing device according to claim 10.

12. the acquiring unit acquires the polar coordinate position information and the audio data for the first object, or acquires reverb parameters; a reverb processing unit that, when the reverb parameters are acquired, performs reverb processing based on the audio data of the second object corresponding to the first object and the reverb parameters to generate the audio data of the first object. The signal processing device according to claim 1 .

13. A signal processing device, obtaining polar coordinate position information indicating a position of a first object expressed in polar coordinates, audio data of the first object, absolute coordinate position information indicating a position of a second object expressed in absolute coordinates, and audio data of the second object; converting the absolute coordinate position information into polar coordinate position information indicative of a position of the second object; A rendering process is performed based on the polar coordinate position information and the audio data of the first object and the polar coordinate position information and the audio data of the second object. Signal processing methods.

14. obtaining polar coordinate position information indicating a position of a first object expressed in polar coordinates, audio data of the first object, absolute coordinate position information indicating a position of a second object expressed in absolute coordinates, and audio data of the second object; converting the absolute coordinate position information into polar coordinate position information indicative of a position of the second object; A rendering process is performed based on the polar coordinate position information and the audio data of the first object and the polar coordinate position information and the audio data of the second object. A program that causes a computer to execute a process including steps.

15. a polar coordinate position information encoding unit that encodes polar coordinate position information indicating a position of a first object expressed in polar coordinates; an absolute coordinate position information encoding unit that encodes absolute coordinate position information indicating a position of a second object expressed in absolute coordinates; an audio encoding unit that encodes the audio data of the first object and the audio data of the second object; a bitstream generating unit that generates a bitstream including the encoded polar coordinate position information, the encoded absolute coordinate position information, the encoded audio data of the first object, and the encoded audio data of the second object; A signal processing device comprising:

16. The absolute coordinate position information encoding unit encodes the absolute coordinate position information with an accuracy corresponding to listener position information indicating an absolute position of a listener. The signal processing device according to claim 15.

17. The absolute coordinate position information encoding unit encodes the absolute coordinate position information with an accuracy according to a positional relationship between the listener and the second object. The signal processing device according to claim 16.

18. The polar coordinate position information encoding unit encodes the polar coordinate position information indicating a position of the first object as seen by the listener. The signal processing device according to claim 16.

19. A signal processing device, Encoding polar coordinate position information indicating a position of the first object expressed in polar coordinates; encoding absolute coordinate position information indicating a position of a second object expressed in absolute coordinates; encoding the audio data of the first object and the audio data of the second object; generating a bitstream including the encoded polar coordinate position information, the encoded absolute coordinate position information, the encoded audio data of the first object, and the encoded audio data of the second object; Signal processing methods.

20. Encoding polar coordinate position information indicating a position of the first object expressed in polar coordinates; encoding absolute coordinate position information indicating a position of a second object expressed in absolute coordinates; encoding the audio data of the first object and the audio data of the second object; generating a bitstream including the encoded polar coordinate position information, the encoded absolute coordinate position information, the encoded audio data of the first object, and the encoded audio data of the second object; A program that causes a computer to execute a process including steps.

Citation Information

Patent Citations

  • Signal processing device, method, and program

    JP7552617B2

  • Information processing device, method, and program

    WO2019198540A1

  • IEC23008-3