Information processing device, method, and program

Abstract

The present technology relates to an information processing device and a method, and a program which make it possible to reproduce content based on the intention of content producer. This information processing device comprises: a listener position information acquisition unit that acquires listener position information for a viewpoint of a listener; a reference viewpoint information acquisition unit that acquires position information for a first reference viewpoint and object position information for an object at the first reference viewpoint, and position information for a second reference viewpoint and object position information for the object at the second reference viewpoint; and an object position calculation unit that calculates position information for the object at the viewpoint of the listener on the basis of the listener position information, the first reference viewpoint position information and the object position information for the object at the first reference viewpoint, and the second reference viewpoint position information and the object position information for the object at the second reference viewpoint. The present technology can be applied to a content reproduction system.

Description

[Technical field]

[0001] The present technology relates to an information processing device, method, and program, and in particular to an information processing device, method, and program that enable content playback based on the intentions of a content creator. [Background technology]

[0002] For example, in a free viewpoint space, each object placed in the space is fixedly positioned using an absolute coordinate system (see, for example, Patent Document 1).

[0003] In this case, the direction of each object as seen from any listening position is uniquely determined based on the relationship between the listener's coordinate position in absolute space, the direction of his or her face, and the object, and the gain of each object is uniquely determined based on the distance from the listening position, and the sound of each object is reproduced. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2019 / 198540 Summary of the Invention [Problem to be solved by the invention]

[0005] On the other hand, there are artistic aspects of the content and points that we want to emphasize to listeners.

[0006] For example, in the case of music content, there are cases where it is desirable for an object to be closer to the foreground, such as an instrument or player at a certain listening point that is to be emphasized in terms of the content, or a player that is to be emphasized in the case of sports content.

[0007] Taking this into consideration, the mere physical relationship between the listener and the object as described above may not adequately convey the interesting aspects of the content.

[0008] The present technology has been made in consideration of such circumstances, and makes it possible to realize content playback based on the intentions of the content creator while following the listener's free position. [Means for solving the problem]

[0009] An information processing device according to one aspect of the present technology includes a listener position information acquisition unit that acquires listener position information of a listener's viewpoint, a reference viewpoint information acquisition unit that acquires position information of a first reference viewpoint and object position information of an object at the first reference viewpoint, and position information of a second reference viewpoint and object position information of the object at the second reference viewpoint, and an object position calculation unit that calculates position information of the object at the listener's viewpoint based on the listener position information, the position information of the first reference viewpoint and the object position information at the first reference viewpoint, and the position information of the second reference viewpoint and the object position information at the second reference viewpoint.

[0010] An information processing method or program of one aspect of the present technology includes steps of acquiring listener position information of a listener's viewpoint, acquiring position information of a first reference viewpoint and object position information of an object at the first reference viewpoint, and position information of a second reference viewpoint and object position information of the object at the second reference viewpoint, and calculating position information of the object at the listener's viewpoint based on the listener position information, the position information of the first reference viewpoint and the object position information at the first reference viewpoint, and the position information of the second reference viewpoint and the object position information at the second reference viewpoint.

[0011] In one aspect of the present technology, listener position information of a listener's viewpoint is acquired, position information of a first reference viewpoint and object position information of an object at the first reference viewpoint, and position information of a second reference viewpoint and object position information of the object at the second reference viewpoint are acquired, and position information of the object at the listener's viewpoint is calculated based on the listener position information, the position information of the first reference viewpoint and the object position information at the first reference viewpoint, and the position information of the second reference viewpoint and the object position information at the second reference viewpoint. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing a configuration of a content reproduction system. [Diagram 2] FIG. 1 is a diagram showing a configuration of a content reproduction system. [Diagram 3] FIG. 2 is a diagram illustrating a reference viewpoint. [Figure 4] FIG. 4 is a diagram illustrating an example of system configuration information. [Diagram 5] FIG. 4 is a diagram illustrating an example of system configuration information. [Figure 6] FIG. 1 is a diagram illustrating coordinate transformation. [Figure 7] FIG. 11 is a diagram illustrating a coordinate axis conversion process. [Figure 8] 11A and 11B are diagrams illustrating examples of conversion results obtained by coordinate axis conversion processing. [Figure 9] FIG. 11 is a diagram illustrating an interpolation process. [Figure 10] FIG. 1 is a diagram illustrating an example of a sequence of a content reproduction system. [Figure 11] 11A and 11B are diagrams illustrating an example of moving an object closer to its arrangement at the reference viewpoint. [Figure 12] 11A and 11B are diagrams illustrating interpolation of object absolute coordinate position information. [Figure 13] 11A and 11B are diagrams illustrating internal division ratios in a triangular mesh on the viewpoint side. [Figure 14] 11A and 11B are diagrams illustrating calculation of an object position using an internal division ratio. [Figure 15] 11 is a diagram illustrating calculation of gain information based on an internal division ratio. FIG. [Figure 16] FIG. 13 is a diagram illustrating selection of a triangular mesh. [Figure 17] FIG. 1 is a diagram showing a configuration of a content reproduction system. [Figure 18] 11 is a flowchart illustrating a providing process and a playback audio data generating process. [Figure 19] 11 is a flowchart illustrating a viewpoint selection process. [Figure 20] FIG. 1 illustrates an example of the configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, embodiments to which the present technology is applied will be described with reference to the drawings.

[0014] First Embodiment Example of a Content Playback System Configuration The present technology has the following features F1 to F6.

[0015] (Feature F1) The present invention is characterized in that object arrangement and gain information are prepared in advance for a plurality of reference viewpoints in a free viewpoint space. (Feature F2) The system is characterized by determining the object position and gain information at an arbitrary listening point (listening position) based on the object arrangement and gain information at multiple reference viewpoints that sandwich or surround the arbitrary listening point. (Feature F3) When determining the object position and gain amount of an arbitrary listening point, a proportional ratio is calculated using a plurality of reference viewpoints that sandwich or surround the arbitrary listening point and the arbitrary listening point, and the object position for the arbitrary listening point is determined using the proportional ratio. (Feature F4) The object placement information at a plurality of reference viewpoints prepared in advance is transmitted using a polar coordinate system. (Feature F5) The object placement information at a plurality of reference viewpoints prepared in advance is transmitted using an absolute coordinate system. (Feature F6) When calculating the object position at an arbitrary listening point, a specific bias coefficient is used to enable the listener to hear the object at a position close to one of the reference viewpoints.

[0016] First, a content playback system to which the present technology is applied will be described.

[0017] The content playback system includes a server and a client that perform encoding, transmission, and decoding of each data.

[0018] For example, listener position information is transmitted from the client to the server as necessary, and based on the result, some object position information is transmitted from the server to the client. Then, based on the object position information received on the client, rendering processing is performed for each object, and content consisting of the sound of each object is played back.

[0019] Such a content reproduction system is configured, for example, as shown in FIG.

[0020] That is, the content reproduction system shown in FIG.

[0021] The server 11 includes a configuration information sending unit 21 and an encoded data sending unit 22 .

[0022] The configuration information sending unit 21 sends (transmits) system configuration information prepared in advance to the client 12, and receives viewpoint selection information, etc. sent from the client 12 and supplies it to the encoded data sending unit 22.

[0023] In a content reproduction system, a plurality of listening positions in a predetermined common absolute coordinate space are designated (set) in advance by a content creator as reference viewpoint positions (hereinafter also referred to as reference viewpoint positions).

[0024] Here, the content creator specifies (sets) in advance as a reference viewpoint the position in the common absolute coordinate space that the content creator wants the listener to use as the listening position when the content is played back, and the facial direction that the content creator wants the listener to face at that position, i.e., the viewpoint from which the content creator wants the listener to listen to the sound of the content.

[0025] The server 11 is provided in advance with system configuration information, which is information relating to each reference viewpoint, and object polar coordinate encoded data for each reference viewpoint.

[0026] Here, the object polar coordinate encoded data for each reference viewpoint is obtained by encoding object polar coordinate position information that indicates the relative position of an object as seen from the reference viewpoint. In the object polar coordinate position information, the position of an object as seen from the reference viewpoint is expressed by polar coordinates. Note that even for the same object, the absolute placement position of the object in the common absolute coordinate space differs for each reference viewpoint.

[0027] The configuration information sending unit 21 sends the system configuration information to the client 12 via a network or the like immediately after the content reproduction system starts operating, that is, for example, immediately after a connection with the client 12 is established.

[0028] The encoded data sending unit 22 selects two reference viewpoints from among the multiple reference viewpoints based on the viewpoint selection information supplied from the configuration information sending unit 21, and sends the object polar coordinate encoded data of each of the two selected reference viewpoints to the client 12 via a network or the like.

[0029] Here, the viewpoint selection information is information indicating two reference viewpoints selected on the client 12 side, for example.

[0030] Therefore, the encoded data sending unit 22 acquires the object polar coordinate encoded data of the reference viewpoint requested by the client 12 and sends it to the client 12. Note that the number of reference viewpoints selected by the viewpoint selection information is not limited to two, and may be three or more.

[0031] The client 12 also has a listener position information acquisition unit 41, a viewpoint selection unit 42, a configuration information acquisition unit 43, an encoded data acquisition unit 44, a decoding unit 45, a coordinate conversion unit 46, a coordinate axis conversion processing unit 47, an object position calculation unit 48, and a polar coordinate conversion unit 49.

[0032] The listener position information acquisition unit 41 acquires listener position information indicating the absolute position (listening position) of the listener in the common absolute coordinate space in response to a specification operation by a user (listener), etc., and supplies the information to the viewpoint selection unit 42, the object position calculation unit 48, and the polar coordinate conversion unit 49.

[0033] For example, in the listener position information, the listener's position in the common absolute coordinate space is expressed by absolute coordinates. Note that hereinafter, the coordinate system of the absolute coordinates indicated by the listener position information will also be referred to as the common absolute coordinate system.

[0034] The viewpoint selection unit 42 selects two reference viewpoints based on the system configuration information supplied from the configuration information acquisition unit 43 and the listener position information supplied from the listener position information acquisition unit 41, and supplies viewpoint selection information indicating the selection result to the configuration information acquisition unit 43.

[0035] For example, in the viewpoint selection unit 42, a section is specified based on the position of the listener (listening position) and the assumed absolute coordinate positions of each reference viewpoint, and two reference viewpoints are selected based on the result of specifying the section.

[0036] The configuration information acquisition unit 43 receives system configuration information transmitted from the server 11 and supplies it to the viewpoint selection unit 42 and the coordinate axis transformation processing unit 47, and transmits the viewpoint selection information supplied from the viewpoint selection unit 42 to the server 11 via a network, etc.

[0037] Note that, here, an example will be described in which a viewpoint selection unit 42 that selects a reference viewpoint based on listener position information and system configuration information is provided in the client 12, but the viewpoint selection unit 42 may also be provided on the server 11 side.

[0038] The encoded data acquisition unit 44 receives the object polar coordinate encoded data transmitted from the server 11 and supplies it to the decoding unit 45. That is, the encoded data acquisition unit 44 acquires the object polar coordinate encoded data from the server 11.

[0039] The decoding unit 45 decodes the object polar coordinate encoded data supplied from the encoded data acquisition unit 44, and supplies the object polar coordinate position information obtained as a result to a coordinate conversion unit 46.

[0040] The coordinate conversion unit 46 performs coordinate conversion on the object polar coordinate position information supplied from the decoding unit 45 , and supplies the resulting object absolute coordinate position information to a coordinate axis conversion processing unit 47 .

[0041] The coordinate conversion unit 46 performs coordinate conversion to convert polar coordinates into absolute coordinates. As a result, object polar coordinate position information, which is polar coordinates indicating the position of an object as seen from the reference viewpoint, is converted into object absolute coordinate position information, which is absolute coordinates indicating the position of an object in an absolute coordinate system with the position of the reference viewpoint as the origin.

[0042] The coordinate axis conversion processing unit 47 performs coordinate axis conversion processing on the object absolute coordinate position information supplied from the coordinate conversion unit 46 based on the system configuration information supplied from the configuration information acquisition unit 43 .

[0043] Here, the coordinate axis transformation process is a process that combines coordinate transformation (coordinate axis transformation) and offset shift, and the coordinate axis transformation process obtains object absolute coordinate position information that indicates the absolute coordinates of the object projected onto the common absolute coordinate space. That is, the object absolute coordinate position information obtained by the coordinate axis transformation process is the absolute coordinates of the common absolute coordinate system that indicates the absolute position of the object on the common absolute coordinate space.

[0044] The object position calculation unit 48 performs an interpolation process based on the listener position information supplied from the listener position information acquisition unit 41 and the object absolute coordinate position information supplied from the coordinate axis conversion processing unit 47, and supplies the final object absolute coordinate position information obtained as a result to the polar coordinate conversion unit 49. The final object absolute coordinate position information here refers to information indicating the position of the object in the common absolute coordinate system when the listener's viewpoint is at the listening position indicated by the listener position information.

[0045] The object position calculation unit 48 calculates the absolute position of the object in the common absolute coordinate space corresponding to the listening position, that is, the absolute coordinates of the common absolute coordinate system, from the listening position indicated by the listener position information and the positions of the two reference viewpoints indicated by the viewpoint selection information, and sets this as the final object absolute coordinate position information. At this time, the object position calculation unit 48 acquires system configuration information from the configuration information acquisition unit 43 and acquires viewpoint selection information from the viewpoint selection unit 42 as necessary.

[0046] The polar coordinate conversion unit 49 performs polar coordinate conversion on the object absolute coordinate position information supplied from the object position calculation unit 48, based on the listener position information supplied from the listener position information acquisition unit 41, and outputs the resulting polar coordinate position information to a downstream rendering processing unit (not shown).

[0047] The polar coordinate conversion unit 49 performs polar coordinate conversion to convert the object absolute coordinate position information, which is the absolute coordinates in the common absolute coordinate system, into polar coordinate position information, which is the polar coordinates indicating the relative position of the object as viewed from the listening position.

[0048] In the above, an example has been described in which the server 11 prepares object polar coordinate encoded data for each reference viewpoint in advance, but the server 11 may also prepare object absolute coordinate position information in advance, which is output from the coordinate axis transformation processing unit 47.

[0049] In such a case, the content reproduction system may be configured, for example, as shown in Fig. 2. Note that in Fig. 2, parts corresponding to those in Fig. 1 are given the same reference numerals, and their explanation will be omitted as appropriate.

[0050] The content reproduction system shown in FIG.

[0051] In addition, the server 11 has a configuration information sending unit 21 and an encoded data sending unit 22, and in this example, the encoded data sending unit 22 acquires object absolute coordinate encoded data of the two reference viewpoints indicated by the viewpoint selection information and sends it to the client 12.

[0052] That is, in the server 11, object absolute coordinate encoded data obtained by encoding the object absolute coordinate position information output from the coordinate axis transformation processing unit 47 shown in FIG. 1 is prepared in advance for each of the multiple reference viewpoints.

[0053] Therefore, in this example, the client 12 is not provided with the coordinate conversion unit 46 and the coordinate axis conversion processing unit 47 shown in FIG.

[0054] That is, the client 12 shown in FIG. 2 is configured to have a listener position information acquisition unit 41, a viewpoint selection unit 42, a configuration information acquisition unit 43, an encoded data acquisition unit 44, a decoding unit 45, an object position calculation unit 48, and a polar coordinate conversion unit 49.

[0055] The configuration of the client 12 shown in FIG. 2 differs from the configuration of the client 12 shown in FIG. 1 in that the coordinate conversion unit 46 and the coordinate axis conversion processing unit 47 are not provided, but in other respects it has the same configuration as the client 12 shown in FIG. 1.

[0056] The encoded data acquisition unit 44 receives the object absolute coordinate encoded data transmitted from the server 11 and supplies it to a decoding unit 45 .

[0057] The decoding unit 45 decodes the object absolute coordinate encoded data supplied from the encoded data acquisition unit 44, and supplies the resulting object absolute coordinate position information to an object position calculation unit .

[0058] About this technology Next, the present technology will be further described.

[0059] First, the process of creating the content provided from the server 11 to the client 12 will be described.

[0060] First, an example using a transmission method based on a polar coordinate system, that is, an example of transmitting object polar coordinate encoded data as shown in FIG. 1, will be described.

[0061] Content production using the polar coordinate system is currently used for fixed-viewpoint-based 3D audio, and this has the advantage that such production techniques can be used as is.

[0062] A content creator (hereinafter simply referred to as a creator) sets multiple reference viewpoints in a three-dimensional space from which the creator wants the listener to hear the content.

[0063] Specifically, for example, four reference viewpoints are set in a common absolute coordinate space, which is a three-dimensional space, as shown in Fig. 3. Here, four positions P11 to P14 designated by the creator are set as reference viewpoints, more specifically, positions of the reference viewpoints.

[0064] Reference viewpoint information, which is information relating to each reference viewpoint, is composed of reference viewpoint position information, which is the standing position within a common absolute coordinate space, i.e., the absolute coordinates of the common absolute coordinate system indicating the position of the reference viewpoint, and listener orientation information, which indicates the direction of the listener's face.

[0065] Here, the listener orientation information is composed of, for example, the horizontal rotation angle (horizontal angle) of the listener's face from the reference viewpoint, and a vertical angle indicating the vertical orientation of the listener's face.

[0066] In FIG. 3, the arrows drawn adjacent to each of the positions P11 to P14 represent listener orientation information at the reference viewpoints indicated by each of the positions P11 to P14, that is, the direction of the listener's face.

[0067] 3, region R11 shows an example of a region where an object exists, and in this example, it can be seen that the listener's face direction indicated by the listener orientation information is in the direction of region R11 at each reference viewpoint. For example, at position P14, the listener's face direction indicated by the listener orientation information is backward.

[0068] Next, the creator sets object polar coordinate position information, which expresses the position of each object at each of the multiple reference viewpoints in polar coordinate format, and a gain amount for each object at each of the reference viewpoints. For example, the object polar coordinate position information includes the horizontal angle and vertical angle of the object as seen from the reference viewpoint, and a radius indicating the distance from the reference viewpoint to the object.

[0069] When the position of the object and the like are set for each of the multiple reference viewpoints in this manner, the following information IFP1 to information IFP5 are obtained as information relating to the reference viewpoints.

[0070] (Information IFP1) Number of objects (Information IFP2) Number of reference viewpoints (Information IFP3) The listener's face orientation at the reference viewpoint (horizontal angle, vertical angle) (Information IFP4) Absolute coordinate position of the reference eye in absolute space (common absolute coordinate space) (Information IFP5) Polar coordinate position (horizontal angle, vertical angle, radius) and gain amount of each object seen from information IFP3 and information IFP4

[0071] Here, information IFP3 is the above-mentioned listener-oriented information, and information IFP4 is the above-mentioned reference viewpoint position information.

[0072] The polar coordinate position as information IFP5 is object polar coordinate position information consisting of a horizontal angle, a vertical angle, and a radius, and indicating the relative position of the object with respect to the reference viewpoint. This object polar coordinate position information is equivalent to the polar coordinate encoding information of MPEG (Moving Picture Experts Group)-H, so the MPEG-H encoding method can be utilized.

[0073] Of these information IFP1 to information IFP5, the information included in information IFP1 to information IFP4 constitutes the above-mentioned system configuration information.

[0074] This system configuration information is transmitted to the client 12 prior to transmission of data relating to the object, that is, the object polar coordinate encoded data and the encoded audio data obtained by encoding the audio data of the object.

[0075] A specific example of the system configuration information is as shown in FIG.

[0076] In the example shown in FIG. 4, "NumOfObjs" indicates the number of objects that constitute the content, that is, the above-mentioned information IFP1, and "NumfOfRefViewPoint" indicates the number of reference viewpoints, that is, the above-mentioned information IFP2.

[0077] Moreover, the system configuration information shown in FIG. 4 includes reference viewpoint information for the number of reference viewpoints "NumfOfRefViewPoint".

[0078] In other words, "RefViewX[i]", "RefViewY[i]", and "RefViewZ[i]" respectively indicate the X coordinate, Y coordinate, and Z coordinate of a common absolute coordinate system indicating the position of the reference viewpoint, which constitute the reference viewpoint position information of the i-th reference viewpoint as information IFP4.

[0079] Furthermore, "ListenerYaw[i]" and "ListenerPitch[i]" are the horizontal angle (yaw angle) and vertical angle (pitch angle) that constitute the listener direction information of the i-th reference viewpoint as information IFP3.

[0080] Furthermore, in this example, the system configuration information includes information "ObjectOverLapMode[i]" that indicates the playback mode for each object when the listener and the object overlap, i.e., when the listener (listening position) and the object are in the same position.

[0081] Next, an example of using a transmission method based on an absolute coordinate system, that is, an example of transmitting object absolute coordinate coded data as shown in FIG. 2, will be described.

[0082] When transmitting object absolute coordinate coded data, the object positions with respect to each reference viewpoint are recorded as absolute coordinate position information, as in the case of transmitting object polar coordinate coded data. In other words, the creator prepares object absolute coordinate position information of each object for each reference viewpoint.

[0083] However, in this example, unlike the example of the transmission method using the polar coordinate system, there is no need to transmit listener direction information indicating the direction of the listener's face.

[0084] In the example using the transmission method based on the absolute coordinate system, the following information IFA1 to information IFA4 are obtained as information related to the reference viewpoint.

[0085] (Information IFA1) Number of objects (Information IFA2) Number of reference viewpoints (Information IFA3) Absolute coordinate position of the reference eye in absolute space (Information IFA4) Absolute coordinate position and gain amount of each object when the listener is at the absolute coordinate position shown in information IFA3

[0086] Here, the information IFA1 and the information IFA2 are the same information as the above-mentioned information IFP1 and information IFP2, and the information IFA3 is the above-mentioned reference viewpoint position information.

[0087] Moreover, the absolute coordinate position of the object indicated by the information IFA4 is object absolute coordinate position information indicating the absolute position of the object on the common absolute coordinate space, which is indicated by the absolute coordinates of the common absolute coordinate system.

[0088] When transmitting the object absolute coordinate encoded data from the server 11 to the client 12, object absolute coordinate position information indicating the position of the object with an accuracy according to the positional relationship between the listener and the object, for example, the distance from the listener to the object, may be generated and transmitted. In this case, the amount of information (number of bits) of the object absolute coordinate position information can be reduced without making the listener feel a shift in the sound image position.

[0089] For example, the shorter the distance from the listener to the object, the more accurate the object absolute coordinate position information (object absolute coordinate encoded data), that is, the more accurate the object absolute coordinate position information that is generated.

[0090] This is because the quantization precision (quantization step width) used during encoding causes a shift in the position of an object, and the greater the distance from the listener to the object, the greater the amount of positional shift (tolerable error) that does not cause the listener to perceive a shift in the position of the sound image.

[0091] Specifically, for example, object absolute coordinate encoded data obtained by encoding the object absolute coordinate position information with the highest accuracy is prepared in advance and held in the server 11 .

[0092] Then, by extracting a portion of this highest precision object absolute coordinate encoded data, it is possible to obtain object absolute coordinate encoded data obtained by quantizing the object absolute coordinate position information with any quantization precision.

[0093] Therefore, the encoded data sending unit 22 extracts a part or all of the object absolute coordinate encoded data with the highest accuracy depending on the distance from the listening position to the object, and sends the resulting object absolute coordinate encoded data with a predetermined accuracy to the client 12. In this case, the encoded data sending unit 22 only needs to acquire listener position information from the listener position information acquiring unit 41 via the configuration information sending unit 21, the configuration information acquiring unit 43, and the viewpoint selecting unit 42.

[0094] In the content reproduction system shown in FIG. 2, system configuration information including each piece of information IFA1 to IFA3 among information IFA1 to information IFA4 is prepared in advance.

[0095] This system configuration information is transmitted to the client 12 prior to transmission of data relating to the object, that is, the encoded absolute coordinate data of the object and the encoded audio data.

[0096] A specific example of such system configuration information is as shown in FIG.

[0097] In the example shown in FIG. 5, similarly to the example shown in FIG. 4, the system configuration information includes the number of objects "NumOfObjs" and the number of reference viewpoints "NumfOfRefViewPoint".

[0098] The system configuration information also includes reference viewpoint information for the number of reference viewpoints "NumfOfRefViewPoint".

[0099] That is, the system configuration information includes the X coordinate "RefViewX[i]", the Y coordinate "RefViewY[i]", and the Z coordinate "RefViewZ[i]" of the common absolute coordinate system indicating the position of the reference viewpoint, which constitute the reference viewpoint position information of the i-th reference viewpoint. As described above, in this example, the reference viewpoint information does not include listener orientation information, and only reference viewpoint position information is included.

[0100] Furthermore, the system configuration information includes, for each object, a playback mode "ObjectOverLapMode[i]" for when the listener and the object overlap each other.

[0101] The system configuration information obtained in the above manner, the object polar coordinate encoded data and object absolute coordinate encoded data of each object for each reference viewpoint, and encoding gain information obtained by encoding gain information indicating the gain amount are stored in the server 11.

[0102] In the following, when there is no need to distinguish between object polar coordinate position information and object absolute coordinate position information, they will be referred to simply as object position information. Similarly, in the following, when there is no need to distinguish between object polar coordinate encoded data and object absolute coordinate encoded data, they will be referred to simply as object coordinate encoded data.

[0103] When the content playback system starts operating, the configuration information transmitter 21 of the server 11 transmits the system configuration information to the client 12 prior to transmission of the object coordinate encoded data. This enables the client 12 to grasp the number of objects constituting the content, the number of reference viewpoints, the positions of the reference viewpoints in the common absolute coordinate space, and the like.

[0104] Next, the viewpoint selection unit 42 of the client 12 selects a reference viewpoint in accordance with the listener position information, and the configuration information acquisition unit 43 sends to the server 11 viewpoint selection information indicating the selection result.

[0105] As described above, the viewpoint selection unit 42 may be provided in the server 11, and the reference viewpoint may be selected on the server 11 side.

[0106] In such a case, the viewpoint selection unit 42 selects a reference viewpoint based on the listener position information received from the client 12 by the configuration information transmission unit 21 and the system configuration information, and supplies viewpoint selection information indicating the selection result to the encoded data transmission unit 22.

[0107] At this time, the viewpoint selection unit 42 identifies and selects two (or more) reference viewpoints that sandwich the listening position indicated by the listener position information, for example. In other words, the two reference viewpoints are selected so that the listening position is located between the two reference viewpoints.

[0108] As a result, the object coordinate encoded data for each of the selected reference viewpoints is transmitted to the client 12. More specifically, the encoded data transmitter 22 transmits not only the object coordinate encoded data but also the encoding gain information to the client 12 for the two reference viewpoints indicated by the viewpoint selection information.

[0109] On the client 12 side, based on the object coordinate encoded data, encoding gain information, and listener position information at each of the multiple reference viewpoints received from the server 11, absolute object coordinate position information and gain information at the current arbitrary viewpoint of the listener are calculated by interpolation processing or the like.

[0110] Here, a specific example of calculation of final object absolute coordinate position information and gain information at an arbitrary viewpoint of the current listener will be described.

[0111] In particular, an example of the interpolation process using a data set of reference viewpoints in a polar coordinate system as two reference viewpoints sandwiching a listener will be described below.

[0112] In such a case, the client 12 performs the following processes PC1 to PC4 in order to obtain final absolute coordinate position information and gain information of the object from the listener's viewpoint.

[0113] (Processing PC1) In process PC1, the data sets at the reference viewpoints of the two polar coordinate systems are converted into absolute coordinate system positions for the objects included in each data set, with each reference viewpoint being set as the origin. That is, the coordinate conversion unit 46 performs coordinate conversion as process PC1 on the object polar coordinate position information of each object for each reference viewpoint, and object absolute coordinate position information is generated.

[0114] For example, as shown in Fig. 6, assume that there is one object OBJ11 in a polar coordinate system space with origin O as the reference point. Also, a three-dimensional Cartesian coordinate system (absolute coordinate system) with origin O as the reference point (origin) and x-axis, y-axis, and z-axis as the respective axes will be called an xyz coordinate system.

[0115] In this case, the position of object OBJ11 in the polar coordinate system can be represented by polar coordinates consisting of a horizontal angle θ, which is the angle in the horizontal direction, a vertical angle γ, which is the angle in the vertical direction, and a radius r, which indicates the distance from the origin O to the object OBJ11. In this example, the polar coordinates (θ, γ, r) are the object polar coordinate position information of object OBJ11.

[0116] The horizontal angle θ is the horizontal angle starting from the origin O, i.e., in front of the listener. In this example, if the straight line (line segment) connecting the origin O and the object OBJ11 is LN, and the straight line obtained by projecting the straight line LN onto the xy plane is LN', the angle between the y-axis and the straight line LN' is the horizontal angle θ.

[0117] Moreover, the vertical angle γ is the vertical angle starting from the origin O, i.e., in front of the listener, and in this example, the angle between the straight line LN and the xy plane is the vertical angle γ. Furthermore, the radius r is the distance from the listener (the origin O) to the object OBJ11, i.e., the length of the straight line LN.

[0118] If the position of such an object OBJ11 is expressed by the coordinates (x, y, z) of the xyz coordinate system, that is, by absolute coordinates, it is as shown in the following formula (1).

[0119]

number

[0120] In process PC1, by calculating equation (1) based on the object polar coordinate position information, which is a polar coordinate, object absolute coordinate position information, which is absolute coordinates indicating the position of the object in an xyz coordinate system (absolute coordinate system) with the position of the reference viewpoint as the origin O, is calculated.

[0121] In particular, in the process PC1, for each of the two reference viewpoints, coordinate conversion is performed on the object polar coordinate position information of each of a plurality of objects at those reference viewpoints.

[0122] (Processing PC2) In the process PC2, for each of the two reference viewpoints, a coordinate axis conversion process is performed on the object absolute coordinate position information obtained in the process PC1 for each object. That is, the coordinate axis conversion processing unit 47 performs the coordinate axis conversion process in the process PC2.

[0123] The object absolute coordinate position information at each of the two reference viewpoints obtained in the above-mentioned process PC1, i.e., obtained by the coordinate conversion unit 46, indicates a position in an xyz coordinate system with each reference viewpoint as its origin O. Therefore, the coordinates (coordinate system) of the object absolute coordinate position information differ for each reference viewpoint.

[0124] Therefore, a coordinate axis conversion process is performed as process PC2 to consolidate the object absolute coordinate position information at each reference viewpoint into absolute coordinates in one common absolute coordinate system, that is, absolute coordinates in a common absolute coordinate system (common absolute coordinate space).

[0125] To perform this coordinate axis transformation process, in addition to a data set for each reference viewpoint, i.e., object absolute coordinate position information of each object for each reference viewpoint, absolute position information of the listener (reference viewpoint position information) and listener orientation information indicating the direction of the listener's face are required.

[0126] In other words, the coordinate axis conversion process requires the object absolute coordinate position information obtained by process PC1, reference viewpoint position information indicating the position of the reference viewpoint in the common absolute coordinate system, and system configuration information including listener direction information at the reference viewpoint.

[0127] For simplicity of explanation, only the horizontal rotation angle is used as the face direction indicated by the listener direction information, but pitch information of the face can also be added.

[0128] Now, assuming that the common absolute coordinate system is an XYZ coordinate system with the X-axis, Y-axis, and Z-axis as its respective axes, and that the rotation angle according to the face direction indicated by the listener direction information is φ, then, for example, coordinate axis conversion processing is performed as shown in FIG. 7.

[0129] That is, in the example shown in FIG. 7, the coordinate axis transformation process includes coordinate axis rotation, which rotates the coordinate axes by a rotation angle φ, and a process of shifting the origin of the coordinate axes from the position of the reference viewpoint to the origin position of the common absolute coordinate system; more specifically, a process of shifting the position of the object in accordance with the positional relationship between the reference viewpoint and the origin of the common absolute coordinate system.

[0130] 7, position P21 indicates the position of the reference viewpoint, and arrow Q11 indicates the direction of the listener's face indicated by the listener orientation information at the reference viewpoint. In particular, the X and Y coordinates of position P21 in the common absolute coordinate system (XYZ coordinate system) are (Xref, Yref).

[0131] Moreover, position P22 indicates the position of the object when the reference viewpoint is at position P21. Here, the X and Y coordinates of the common absolute coordinate system indicating the object position P22 are (Xobj, Yobj), and the x and y coordinates of the xyz coordinate system with the reference viewpoint as the origin indicating the object position P22 are (xobj, yobj).

[0132] Furthermore, in this example, the angle φ between the X axis of the common absolute coordinate system (XYZ coordinate system) and the x axis of the xyz coordinate system is the rotation angle φ of the coordinate axis transformation determined from the listener orientation information.

[0133] Therefore, for example, the coordinate axis X (X coordinate) and the coordinate axis Y (Y coordinate) after the transformation are as shown in the following equation (2).

[0134]

number

[0135] In formula (2), x and y indicate the x-axis (x-coordinate) and y-axis (y-coordinate) before conversion, i.e., in the xyz coordinate system. Also, the "reference viewpoint X coordinate value" and "reference viewpoint Y coordinate value" in formula (2) indicate the X-coordinate and Y-coordinate indicating the position of the reference viewpoint in the XYZ coordinate system (common absolute coordinate system), i.e., the X-coordinate and Y-coordinate constituting the reference viewpoint position information.

[0136] Therefore, in the example of FIG. 7, the X coordinate value Xobj and the Y coordinate value Yobj indicating the position of the object after the coordinate axis transformation process can be obtained from equation (2).

[0137] In other words, φ in equation (2) is set to the rotation angle φ calculated from the listener orientation information at position P21, and the X coordinate value Xobj can be obtained by substituting “Xref”, “xobj”, and “yobj” for the “reference viewpoint X coordinate value”, “x”, and “y” in equation (2), respectively.

[0138] In addition, φ in equation (2) is set to the rotation angle φ calculated from the listener orientation information at position P21, and the Y coordinate value Yobj can be obtained by substituting “Yref”, “xobj”, and “yobj” for “reference viewpoint Y coordinate value”, “x”, and “y” in equation (2), respectively.

[0139] Similarly, if two reference viewpoints, A and B, are selected by the viewpoint selection information, the X and Y coordinate values ​​indicating the position of the object after the coordinate axis transformation process for those reference viewpoints will be as shown in the following equation (3).

[0140]

number

[0141] In equation (3), xa and ya indicate the X and Y coordinate values ​​in the XYZ coordinate system after axis transformation (after coordinate axis transformation processing) for reference viewpoint A, and φa indicates the rotation angle of the axis transformation for reference viewpoint A, i.e., the above-mentioned rotation angle φ.

[0142] Therefore, by substituting the x and y coordinates constituting the object absolute coordinate position information at the reference viewpoint A obtained in process PC1 into equation (3), coordinates xa and ya are obtained as the X and Y coordinates indicating the position of the object in the XYZ coordinate system (common absolute coordinate system) at the reference viewpoint A. The absolute coordinates consisting of the thus obtained coordinates xa and coordinate ya and the Z coordinate are the object absolute coordinate position information output from coordinate axis conversion processing unit 47.

[0143] In this example, since only the horizontal rotation angle φ is handled, no coordinate axis conversion is performed on the Z axis (Z coordinate). Therefore, for example, the z coordinate constituting the object absolute coordinate position information obtained in process PC1 can be used as it is as the Z coordinate indicating the position of the object in the common absolute coordinate system.

[0144] As with reference viewpoint A, in equation (3), xb and yb indicate the X and Y coordinate values ​​in the XYZ coordinate system after axis transformation (after coordinate axis transformation processing) for reference viewpoint B, and φb indicates the rotation angle (rotation angle φ) of the axis transformation for reference viewpoint B.

[0145] In the coordinate axis conversion processing unit 47, the above-mentioned coordinate axis conversion process is carried out as process PC2.

[0146] Therefore, for example, when a coordinate axis transformation process is performed for each of the four reference viewpoints shown in Fig. 3, the transformation result shown in Fig. 8 is obtained. Note that in Fig. 8, the same reference symbols are used for parts corresponding to those in Fig. 3, and their explanation will be omitted as appropriate.

[0147] Each circle in Fig. 8 represents one object. Also, in Fig. 8, the upper part shows the position of each object on the polar coordinate system indicated by the object polar coordinate position information, and the lower part shows the position of each object in the common absolute coordinate system.

[0148] In particular, the leftmost part of Figure 8 shows the result of coordinate axis transformation for the reference viewpoint "Origin" at position P11 shown in Figure 3, and the second from the left in Figure 8 shows the result of coordinate axis transformation for the reference viewpoint "Near" at position P12 shown in Figure 3.

[0149] In addition, the third from the left in Figure 8 shows the result of coordinate axis transformation for the reference viewpoint "Far" at position P13 shown in Figure 3, and the rightmost part in Figure 8 shows the result of coordinate axis transformation for the reference viewpoint "Back" at position P14 shown in Figure 3.

[0150] For example, the reference viewpoint "Origin" is an origin viewpoint whose position of the origin of the polar coordinate system is the position of the origin of the common absolute coordinate system, so the position of the object as seen from the origin does not change before and after the transformation. In contrast, for the remaining three reference viewpoints "Near," "Far," and "Back," it can be seen that the object's position has been shifted to the absolute coordinate position as seen from each viewpoint position. In particular, for the reference viewpoint "Back," the listener's face, as indicated by the listener orientation information, was facing backwards, so after the coordinate axis transformation process, the object is positioned behind the reference viewpoint.

[0151] (Processing PC3) In process PC3, the allocation ratio for the interpolation process is calculated based on the positional relationship between the absolute coordinate positions of each of the two reference viewpoints, i.e., the position indicated by the reference viewpoint position information included in the system configuration information, and an arbitrary listening position sandwiched between the positions of the two reference viewpoints.

[0152] That is, the object position calculation unit 48 performs processing to obtain the apportionment ratio (m:n) as processing PC3 based on the listener position information supplied from the listener position information acquisition unit 41 and the reference viewpoint position information included in the system configuration information.

[0153] Here, it is assumed that the reference viewpoint position information indicating the position of the first reference viewpoint A is (x1, y1, z1), the reference viewpoint position information indicating the position of the second reference viewpoint B is (x2, y2, z2), and the listener position information indicating the listening position is (x3, y3, z3).

[0154] In this case, the object position calculation unit 48 calculates the proportional ratio (m:n), that is, the proportional ratio m and n, by performing the calculation of the following equation (4).

[0155]

number

[0156] (Processing PC4) Next, the object position calculation unit 48 performs an interpolation process as process PC4 based on the proportion ratio (m:n) obtained in process PC3 and the object absolute coordinate position information of each object of the two reference viewpoints supplied from the coordinate axis transformation processing unit 47.

[0157] In other words, in process PC4, the allocation ratio (m:n) obtained in process PC3 is applied to the same object corresponding to the two reference viewpoints obtained in process PC2, thereby determining the object position and gain amount corresponding to an arbitrary listening position.

[0158] Here, the absolute coordinate position of a specified object as seen from reference viewpoint A, i.e., the object absolute coordinate position information of reference viewpoint A obtained by processing PC2, is defined as (xa, ya, za), and the gain amount indicated by the gain information of the specified object for reference viewpoint A is defined as g1.

[0159] Similarly, the absolute coordinate position of the above-mentioned specified object as seen from reference viewpoint B, i.e., the object absolute coordinate position information of reference viewpoint B obtained by processing PC2, is defined as (xb, yb, zb), and the gain amount indicated by the gain information of the object for reference viewpoint B is defined as g2.

[0160] Also, the absolute coordinates indicating the position in the XYZ coordinate system (common absolute coordinate system) of the above-mentioned predetermined object corresponding to an arbitrary viewpoint position between reference viewpoint A and reference viewpoint B, i.e., the listening position indicated by the listener position information, and the gain amount are (xc, yc, zc) and gain_c, respectively. The absolute coordinates (xc, yc, zc) are the final object absolute coordinate position information output from object position calculation unit 48 to polar coordinate conversion unit 49.

[0161] At this time, the final object absolute coordinate position information (xc, yc, zc) and the gain amount gain_c for the predetermined object can be obtained by calculating the following equation (5) using the proportional division ratio (m:n).

[0162]

number

[0163] The positional relationship between reference viewpoint A, reference viewpoint B, and the listening position described above, and the positional relationship between the same object at each of reference viewpoint A, reference viewpoint B, and the listening position, are as shown in FIG.

[0164] 9, the horizontal axis and the vertical axis respectively represent the X-axis and the Y-axis of the XYZ coordinate system (common absolute coordinate system). Note that, for the sake of simplicity, only the X-axis and the Y-axis directions are shown.

[0165] In this example, position P51 is a position indicated by reference viewpoint position information (x1, y1, z1) of reference viewpoint A, and position P52 is a position indicated by reference viewpoint position information (x2, y2, z2) of reference viewpoint B.

[0166] Moreover, a position P53 between reference viewpoint A and reference viewpoint B is a listening position indicated by listener position information (x3, y3, z3).

[0167] In the above-mentioned equation (4), the proportion ratio (m:n) is found based on the positional relationship between the reference viewpoint A, the reference viewpoint B, and the listening position.

[0168] Furthermore, position P61 is a position indicated by absolute object coordinate position information (xa, ya, za) at reference viewpoint A, and position P62 is a position indicated by absolute object coordinate position information (xb, yb, zb) at reference viewpoint B.

[0169] Furthermore, a position P63 located between positions P61 and P62 is a position indicated by the object absolute coordinate position information (xc, yc, zc) at the listening position.

[0170] In this way, by performing the calculation of equation (5), that is, the interpolation process, it is possible to obtain absolute object coordinate position information indicating an appropriate object position for any listening position.

[0171] In the above, an example has been described in which the apportionment ratio (m:n) is used to determine the object position, i.e., the final object absolute coordinate position information. However, the present invention is not limited to this example, and the final object absolute coordinate position information may be estimated using machine learning or the like.

[0172] Furthermore, when an editor in an absolute coordinate system is used, i.e., in the case of the content playback system shown in Fig. 2, the position of each object at each reference viewpoint, i.e., the position indicated by the object absolute coordinate position information, is a position on one common absolute coordinate system. In other words, the position of an object at each reference viewpoint is expressed by the absolute coordinates of the common absolute coordinate system.

[0173] 2, the object absolute coordinate position information obtained by decoding in the decoding unit 45 may be input to the above-mentioned processing PC3. That is, the calculation of formula (4) may be performed based on the object absolute coordinate position information obtained by decoding.

[0174] <Content playback system operation> Next, with reference to FIG. 10, a process flow (sequence) performed in the content reproduction system described above will be described.

[0175] Here, an example will be described in which the reference viewpoint is selected on the server 11 side and the object polar coordinate encoded data is prepared in advance on the server 11 side. That is, an example will be described in which the viewpoint selection unit 42 is provided on the server 11 side in the example of the content reproduction system shown in FIG.

[0176] First, on the server 11 side, for all reference viewpoints, polar coordinate system object position information, i.e., object polar coordinate encoded data, is generated and stored by a polar coordinate system editor, and system configuration information is also generated and stored.

[0177] Then, the configuration information sending unit 21 transmits the system configuration information to the client 12 via a network or the like.

[0178] Then, the configuration information acquisition unit 43 of the client 12 receives the system configuration information transmitted from the server 11 and supplies it to the coordinate axis conversion processing unit 47. At this time, the client 12 decodes the received system configuration information and initializes the client system.

[0179] Next, the listener position information acquisition unit 41 acquires the listener position information and supplies it to the configuration information acquisition unit 43, and the configuration information acquisition unit 43 transmits the listener position information supplied from the listener position information acquisition unit 41 to the server 11.

[0180] Furthermore, the configuration information transmitter 21 receives listener position information transmitted from the client 12, and supplies it to the viewpoint selection unit 42. Then, based on the listener position information and system configuration information supplied from the configuration information transmitter 21, the viewpoint selection unit 42 selects reference viewpoints required for the interpolation process, i.e., for example, two reference viewpoints sandwiching the above-mentioned listening position, and supplies viewpoint selection information indicating the selection result to the encoded data transmitter 22.

[0181] The encoded data sending unit 22 prepares for transmission of polar coordinate system object position information of the reference viewpoint required for the interpolation process, in accordance with the viewpoint selection information supplied from the viewpoint selection unit 42 .

[0182] That is, the coded data sending unit 22 generates a bit stream by reading and multiplexing the object polar coordinate coded data of the reference viewpoint indicated by the viewpoint selection information and the coding gain information. Then, the coded data sending unit 22 transmits the generated bit stream to the client 12.

[0183] The encoded data acquisition unit 44 receives and demultiplexes the bit stream transmitted from the server 11 , and supplies the resulting object polar coordinate encoded data and encoding gain information to a decoding unit 45 .

[0184] The decoding unit 45 decodes the object polar coordinate encoded data supplied from the encoded data acquisition unit 44, and supplies the object polar coordinate position information obtained as a result to the coordinate conversion unit 46. The decoding unit 45 also decodes the encoding gain information supplied from the encoded data acquisition unit 44, and supplies the gain information obtained as a result to the object position calculation unit 48 via the coordinate conversion unit 46 and the coordinate axis conversion processing unit 47.

[0185] The coordinate conversion unit 46 converts the object polar coordinate position information supplied from the decoding unit 45 into listener-centered absolute coordinate position information.

[0186] That is, for example, the coordinate conversion unit 46 calculates the above-mentioned formula (1) based on the object polar coordinate position information, and supplies the object absolute coordinate position information obtained as a result to the coordinate axis conversion processing unit 47.

[0187] Next, the coordinate axis conversion processing unit 47 performs coordinate axis conversion to expand the listener-centered absolute coordinate position information into a common absolute coordinate space.

[0188] For example, the coordinate axis transformation processing unit 47 performs coordinate axis transformation processing by calculating the above-mentioned equation (3) based on the system configuration information supplied from the configuration information acquisition unit 43 and the object absolute coordinate position information supplied from the coordinate transformation unit 46, and supplies the object absolute coordinate position information obtained as a result to the object position calculation unit 48.

[0189] The object position calculation unit 48 calculates a proportion ratio for the interpolation process from the current listener position and the reference viewpoint.

[0190] For example, the object position calculation unit 48 calculates the above-mentioned equation (4) based on the listener position information supplied from the listener position information acquisition unit 41 and the reference viewpoint position information of the multiple reference viewpoints selected by the viewpoint selection unit 42, and calculates the apportionment ratio (m:n).

[0191] Furthermore, the object position calculation unit 48 calculates the object position and the gain amount corresponding to the current listener position by using a proportional ratio from the object position and the gain amount corresponding to the reference viewpoints sandwiching the listener position.

[0192] For example, the object position calculation unit 48 performs an interpolation process by calculating the above-mentioned equation (5) based on the object absolute coordinate position information and gain information supplied from the coordinate axis transformation processing unit 47 and the proportion ratio (m:n), and supplies the final object absolute coordinate position information and gain information obtained as a result to the polar coordinate transformation unit 49.

[0193] Thereafter, the client 12 performs a rendering process to which the calculated object position and gain amount are applied.

[0194] For example, the polar coordinate conversion unit 49 converts absolute coordinate position information into polar coordinates.

[0195] That is, for example, the polar coordinate conversion unit 49 performs polar coordinate conversion on the object absolute coordinate position information supplied from the object position calculation unit 48 based on the listener position information supplied from the listener position information acquisition unit 41 .

[0196] The polar coordinate conversion unit 49 supplies the polar coordinate position information obtained by the polar coordinate conversion and the gain information supplied from the object position calculation unit 48 to a rendering processing unit at a subsequent stage.

[0197] The rendering processor then performs polar coordinate rendering on all objects.

[0198] That is, the rendering processing unit performs rendering processing, for example in a polar coordinate system defined in MPEG-H, based on the polar coordinate position information and gain information of all objects supplied from the polar coordinate conversion unit 49, and generates playback audio data for playing back the sound of the content.

[0199] Here, for example, VBAP (Vector Based Amplitude Panning) is performed as a rendering process in a polar coordinate system defined by MPEG-H. More specifically, before the rendering process, gain adjustment based on gain information is performed on the audio data, but the gain adjustment may be performed by the polar coordinate conversion unit 49 at the previous stage, instead of the rendering processing unit.

[0200] When the above process is performed for a predetermined frame and playback audio data is generated, content is played back based on the playback audio data as appropriate. After that, listener position information is transmitted from the client 12 to the server 11 as appropriate, and the above process is repeated.

[0201] In this manner, the content playback system calculates absolute object coordinate position information and gain information for any listening position by interpolation processing from object position information from a plurality of reference viewpoints. In this manner, it is possible to realize object arrangement based on the intention of the content creator according to the listening position, rather than simply the physical relationship between the listener and the object. This allows content playback based on the intention of the content creator to fully convey the entertainment value of the content to the listener.

[0202] About listeners and objects Incidentally, there are two possible examples of the reference viewpoint: one that assumes the viewpoint of a listener, and one that assumes the viewpoint of a performer imagining becoming the object.

[0203] In the latter case, the listener and the object overlap in the reference viewpoint, that is, the listener and the object are in the same position, so the following cases CA1 to CA3 are possible.

[0204] (Case CA1) Prevent listeners from overlapping objects or from entering a certain range (Case CA2) The listener becomes one with the object, and the sound generated by the object is output from all channels. (Case CA3) Sounds emanating from overlapping objects are muted or attenuated

[0205] For example, in case CA2, it is possible to recreate the sensation of being localized inside the listener's head.

[0206] In case CA3, the sound of the object can be muted or attenuated, allowing the listener to become the performer, for example in a karaoke mode. In this case, the listener can get the feeling that they are singing in a surrounding environment, surrounded by other sounds besides the performer's singing voice.

[0207] If the content creator has such an intention, the content creator can store an identifier indicating these cases CA1 to CA3 in the encoded bit stream transmitted from the server 11 and transmit it to the client 12. For example, such an identifier is information indicating the above-mentioned playback mode.

[0208] Furthermore, in the content reproduction system described above, the listener may move around between the two reference viewpoints.

[0209] In such a case, some listeners may want to intentionally move the object (viewpoint) closer to one of the two reference viewpoints. Specifically, for example, the listener may want to maintain an angle that makes it easy to see an artist that the listener likes.

[0210] Therefore, for example, the degree of convergence may be controlled by applying a bias to the proportional division process of the internal division ratio. This can be realized by introducing a new bias coefficient α into the above-mentioned equation (5) for calculating the interpolation, as shown in FIG.

[0211] 11 shows the characteristics when the bias coefficient α is applied. In particular, the upper part of the figure shows an example in which an object is moved closer to the viewpoint X1 side, that is, the above-mentioned reference viewpoint A side.

[0212] In contrast to this, the lower part of the figure shows an example in which the objects are moved closer to the viewpoint X2 side, that is, the arrangement on the reference viewpoint B side described above.

[0213] 11, the horizontal axis indicates the position of the predetermined viewpoint X3 when the bias coefficient α is not introduced, and the vertical axis indicates the position of the predetermined viewpoint X3 when the bias coefficient α is introduced. Here, the position of the reference viewpoint A (viewpoint X1) is set to "0", and the position of the reference viewpoint B (viewpoint X2) is set to "1".

[0214] In the upper example of the figure, for example, when a listener moves from reference viewpoint A (viewpoint X1) to the position of reference viewpoint B (viewpoint X2), the smaller the bias coefficient α, the more the listener feels that it is difficult to reach the position of reference viewpoint B (viewpoint X2).

[0215] Conversely, in the example at the bottom of the figure, when a listener moves from reference viewpoint A to the position of reference viewpoint B, the smaller the bias coefficient α, the more quickly the listener will feel as if they have reached the position of reference viewpoint B.

[0216] For example, when an object is moved closer to the position on the reference viewpoint A side, the final object absolute coordinate position information (xc, yc, zc) and the gain amount gain_c can be obtained by calculating the following equation (6).

[0217] On the other hand, when the object is moved closer to the arrangement on the reference viewpoint B side, the final object absolute coordinate position information (xc, yc, zc) and the gain amount gain_c can be obtained by calculating the following equation (7).

[0218] In the equations (6) and (7), m and n in the proportion ratio (m:n) and the bias coefficient α are as shown in the following equation (8).

[0219]

number

[0220]

number

[0221]

number

[0222] In equation (8), the reference viewpoint position information (x1, y1, z1), the reference viewpoint position information (x2, y2, z2), and the listener position information (x3, y3, z3) are the same as in equation (4) above.

[0223] Obtaining the final object absolute coordinate position information and the gain amount using the bias coefficient α as in equation (6) or (7) means performing an interpolation process on the object absolute coordinate position information and the gain information of a specified reference viewpoint by applying a weighting factor called the bias coefficient α to obtain the final object absolute coordinate position information and the gain amount.

[0224] The object position information in absolute coordinates after the interpolation process obtained in this way, i.e., the object absolute coordinate position information, is converted into polar coordinate information (polar coordinate position information) by combining it with the listener position information, making it possible to perform the polar coordinate rendering process used in the existing MPEG-H at a later stage.

[0225] Interpolation of absolute object coordinate position information and gain information In the above, two-point interpolation using information from two reference viewpoints has been described as an example in which object position calculation unit 48 calculates absolute object coordinate position information and gain information at an arbitrary viewpoint position, that is, the listening position, by interpolation processing.

[0226] However, without being limited thereto, three-point interpolation using information from three reference viewpoints may be performed to obtain absolute object coordinate position information and gain information at any listening position. Also, absolute object coordinate position information and gain information at any listening position may be obtained using information from four or more reference viewpoints. A specific example of performing three-point interpolation will be described below.

[0227] For example, as shown on the left side of FIG. 12, it is considered that absolute object coordinate position information at an arbitrary listening position F is obtained by interpolation processing.

[0228] In this example, there are three reference viewpoints A, B, and C surrounding listening position F, and it is assumed here that information from these reference viewpoints A to C is used to perform the interpolation process.

[0229] In the following, the X and Y coordinates of the listening position F in the common absolute coordinate system, i.e., the XYZ coordinate system, are (x f ,y f )

[0230] Similarly, the X and Y coordinates of the positions of reference viewpoint A, reference viewpoint B, and reference viewpoint C are (x a ,y a ), (x b ,y b ), and (x c ,y c )

[0231] In this case, object position F' at listening position F is determined based on the coordinates of object position A', object position B', and object position C' corresponding to reference viewpoint A, reference viewpoint B, and reference viewpoint C, respectively, as shown on the right side of Figure 12.

[0232] Here, for example, object position A' indicates the position of the object when the viewpoint is at reference viewpoint A, that is, the position of the object in the common absolute coordinate system indicated by the object absolute coordinate position information of reference viewpoint A.

[0233] Moreover, object position F′ indicates the position of the object in the common absolute coordinate system when the listener is at listening position F, that is, the position indicated by the object absolute coordinate position information output from object position calculation section 48 .

[0234] In the following, the X and Y coordinates of object position A', object position B', and object position C' are (x a ',y a '), (x b ',y b '), and (x c ',y c '), and the X and Y coordinates of object position F' are (x f ',y f ').

[0235] In the following, a triangular area surrounded by any three reference viewpoints such as reference viewpoint A to reference viewpoint C, that is, a triangular area formed by three reference viewpoints, will also be referred to as a triangular mesh.

[0236] Since a plurality of reference viewpoints exist in the common absolute coordinate space, a plurality of triangular meshes having the reference viewpoints as vertices can be formed in the common absolute coordinate space.

[0237] Similarly, in the following, a triangular area surrounded (formed) by object positions indicated by the object absolute coordinate position information of any three reference viewpoints, such as object position A' to object position C', will also be referred to as a triangular mesh.

[0238] For example, in the case of two-point interpolation, the listener can move to any position on the line segment connecting the two reference viewpoints and listen to the sound of the content.

[0239] In contrast, when three-point interpolation is used, the listener can move to any position within the triangular mesh area surrounded by the three reference viewpoints and listen to the sound of the content. In other words, the listener can cover the area other than the line segment connecting the two reference viewpoints in the case of two-point interpolation as a listening position.

[0240] As in the case of two-point interpolation, when performing three-point interpolation, the coordinates indicating any position in the common absolute coordinate system (XYZ coordinate system) can be obtained from the coordinates of the arbitrary position in the xyz coordinate system, the listener direction information, and the reference viewpoint position information using the above formula (2).

[0241] Note that, here, the Z coordinate value in the XYZ coordinate system is assumed to be the same as the z coordinate value in the xyz coordinate system. However, if the Z coordinate value and the z coordinate value are different, the Z coordinate value indicating any position can be calculated by adding the Z coordinate value indicating the position of the reference viewpoint in the XYZ coordinate system to the z coordinate value of the arbitrary position.

[0242] Ceva's theorem proves that any listening position within a triangular mesh formed by three reference viewpoints can be uniquely determined as the intersection of line segments from each of the three vertices of the triangular mesh to each of the internal division points of the three sides that are not adjacent to those vertices, if the internal division ratios of each side of the triangular mesh are appropriately determined.

[0243] From the proof formula, this holds true for all triangular meshes, regardless of the shape of the triangular mesh, once the internal division ratio of the three sides of the triangular mesh is determined.

[0244] Therefore, by calculating the internal division ratio of the triangular mesh including the listening position for the viewpoint side, i.e., the reference viewpoint, and applying that internal division ratio to the triangular mesh on the object side, i.e., the object position, an appropriate object position for any listening position can be obtained.

[0245] In the following, an example will be described in which such a property of the internal division ratio is used to obtain absolute object coordinate position information indicating the position of an object when the listener is at an arbitrary listening position.

[0246] In this case, first, the internal division ratio of the sides of the triangular mesh of the reference viewpoint on the XY plane of the XYZ coordinate system, which is a two-dimensional space, is calculated.

[0247] Next, the above-mentioned internal division ratio is applied to the triangular mesh of the object position corresponding to the three reference viewpoints on the XY plane, and the X and Y coordinates of the object position corresponding to the listening position on the XY plane are obtained.

[0248] Furthermore, the Z coordinate of the object corresponding to the listening position is determined based on a three-dimensional plane containing the positions of the three objects corresponding to the three reference viewpoints in three-dimensional space (XYZ coordinate system) and the X and Y coordinates of the object at the listening position on the XY plane.

[0249] Here, with reference to Figs. 13 to 15, an example will be described in which, for the listening position F shown in Fig. 12, object absolute coordinate position information and gain information indicating object position F' are obtained by interpolation processing.

[0250] For example, as shown in FIG. 13, first, the X and Y coordinates of an internal division point in a triangular mesh consisting of reference viewpoints A to C, including listening position F, are found.

[0251] Let the intersection point of the line passing through the listening position F and the reference viewpoint C and the line segment AB from the reference viewpoint A to the reference viewpoint B be point D. Let the coordinates of the position of point D on the XY plane be (x d ,y d ) In other words, point D is an internal division point on line segment AB (side AB).

[0252] In this case, the relationship shown in the following equation (9) holds for the X and Y coordinates indicating the position of an arbitrary point on the line segment CF from the reference viewpoint C to the listening position F, and for the X and Y coordinates indicating the position of an arbitrary point on the line segment AB.

[0253]

number

[0254] In addition, point D is the intersection point of the line passing through reference viewpoint C and listening position F and line segment AB. Therefore, from equation (9), the coordinates of point D on the XY plane (x d ,y d ) and its coordinates (x d ,y d ) is as shown in the following equation (10).

[0255]

number

[0256] Therefore, as shown in the following equation (11), the coordinates of point D (x d ,y d ), the coordinates of reference viewpoint A (x a ,y a ), and the coordinates of reference viewpoint B (x b ,y b ), we can obtain the internal division ratio (m,n) of the line segment AB at point D, that is, the division ratio.

[0257]

number

[0258] Similarly, the intersection point of the line passing through the listening position F and the reference viewpoint B with the line segment AC from the reference viewpoint A to the reference viewpoint C is defined as point E, and the coordinates indicating the position of point E on the XY plane are (x e ,y e ) In other words, point E is an internal division point on line segment AC (side AC).

[0259] In this case, the relationship shown in the following equation (12) holds for the X and Y coordinates indicating the position of an arbitrary point on the line segment BF from the reference viewpoint B to the listening position F, and for the X and Y coordinates indicating the position of an arbitrary point on the line segment AC.

[0260]

number

[0261] In addition, point E is the intersection point of the line passing through reference viewpoint B and listening position F with line segment AC. Therefore, from equation (12), the coordinates of point E on the XY plane (x e ,y e ) and its coordinates (x e ,y e ) is as shown in the following equation (13).

[0262]

number

[0263] Therefore, as shown in the following equation (14), the coordinates of point E (x e ,y e ), the coordinates of reference viewpoint A (x a ,y a ), and the coordinates of the reference viewpoint C (x c ,y c ), we can obtain the internal division ratio (k,l) of the line segment AC at point E, that is, the division ratio.

[0264]

number

[0265] Next, the ratio of the two sides thus obtained, that is, the internal division ratio (m,n) and the internal division ratio (k,l), are applied to the triangular mesh on the object side as shown in FIG. 14 to obtain the coordinates (x f ',y f ') is required.

[0266] Specifically, in this example, the point on the line segment A'B' connecting object position A' and object position B', which corresponds to point D, is set as point D'.

[0267] Similarly, a point on a line segment A'C' connecting object position A' and object position C', which corresponds to point E, is set as point E'.

[0268] Moreover, the intersection point of a line passing through object position C' and point D' and a line passing through object position B' and point E' is object position F' corresponding to listening position F.

[0269] Here, let us assume that the internal division ratio of the line segment A'B' at point D' is the same as that at point D (m, n). In this case, the coordinates of point D' on the XY plane (x d ',y d As shown in the following equation (15), the internal division ratio (m, n) and the coordinates of the object position A' (x a ',y a '), and the coordinates of object position B' (x b ',y b ') can be obtained.

[0270]

number

[0271] In addition, the internal division ratio of the line segment A'C' at point E' is the same as that at point E (k, l). In this case, the coordinates of point E' on the XY plane (x e ',ye As shown in the following equation (16), the internal division ratio (k, l) and the coordinates of the object position A' (x a ',y a '), and the coordinates of object position C' (x c ',y c ') can be obtained.

[0272]

number

[0273] Therefore, the relationship shown in the following equation (17) holds for the X and Y coordinates indicating the position of an arbitrary point on the line segment B'E' from object position B' to point E', and for the X and Y coordinates indicating the position of an arbitrary point on the line segment C'D' from object position C' to point D'.

[0274]

number

[0275] The target object position F' is the intersection of the line segments B'E' and C'D'. Therefore, from the relationship in equation (17), the coordinates of the object position F' (x f ',y f ') can be obtained.

[0276]

number

[0277] By the above process, the coordinates of the object position F' on the XY plane (x f ',y f ') is obtained.

[0278] Next, the coordinates of the object position F' on the XY plane (x f ',y f ') and the coordinates of object position A' in the XYZ coordinate system (x a ',ya ',z a '), coordinates of object position B' (x b ',y b ',z b '), and the coordinates of object position C' (x c ',y c ',z c Based on this, the coordinates (x f ',y f ',z f That is, the Z coordinate z of the object position F' in the XYZ coordinate system is calculated. f ' is required.

[0279] For example, a triangle in three-dimensional space with vertices of object position A', object position B', and object position C' in the XYZ coordinate system (common absolute coordinate space) is obtained, that is, a three-dimensional plane A'B'C' including object position A', object position B', and object position C'. Then, the X and Y coordinates on the three-dimensional plane A'B'C' are calculated as follows: f ',y f '), and the Z coordinate of that point is z f 'It is said.

[0280] Specifically, the vector that starts at object position A' in the XYZ coordinate system and ends at object position B' is called vector A'B'=(x ab ',y ab ',z ab ').

[0281] Similarly, the vector that starts at object position A' in the XYZ coordinate system and ends at object position C' is called vector A'C'=(x ac ',y ac ',z ac ').

[0282] These vectors A'B' and A'C' are the coordinates of the object position A' (x a ',y a ',z a'), coordinates of object position B' (x b ',y b ',z b '), and the coordinates of object position C' (x c ',y c ',z c That is, vector A'B' and vector A'C' can be obtained by the following equation (19).

[0283]

number

[0284] Moreover, the normal vector (s, t, u) of the three-dimensional plane A'B'C' is the cross product of vector A'B' and vector A'C', and can be obtained by the following equation (20).

[0285]

number

[0286] Therefore, the normal vector (s,t,u) and the coordinates of the object position A' (x a ',y a ',z a From these, the plane equation for the three-dimensional plane A'B'C' is given by the following equation (21).

[0287]

number

[0288] Here, the X coordinate x of the object position F' on the 3D plane A'B'C' is f ' and the Y coordinate y f Since ' has already been found, we can use the X coordinate x in the X and Y parts of the plane equation (21). f ' and the Y coordinate y f By substituting ', the Z coordinate z is calculated as shown in the following equation (22). f ' can be asked for.

[0289]

number

[0290] From the above calculations, the coordinates of the desired object position F' (x f ',y f ',z f The object position calculation unit 48 calculates the coordinates (x f ',y f ',z f ') is output as absolute object coordinate position information.

[0291] Moreover, similarly to the case of the object absolute coordinate position information, the gain information can also be obtained by three-point interpolation.

[0292] That is, the gain information of the object at object position F′ can be obtained by performing an interpolation process based on the gain information of the object when the viewpoint is at each of reference viewpoints A to C.

[0293] For example, as shown in FIG. 15, the gain information G of an object at object position F' in a triangular mesh formed by object positions A', B', and C' is f Think about asking for '.

[0294] Now, when the viewpoint is at the reference viewpoint A, the gain information of the object at the object position A' is G a ', and the gain information of the object at object position B' is G b ', and the gain information of the object at object position C' is G c '.

[0295] In this case, first, the gain information G of the object at point D', which is the internal division point of the line segment A'B' when the viewpoint is at point D. d' is required.

[0296] Specifically, the gain information G d ' is the internal division ratio (m,n) of the above-mentioned line segment A'B' and the gain information G a ' and gain information G for object position B' b Based on these, it can be obtained by calculating the following equation (23).

[0297]

number

[0298] That is, in equation (23), the gain information G a ' and gain information G b The gain information G of point D' is obtained by the interpolation process based on d ' is required.

[0299] Next, the internal division ratio (o,p) of the line segment C'D' from the object position C' to the point D' by the object position F' and the gain information G c Gain information G at point ' and point D' d By performing the interpolation process based on f That is, the gain information G f ' is required.

[0300]

number

[0301] The object position calculation unit 48 outputs the gain information G f ' is output as the gain information of the object corresponding to the listening position F.

[0302] By performing three-point interpolation as described above, absolute object coordinate position information and gain information can be obtained for any listening position.

[0303] Incidentally, when performing three-point interpolation, if there are four or more reference viewpoints in a common absolute coordinate space, multiple triangular meshes can be constructed by combining three of those reference viewpoints.

[0304] For example, assume that there are five reference viewpoints at positions P91 to P95 as shown on the left side of FIG.

[0305] In such a case, a plurality of triangular meshes such as triangular mesh MS11 to triangular mesh MS13 will be formed (configured).

[0306] Here, triangular mesh MS11 is formed by positions P91 to P93 which are reference viewpoints, triangular mesh MS12 is formed by positions P92, P93, and P95, and triangular mesh MS13 is formed by positions P93, P94, and P95.

[0307] The listener can freely move within the area surrounded by these triangular meshes MS11 to MS13, that is, the area surrounded by all the reference viewpoints.

[0308] Therefore, as the listener moves, that is, as the listening position moves (changes), the triangular mesh for obtaining the object absolute coordinate position information and gain information at the listening position is switched.

[0309] In the following description, the triangular mesh on the viewpoint side for obtaining the object absolute coordinate position information and gain information at the listening position will also be referred to as the selected triangular mesh. Also, the triangular mesh on the object side corresponding to the selected triangular mesh on the viewpoint side will also be referred to as the selected triangular mesh, as appropriate.

[0310] The left side of Fig. 16 shows an example in which the listening position, which was originally at position P96, has subsequently moved to position P96'. That is, position P96 is the position of the listener's viewpoint (listening position) before the movement, and position P96' is the position of the listener's viewpoint after the movement.

[0311] When selecting a triangular mesh for three-point interpolation, the sum (sum) of the distances from the listening position to each vertex of the triangular mesh is basically calculated as the total distance, and the triangular mesh containing the listening position with the smallest total distance is selected as the selected triangular mesh.

[0312] That is, the selected triangular mesh is basically determined by a condition process that selects the one with the smallest total distance from among the triangular meshes that include the listening position. Hereinafter, the condition that the total distance is the smallest among the triangular meshes that include the listening position is also referred to as the viewpoint side selection condition.

[0313] When performing three-point interpolation, basically, a triangular mesh that satisfies such viewpoint-side selection conditions is selected as the selected triangular mesh.

[0314] Therefore, in the example shown on the left side of Figure 16, when the listening position is at position P96, triangular mesh MS11 is selected as the selected triangular mesh, and when the listening position moves to position P96', triangular mesh MS13 is selected as the selected triangular mesh.

[0315] However, if the selected triangular mesh is simply selected based on the minimum total distance, a discontinuous transition in object position, that is, a jump in object position, may occur.

[0316] For example, as shown in the center of FIG. 16, suppose there are triangular meshes MS21 to MS23 as triangular meshes on the object side, that is, triangular meshes consisting of object positions corresponding to the respective reference viewpoints.

[0317] In this example, the triangular meshes MS21 and MS22 are adjacent to each other, and the triangular meshes MS22 and MS23 are also adjacent to each other.

[0318] That is, the triangular meshes MS21 and MS22 have a common side, and the triangular meshes MS22 and MS23 also have a common side. Hereinafter, the common side between two adjacent triangular meshes will also be referred to as a common side.

[0319] On the other hand, the triangular meshes MS21 and MS23 are not adjacent to each other, and therefore the two triangular meshes do not have a common side.

[0320] Here, it is assumed that triangular mesh MS21 is an object-side triangular mesh corresponding to triangular mesh MS11 on the viewpoint side. In other words, it is assumed that triangular mesh MS21 is a triangular mesh whose vertices are the object positions of the same object when the viewpoint (listening position) is at each of positions P91 to P93, which are reference viewpoints.

[0321] Similarly, triangular mesh MS22 is an object-side triangular mesh corresponding to triangular mesh MS12 on the viewpoint side, and triangular mesh MS23 is an object-side triangular mesh corresponding to triangular mesh MS13 on the viewpoint side.

[0322] For example, if the listening position moves from position P96 to position P96', and the selected triangular mesh on the viewpoint side changes from triangular mesh MS11 to triangular mesh MS13, the selected triangular mesh on the object side changes from triangular mesh MS21 to triangular mesh MS23.

[0323] In the central example of the figure, position P101 indicates the object position obtained by performing three-point interpolation with triangular mesh MS21 as the selected triangular mesh when the listening position is at position P96. Similarly, position P101' indicates the object position obtained by performing three-point interpolation with triangular mesh MS23 as the selected triangular mesh when the listening position is at position P96'.

[0324] Thus, in this example, when the listening position moves from position P96 to position P96', the object position moves from position P101 to position P101'.

[0325] In this case, however, the triangular mesh MS21 including the position P101 and the triangular mesh MS23 including the position P101' are not adjacent to each other and do not share a common side. In other words, the object position moves (transitions) across the triangular mesh MS22 that is between those triangular meshes.

[0326] In such a case, therefore, discontinuous movement (transition) of the object position occurs because the triangular meshes MS21 and MS23 do not have a common side, and therefore the scale of the relationship of the object positions corresponding to each reference viewpoint is different between these triangular meshes.

[0327] In contrast, if the selected triangular meshes on the object side have common edges before and after a movement of the listening position, scale continuity is maintained between the selected triangular meshes before and after the movement, and discontinuous transitions in object position can be prevented.

[0328] Therefore, when three-point interpolation is performed, in addition to the basic conditional processing described above, a conditional processing is also added in which the selected triangular mesh on the viewpoint side after the movement is selected so that the selected triangular meshes on the object side have common edges before and after the movement of the listening position.

[0329] In other words, the selected triangular mesh to be used for three-point interpolation at the viewpoint after the movement can be selected based on the relationship between the selected triangular mesh on the object side used for three-point interpolation at the viewpoint (listening position) before the movement and the triangular mesh on the object side corresponding to the triangular mesh on the viewpoint side that includes the viewpoint position (listening position) after the movement.

[0330] Hereinafter, the condition that a triangular mesh on the object side before the movement of the listening position and a triangular mesh on the object side after the movement of the listening position have a common side will also be referred to as an object-side selection condition.

[0331] When performing three-point interpolation, it is sufficient to select as the selected triangular mesh the one that also satisfies the viewpoint selection conditions from among the viewpoint-side triangular meshes that satisfy the object-side selection conditions. However, if there is no viewpoint-side triangular mesh that satisfies the object-side selection conditions, the one that only satisfies the viewpoint-side selection conditions is selected as the selected triangular mesh.

[0332] In this way, by selecting the selected triangular mesh on the viewpoint side so as to satisfy not only the selection conditions on the viewpoint side but also the selection conditions on the object side, discontinuous movement of the object position can be suppressed, thereby achieving higher quality sound reproduction.

[0333] In this case, for example, in the example shown on the left side of Figure 16, when the listening position moves from position P96 to position P96', triangular mesh MS12 is selected as the selected triangular mesh on the viewpoint side for the listening position after the movement, position P96'.

[0334] For example, as shown on the right side of Figure 16, a triangular mesh MS21 on the object side corresponding to a triangular mesh MS11 on the viewpoint side before movement and a triangular mesh MS22 on the object side corresponding to a triangular mesh MS12 on the viewpoint side after movement share a common side, so it can be seen that in this case the selection condition on the object side is satisfied.

[0335] Moreover, position P101'' indicates the object position when the listening position is at position P96', which is obtained by performing three-point interpolation with triangular mesh MS22 as the selected triangular mesh on the object side.

[0336] Therefore, in this example, when the listening position moves from position P96 to position P96', the position of the object corresponding to the listening position also moves from position P101 to position P101''.

[0337] In this case, since the triangular meshes MS21 and MS22 have a common side, no discontinuous movement of the object position occurs before and after the movement of the listening position.

[0338] For example, in this example, the positions of both ends of the common side of triangular meshes MS21 and MS22, that is, the object position corresponding to position P92, which is the reference viewpoint, and the object position corresponding to position P93, which is the reference viewpoint, are the same before and after the listening position is moved.

[0339] As described above, even if the listening position is the same position P96' in the example shown in Figure 16, the object position, i.e., the position onto which the object is projected, differs depending on whether triangular mesh MS12 or triangular mesh MS13 is selected as the selected triangular mesh on the viewpoint side.

[0340] Therefore, by selecting a more appropriate triangular mesh from among the triangular meshes that include the listening position, it is possible to suppress discontinuous movement of the object position, i.e., the sound image position, and achieve higher quality sound reproduction.

[0341] In addition, by combining three-point interpolation using a triangular mesh consisting of three reference viewpoints surrounding the listening position with the selection of a triangular mesh based on selection conditions, it is possible to realize object placement that takes into account the reference viewpoints for any listening position within the common absolute coordinate space.

[0342] In addition, even when three-point interpolation is performed, similarly to the case where two-point interpolation is performed, an interpolation process weighted based on the bias coefficient α may be performed as appropriate to obtain the final object absolute coordinate position information and gain information.

[0343] Example of a Content Playback System Configuration Here, a more detailed embodiment of the content reproduction system to which the present technology described above is applied will be described.

[0344] Fig. 17 is a diagram showing a configuration example of a content playback system to which the present technology is applied. Note that in Fig. 17, parts corresponding to those in Fig. 1 are given the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0345] The content reproduction system shown in FIG. 17 includes a server 11 that distributes content, and a client 12 that receives the content distributed from the server 11.

[0346] Server 11 also includes configuration information recording unit 101 , configuration information sending unit 21 , recording unit 102 , and encoded data sending unit 22 .

[0347] 4 prepared in advance, and supplies the recorded system configuration information to configuration information sending unit 21. Note that a part of recording unit 102 may be configured as configuration information recording unit 101.

[0348] The recording unit 102 records, for example, encoded audio data obtained by encoding audio data of an object, object polar coordinate encoded data of each object for each reference viewpoint, encoding gain information, and the like that make up the content.

[0349] The recording unit 102 supplies the recorded encoded audio data, the object polar coordinate encoded data, the encoding gain information, and the like to the encoded data sending unit 22 in response to a request or the like.

[0350] The client 12 also includes a listener position information acquisition unit 41 , a viewpoint selection unit 42 , a communication unit 111 , a decoding unit 45 , a position calculation unit 112 , and a rendering processing unit 113 .

[0351] The communication unit 111 corresponds to the configuration information acquisition unit 43 and the encoded data acquisition unit 44 shown in FIG. 1, and communicates with the server 11 to transmit and receive various data.

[0352] For example, the communication unit 111 transmits viewpoint selection information supplied from the viewpoint selection unit 42 to the server 11, and receives system configuration information and a bit stream transmitted from the server 11. That is, the communication unit 111 functions as a reference viewpoint information acquisition unit that acquires from the server 11 the system configuration information, and object polar coordinate encoded data and encoding gain information included in the bit stream.

[0353] The position calculation unit 112 generates polar coordinate position information indicating the position of the object based on the object polar coordinate position information supplied from the decoding unit 45 and the system configuration information supplied from the communication unit 111, and supplies the generated polar coordinate position information to the rendering processing unit 113.

[0354] Furthermore, the position calculation unit 112 performs gain adjustment on the audio data of the object supplied from the decoding unit 45 , and supplies the audio data after the gain adjustment to the rendering processing unit 113 .

[0355] The position calculation unit 112 includes a coordinate conversion unit 46 , a coordinate axis conversion processing unit 47 , an object position calculation unit 48 , and a polar coordinate conversion unit 49 .

[0356] The rendering processing unit 113 performs rendering processing such as VBAP based on the polar coordinate position information and audio data supplied from the polar coordinate conversion unit 49, and generates and outputs playback audio data for playing back the sound of the content.

[0357] <Description of the provision process and the playback audio data generation process> Next, the operation of the content reproduction system shown in FIG. 17 will be described.

[0358] That is, the providing process by the server 11 and the playback audio data generating process by the client 12 will be described below with reference to the flowchart in FIG.

[0359] For example, when the client 12 requests the server 11 to distribute a predetermined content, the server 11 starts the providing process and performs the process of step S41.

[0360] That is, in step S41, the configuration information sending unit 21 reads out the system configuration information of the requested content from the configuration information recording unit 101, and transmits the read system configuration information to the client 12. For example, the system configuration information is prepared in advance, and is transmitted to the client 12 via a network or the like immediately after the content reproduction system starts operating, that is, immediately after a connection between the server 11 and the client 12 is established, and before the encoded audio data, etc. are transmitted.

[0361] Then, in step S61, the communication section 111 of the client 12 receives the system configuration information transmitted from the server 11, and supplies it to the viewpoint selection section , the coordinate axis conversion processing section 47, and the object position calculation section .

[0362] The timing at which the communication unit 111 obtains the system configuration information from the server 11 may be any timing, so long as it is before the start of playback of the content.

[0363] In step S 62 , the listener position information acquisition unit 41 acquires listener position information in response to the listener's operation or the like, and supplies the information to the viewpoint selection unit 42 , the object position calculation unit 48 , and the polar coordinate conversion unit 49 .

[0364] In step S63, the viewpoint selection unit 42 selects two or more reference viewpoints based on the system configuration information supplied from the communication unit 111 and the listener position information supplied from the listener position information acquisition unit 41, and supplies viewpoint selection information indicating the selection result to the communication unit 111.

[0365] For example, when two reference viewpoints are selected for a listening position indicated by the listener position information, two reference viewpoints that sandwich the listening position are selected from among the multiple reference viewpoints indicated by the system configuration information. In other words, the reference viewpoints are selected so that the listening position is located on the line connecting the two selected reference viewpoints.

[0366] In addition, when three-point interpolation is performed in the object position calculation unit 48, three or more reference viewpoints around the listening position indicated by the listener position information are selected from among the multiple reference viewpoints indicated by the system configuration information.

[0367] In step S 64 , the communication section 111 transmits the viewpoint selection information supplied from the viewpoint selection section 42 to the server 11 .

[0368] Then, the server 11 performs the process of step S42. That is, the configuration information sending section 21 receives the viewpoint selection information sent from the client 12 and supplies it to the encoded data sending section 22 in step S42.

[0369] The encoded data sending unit 22 reads out from the recording unit 102 the object polar coordinate encoded data and encoding gain information of the reference viewpoint indicated by the viewpoint selection information supplied from the configuration information sending unit 21 for each object, and also reads out the encoded audio data of each object of the content.

[0370] In step S43, the coded data sending unit 22 multiplexes the coded object polar coordinate data, the coding gain information, and the coded audio data read from the recording unit 102 to generate a bit stream.

[0371] In step S44, the encoded data sending unit 22 sends the generated bit stream to the client 12, and the providing process ends. With this, the content has been distributed to the client 12.

[0372] When the bit stream is transmitted, the process of step S65 is performed in the client 12. That is, the communication unit 111 receives the bit stream transmitted from the server 11 and supplies it to the decoding unit 45 in step S65.

[0373] In step S66, the decoding unit 45 extracts the object polar coordinate encoded data, the encoding gain information, and the encoded audio data from the bit stream supplied from the communication unit 111 and performs decoding.

[0374] The decoding unit 45 supplies the object polar coordinate position information obtained by decoding to a coordinate conversion unit 46, supplies the gain information obtained by decoding to an object position calculation unit 48, and further supplies the audio data obtained by decoding to a polar coordinate conversion unit 49.

[0375] In step S 67 , the coordinate conversion unit 46 performs coordinate conversion on the object polar coordinate position information of each object supplied from the decoding unit 45 , and supplies the resultant object absolute coordinate position information to the coordinate axis conversion processing unit 47 .

[0376] For example, in step S67, for each reference viewpoint, the above-mentioned formula (1) is calculated based on the object polar coordinate position information for each object, and object absolute coordinate position information is calculated.

[0377] In step S 68 , the coordinate axis conversion processing unit 47 performs coordinate axis conversion processing on the object absolute coordinate position information supplied from the coordinate conversion unit 46 based on the system configuration information supplied from the communication unit 111 .

[0378] The coordinate axis transformation processing unit 47 performs coordinate axis transformation processing for each object for each reference viewpoint, and supplies the resulting object absolute coordinate position information indicating the position of the object in the common absolute coordinate system to the object position calculation unit 48. For example, in step S68, a calculation similar to that of the above-mentioned equation (3) is performed to calculate the object absolute coordinate position information.

[0379] In step S69, the object position calculation unit 48 performs interpolation processing based on the system configuration information supplied from the communication unit 111, the listener position information supplied from the listener position information acquisition unit 41, the object absolute coordinate position information supplied from the coordinate axis transformation processing unit 47, and the gain information supplied from the decoding unit 45.

[0380] In step S69, the above-mentioned two-point or three-point interpolation is performed as the interpolation process for each object, and final object absolute coordinate position information and gain information are calculated.

[0381] For example, when two-point interpolation is performed, the object position calculation unit 48 determines the apportionment ratio (m:n) by performing a calculation similar to the above-mentioned equation (4) based on the reference viewpoint position information included in the system configuration information and the listener position information.

[0382] Then, the object position calculation unit 48 performs a two-point interpolation process by performing a calculation similar to the above-mentioned equation (5) based on the determined proportion ratio (m:n) and the object absolute coordinate position information and gain information of the two reference viewpoints.

[0383] In addition, by performing calculations similar to equation (6) or equation (7) instead of equation (5), the object absolute coordinate position information and gain information of the desired reference viewpoint may be weighted to perform interpolation processing (two-point interpolation).

[0384] Furthermore, for example, when three-point interpolation is performed, the object position calculation unit 48 selects three reference viewpoints that form (configure) a triangular mesh that satisfies the selection conditions on the viewpoint side and the object side, based on the listener position information, system configuration information, and object absolute coordinate position information of each reference viewpoint. Then, the object position calculation unit 48 performs three-point interpolation based on the object absolute coordinate position information and gain information of the selected three reference viewpoints.

[0385] That is, the object position calculation unit 48 performs calculations similar to the above-mentioned equations (9) to (14) based on the reference viewpoint position information included in the system configuration information and the listener position information, to obtain the internal division ratios (m, n) and (k, l).

[0386] Then, the object position calculation unit 48 performs the same calculation as the above-mentioned formulas (15) to (24) based on the determined internal division ratio (m,n) and internal division ratio (k,l) and the object absolute coordinate position information and gain information of each reference viewpoint, thereby performing the three-point interpolation. Note that even when performing the three-point interpolation, the object absolute coordinate position information and gain information of a desired reference viewpoint may be weighted before performing the interpolation process (three-point interpolation).

[0387] When the interpolation process is performed in this manner and the final object absolute coordinate position information and gain information are obtained, the object position calculation unit 48 supplies the obtained object absolute coordinate position information and gain information to the polar coordinate conversion unit 49 .

[0388] In step S70, the polar coordinate conversion unit 49 performs polar coordinate conversion on the object absolute coordinate position information supplied from the object position calculation unit 48 based on the listener position information supplied from the listener position information acquisition unit 41, thereby generating polar coordinate position information.

[0389] Furthermore, the polar coordinate conversion unit 49 performs gain adjustment on the audio data of each object supplied from the decoding unit 45 based on the gain information of each object supplied from the object position calculation unit 48 .

[0390] The polar coordinate conversion unit 49 supplies the polar coordinate position information obtained by the polar coordinate conversion and the audio data of each object obtained by the gain adjustment to the rendering processing unit 113.

[0391] In step S71, the rendering processing section 113 performs rendering processing such as VBAP based on the polar coordinate position information of each object and the audio data supplied from the polar coordinate conversion section 49, and outputs the playback audio data obtained as a result.

[0392] For example, the sound of the content is reproduced based on the playback audio data in a speaker or the like downstream of the rendering processing unit 113. When the playback audio data is generated and output in this manner, the playback audio data generation process ends.

[0393] In addition, in the rendering processing unit 113 or the polar coordinate conversion unit 49, before the rendering processing, processing according to the playback mode may be performed on the audio data of the object based on the listener position information and information indicating the playback mode included in the system configuration information.

[0394] In such a case, for example, the audio data of an object located at a position overlapping with the listening position is subjected to attenuation processing such as gain adjustment, or the audio data is muted by being replaced with zero data. Also, for example, the audio data of an object located at a position overlapping with the listening position is made to be output from all channels (speakers).

[0395] The provision process and playback audio data generation process described above are performed for each frame of the content.

[0396] However, the processes in steps S41 and S61 can be performed only when the playback of the content starts. Furthermore, the processes in steps S42 and S62 to S64 do not necessarily have to be performed for each frame.

[0397] In this manner, the server 11 receives the viewpoint selection information, generates a bit stream including information on a reference viewpoint corresponding to the viewpoint selection information, and transmits the bit stream to the client 12. The client 12 also performs an interpolation process based on the information on each reference viewpoint included in the received bit stream, and obtains object absolute coordinate position information and gain information for each object.

[0398] In this way, it is possible to realize object placement based on the intention of the content creator according to the listening position, rather than simply the physical relationship between the listener and the object. This allows content to be played back based on the intention of the content creator, and the enjoyment of the content can be fully conveyed to the listener.

[0399] <Explanation of viewpoint selection process> Furthermore, in the playback audio data generation process described above with reference to FIG. 18, when three-point interpolation is performed in step S69, three reference viewpoints for performing the three-point interpolation are selected.

[0400] Hereinafter, a viewpoint selection process in which the client 12 selects three reference viewpoints when three-point interpolation is performed will be described with reference to the flowchart in Fig. 19. This viewpoint selection process corresponds to the process of step S69 in Fig. 18.

[0401] In step S101, the object position calculation unit 48 calculates the distance from the listening position to each of multiple reference viewpoints based on the listener position information supplied from the listener position information acquisition unit 41 and the system configuration information supplied from the communication unit 111.

[0402] In step S102, the object position calculation unit 48 determines whether or not the frame of audio data about which three-point interpolation is to be performed (hereinafter also referred to as the current frame) is the first frame of the content.

[0403] If it is determined in step S102 that it is the first frame, the process proceeds to step S103.

[0404] In step S103, the object position calculation unit 48 selects the triangular mesh with the smallest total distance from among the triangular meshes formed from any three reference viewpoints among the multiple reference viewpoints. Here, the total distance is the sum of the distances from the listening position to each of the reference viewpoints that form the triangular mesh.

[0405] In step S104, the object position calculation unit 48 determines whether or not the listening position is within (contained in) the triangular mesh selected in step S103.

[0406] If it is determined in step S104 that the listening position is not within the triangular mesh, the triangular mesh does not satisfy the selection condition on the viewpoint side, and the process then proceeds to step S105.

[0407] In step S105, object position calculation unit 48 selects the triangular mesh on the viewpoint side that has not yet been selected in the processing of steps S103 and S105 performed so far for the frame being processed, the one with the smallest total distance.

[0408] When a new triangular mesh on the viewpoint side is selected in step S105, the process returns to step S104, and the above-described process is repeated until it is determined that the listening position is within the triangular mesh. In other words, a triangular mesh that satisfies the selection condition on the viewpoint side is searched for.

[0409] On the other hand, if it is determined in step S104 that the listening position is within a triangular mesh, that triangular mesh is selected as the triangular mesh for performing three-point interpolation, and then the process proceeds to step S110.

[0410] If it is determined in step S102 that the frame is not the first frame, the process proceeds to step S106.

[0411] In step S106, the object position calculation unit 48 determines whether or not the current listening position is within the triangular mesh on the viewpoint side selected in the frame immediately preceding the current frame (hereinafter also referred to as the previous frame).

[0412] If it is determined in step S106 that the listening position is within a triangular mesh, then the process proceeds to step S107.

[0413] In step S107, the object position calculation unit 48 selects the same triangular mesh on the viewpoint side selected for three-point interpolation in the previous frame as a triangular mesh for performing three-point interpolation in the current frame. Once the triangular meshes for three-point interpolation, i.e., the three reference viewpoints, have been selected in this manner, the process then proceeds to step S110.

[0414] Also, if it is determined in step S106 that the listening position is not within the triangular mesh on the viewpoint side selected in the previous frame, then the process proceeds to step S108.

[0415] In step S108, the object position calculation unit 48 judges whether or not any of the triangular meshes on the object side of the current frame has a common side with the selected triangular mesh on the object side of the previous frame. The judgment process in step S108 is performed based on the system configuration information and the object absolute coordinate position information.

[0416] If it is determined in step S108 that there is no triangular mesh with a common side, there is no triangular mesh that satisfies the selection conditions on the object side, and the process then proceeds to step S103. In this case, a triangular mesh that satisfies only the selection conditions on the viewpoint side is selected as the one for three-point interpolation in the current frame.

[0417] If it is determined in step S108 that there is a common edge, then the process proceeds to step S109.

[0418] In step S109, the object position calculation unit 48 selects, from among the triangular meshes on the viewpoint side of the current frame corresponding to the triangular meshes on the object side determined to have a common side in step S108, the one that includes the listening position and has the shortest total distance as a triangular mesh for three-point interpolation. In this case, the triangular mesh that satisfies the selection conditions on the object side and the selection conditions on the viewpoint side is selected. After the triangular mesh for three-point interpolation is selected in this manner, the process proceeds to step S110.

[0419] If it is determined in step S104 that the listening position is within a triangular mesh, or if the process of step S107 is performed, or if the process of step S109 is performed, then the process of step S110 is performed.

[0420] In step S110, the object position calculation unit 48 performs three-point interpolation based on the triangular mesh selected for three-point interpolation, i.e., the object coordinate position information and gain information of the selected three reference viewpoints, to generate final object absolute coordinate position information and gain information. The object position calculation unit 48 supplies the final object absolute coordinate position information and gain information thus obtained to the polar coordinate conversion unit 49.

[0421] In step S111, the object position calculation unit 48 determines whether or not there is a next frame to be processed, that is, whether or not the playback of the content has ended.

[0422] If it is determined in step S111 that there is a next frame, the playback of the content has not yet ended, so the process returns to step S101, and the above-mentioned processes are repeated.

[0423] On the other hand, if it is determined in step S111 that there is no next frame, the playback of the content has ended, and therefore the viewpoint selection process also ends.

[0424] In this way, the client 12 selects an appropriate triangular mesh based on the selection conditions on the viewpoint side and the object side, and performs three-point interpolation. In this way, discontinuous movement of the object position can be suppressed, and higher quality sound reproduction can be achieved.

[0425] According to the technology described above, when a listener moves within a free viewpoint space, playback can be achieved at each reference viewpoint in accordance with the intention of the content creator, rather than playback using the conventional physical positional relationship with respect to a fixed object arrangement.

[0426] In addition, for any listening position between multiple reference viewpoints, it is possible to generate object positions and gains that are suitable for the arbitrary listening position by performing an interpolation process based on the object arrangements of those multiple reference viewpoints, allowing the listener to seamlessly move between each reference viewpoint.

[0427] Furthermore, when the reference viewpoint overlaps with the object position, the signal level of the object can be lowered or muted, giving the listener the sensation of being the object. This makes it possible to realize, for example, a karaoke mode or a minus-one performance mode, giving the listener the sensation of participating in the content together.

[0428] Additionally, in the reference viewpoint interpolation process, if there is a reference viewpoint that the listener would like to move towards, the bias coefficient α can be applied to weight the sense of movement, allowing the content to be played back with object placement that is closer to the listener's preferred viewpoint even if the listener moves.

[0429] Furthermore, when there are four or more reference viewpoints, a triangular mesh can be constructed using three reference viewpoints, and three-point interpolation can be performed. In this case, since multiple triangular meshes can be constructed, even if the listener moves freely within the area consisting of those triangular meshes, i.e., the area surrounded by all the reference viewpoints, any position within that area can be set as the listening position, and content can be played back at an appropriate object position.

[0430] Furthermore, with this technology, when using transmission in a polar coordinate system, it is possible to realize audio playback in a free viewpoint space that reflects the intent of the content creator, simply by adding system configuration information to the conventional MPEG-H encoding method.

[0431] Example of computer configuration The above-mentioned series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed in a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, for example, capable of executing various functions by installing various programs.

[0432] FIG. 20 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes by a program.

[0433] In the computer, a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, and a RAM (Random Access Memory) 503 are interconnected via a bus 504.

[0434] An input / output interface 505 is further connected to the bus 504. An input unit 506, an output unit 507, a recording unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.

[0435] The input unit 506 includes a keyboard, a mouse, a microphone, an image sensor, etc. The output unit 507 includes a display, a speaker, etc. The recording unit 508 includes a hard disk, a non-volatile memory, etc. The communication unit 509 includes a network interface, etc. The drive 510 drives a removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0436] In a computer configured as described above, the CPU 501 loads, for example, a program recorded in the recording unit 508 into the RAM 503 via the input / output interface 505 and the bus 504, and executes the program, thereby performing the series of processes described above.

[0437] The program executed by the computer (CPU 501) can be provided by being recorded on a removable recording medium 511 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0438] In the computer, the program can be installed in the recording unit 508 via the input / output interface 505 by mounting the removable recording medium 511 in the drive 510. The program can also be received by the communication unit 509 via a wired or wireless transmission medium and installed in the recording unit 508. Alternatively, the program can be installed in the ROM 502 or the recording unit 508 in advance.

[0439] In addition, the program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or it may be a program in which processing is performed in parallel or at the required timing, such as when called.

[0440] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0441] For example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0442] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.

[0443] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices.

[0444] Furthermore, the present technology can also be configured as follows.

[0445] (1) a listener position information acquisition unit for acquiring listener position information of a listener's viewpoint; a reference viewpoint information acquisition unit that acquires position information of a first reference viewpoint and object position information of an object at the first reference viewpoint, and position information of a second reference viewpoint and object position information of the object at the second reference viewpoint; an object position calculation unit that calculates position information of the object in the listener's viewpoint based on the listener position information, the position information of the first reference viewpoint and the object position information at the first reference viewpoint, and the position information of the second reference viewpoint and the object position information at the second reference viewpoint; An information processing device comprising: (2) The first reference viewpoint and the second reference viewpoint are viewpoints that are preset by a content creator. An information processing device as described in (1). (3) The first reference viewpoint and the second reference viewpoint are viewpoints selected based on the listener position information. An information processing device according to (1) or (2). (4) the object position information is information indicating a position expressed by polar coordinates or absolute coordinates, The reference viewpoint information acquisition unit acquires gain information of the object at the first reference viewpoint and gain information of the object at the second reference viewpoint. An information processing device according to any one of (1) to (3). (5) The object position calculation unit calculates position information of the object in the listener's viewpoint by an interpolation process based on the listener position information, the position information of the first reference viewpoint and the object position information at the first reference viewpoint, and the position information of the second reference viewpoint and the object position information at the second reference viewpoint. (4) An information processing device. (6) The object position calculation unit calculates gain information of the object at the viewpoint of the listener by an interpolation process based on the listener position information, the position information of the first reference viewpoint and the gain information at the first reference viewpoint, and the position information of the second reference viewpoint and the gain information at the second reference viewpoint. An information processing device according to (4) or (5). (7) The object position calculation unit calculates the position information or the gain information of the object in the viewpoint of the listener by performing an interpolation process while weighting the object position information or the gain information in the first reference viewpoint. An information processing device according to (5) or (6). (8) the reference viewpoint information acquisition unit acquires, for three or more reference viewpoints including the first reference viewpoint and the second reference viewpoint, the position information of the reference viewpoints and the object position information at the reference viewpoints; The object position calculation unit calculates position information of the object in the listener's viewpoint by an interpolation process based on the listener position information, the position information of each of three of the reference viewpoints, and the object position information at each of the three reference viewpoints. An information processing device according to any one of (1) to (4). (9) The object position calculation unit calculates gain information of the object at the listener's viewpoint by an interpolation process based on the listener's position information, the position information of each of the three reference viewpoints, and gain information at each of the three reference viewpoints. An information processing device according to (8). (10) The object position calculation unit performs an interpolation process by weighting the object position information or the gain information at a certain one of the three reference viewpoints, to calculate the position information or the gain information of the object at the viewpoint of the listener. An information processing device according to (9). (11) The object position calculation unit defines an area formed by any three of the reference viewpoints as a triangular mesh, and selects, from among the plurality of triangular meshes, three of the reference viewpoints that form the triangular mesh that satisfies a predetermined condition as the three reference viewpoints to be used in an interpolation process. An information processing device according to any one of (8) to (10). (12) When the viewpoint of the listener moves, the object position calculation unit an area formed by each of the positions of the object indicated by each of the object position information at the three reference viewpoints forming the triangular mesh is defined as an object triangular mesh; Based on a relationship between the object triangular mesh corresponding to the triangular mesh formed by the three reference viewpoints used in the interpolation process at the viewpoint before the movement of the listener, and the object triangular mesh corresponding to the triangular mesh including the viewpoint after the movement of the listener, the three reference viewpoints to be used in the interpolation process at the viewpoint after the movement of the listener are selected. An information processing device according to (11). (13) The object position calculation unit uses, for the interpolation process at the viewpoint before the movement of the listener, three of the reference viewpoints that form the triangular mesh including the viewpoint after the movement of the listener, corresponding to the object triangular mesh having a common side with the object triangular mesh corresponding to the triangular mesh formed by the three reference viewpoints used in the interpolation process at the viewpoint before the movement of the listener. An information processing device according to (12). (14) The object position calculation unit calculates position information of the object in the listener's viewpoint based on the listener position information, the position information of the first reference viewpoint, the object position information at the first reference viewpoint, listener direction information indicating a face direction of the listener set at the first reference viewpoint, the position information of the second reference viewpoint, the object position information at the second reference viewpoint, and the listener direction information at the second reference viewpoint. An information processing device according to any one of (1) to (13). (15) The reference viewpoint information acquisition unit acquires configuration information including the position information and the listener direction information of each of a plurality of reference viewpoints including the first reference viewpoint and the second reference viewpoint. An information processing device according to (14). (16) The configuration information includes information indicating the number of the plurality of reference viewpoints and information indicating the number of the objects. An information processing device according to (15). (17) An information processing device, Obtaining listener position information of the listener's viewpoint; acquiring position information of a first reference viewpoint and object position information of an object at the first reference viewpoint, and position information of a second reference viewpoint and object position information of the object at the second reference viewpoint; Calculating position information of the object in the listener's viewpoint based on the listener position information, the position information of the first reference viewpoint and the object position information at the first reference viewpoint, and the position information of the second reference viewpoint and the object position information at the second reference viewpoint. Information processing methods. (18) Obtaining listener position information of the listener's viewpoint; acquiring position information of a first reference viewpoint and object position information of an object at the first reference viewpoint, and position information of a second reference viewpoint and object position information of the object at the second reference viewpoint; Calculating position information of the object in the listener's viewpoint based on the listener position information, the position information of the first reference viewpoint and the object position information at the first reference viewpoint, and the position information of the second reference viewpoint and the object position information at the second reference viewpoint. A program that causes a computer to execute a process including steps. [Explanation of symbols]

[0446] 11 server, 12 client, 21 configuration information sending unit, 22 encoded data sending unit, 41 listener position information acquisition unit, 42 viewpoint selection unit, 44 encoded data acquisition unit, 46 coordinate conversion unit, 47 coordinate axis conversion processing unit, 48 object position calculation unit, 49 polar coordinate conversion unit, 111 communication unit, 112 position calculation unit, 113 rendering processing unit

Claims

1. a listener position information acquisition unit for acquiring listener position information of a listener's viewpoint; a reference viewpoint information acquisition unit that acquires position information of a first reference viewpoint and first object position information of an object at the first reference viewpoint, and position information of a second reference viewpoint and second object position information of the object at the second reference viewpoint; acquiring first object position absolute coordinates and second object position absolute coordinates of the object in a common absolute coordinate space based on the position information of the first reference viewpoint and the first object position information at the first reference viewpoint, and based on the position information of the second reference viewpoint and the second object position information at the second reference viewpoint; calculating absolute coordinate position information of the object at the position of the listener in the common absolute coordinate space by performing an interpolation process based on the listener position information, the first object position absolute coordinate, and the second object position absolute coordinate; Based on the absolute coordinate position information, position information of the object in the viewpoint of the listener is obtained. An object position calculation unit An information processing device comprising:

2. The object position calculation unit acquires the first object position absolute coordinates and the second object position absolute coordinates of the object based on the position information of the first reference viewpoint, the first object position information at the first reference viewpoint, and first listener orientation information indicating a set face orientation of the listener at the first reference viewpoint, the position information of the second reference viewpoint, the second object position information at the second reference viewpoint, and second listener orientation information at the second reference viewpoint. The information processing device according to claim 1 .

3. The first reference viewpoint and the second reference viewpoint are viewpoints that are preset by a content creator. The information processing device according to claim 1 .

4. The first reference viewpoint and the second reference viewpoint are viewpoints selected based on the listener position information. The information processing device according to claim 1 .

5. The first object position information and the second object position information are information indicating positions expressed in polar coordinates or absolute coordinates, The reference viewpoint information acquisition unit acquires gain information of the object at the first reference viewpoint and gain information of the object at the second reference viewpoint. The information processing device according to claim 1 .

6. The object position calculation unit performs an interpolation process based on the listener position information, the position information of the first reference viewpoint and the first object position absolute coordinates, and the position information of the second reference viewpoint and the second object position absolute coordinates, to calculate the absolute coordinate position information of the object at the listener position. The information processing device according to claim 5 .

7. The object position calculation unit calculates gain information of the object at the viewpoint of the listener by an interpolation process based on the listener position information, the position information of the first reference viewpoint and the gain information at the first reference viewpoint, and the position information of the second reference viewpoint and the gain information at the second reference viewpoint. The information processing device according to claim 5 .

8. The object position calculation unit performs an interpolation process by weighting the first object position absolute coordinate or the gain information at the first reference viewpoint to calculate the absolute coordinate position information of the object or the gain information of the object at the viewpoint of the listener. The information processing device according to claim 7.

9. the reference viewpoint information acquisition unit acquires, for three or more reference viewpoints including the first reference viewpoint and the second reference viewpoint, the position information of the reference viewpoints and object position information at the reference viewpoints; The object position calculation unit calculates the absolute coordinate position information of the object at the position of the listener by an interpolation process based on the listener position information, the position information of each of three of the reference viewpoints, and the object position absolute coordinates at each of the three reference viewpoints. The information processing device according to claim 1 .

10. The object position calculation unit calculates gain information of the object at the listener's viewpoint by an interpolation process based on the listener's position information, the position information of each of the three reference viewpoints, and gain information at each of the three reference viewpoints. The information processing device according to claim 9.

11. The object position calculation unit performs an interpolation process by weighting the object position absolute coordinates or the gain information at a certain reference viewpoint among the three reference viewpoints, to calculate the absolute coordinate position information of the object at the position of the listener or the gain information of the object at the viewpoint of the listener. The information processing device according to claim 10.

12. The object position calculation unit defines an area formed by any three of the reference viewpoints as a triangular mesh, and selects, from among the plurality of triangular meshes, three of the reference viewpoints that form the triangular mesh that satisfies a predetermined condition as the three reference viewpoints to be used in an interpolation process. The information processing device according to claim 9.

13. When the viewpoint of the listener moves, the object position calculation unit an object triangular mesh is defined as a region formed by each of the positions of the object indicated by each of the object position information at the three reference viewpoints that form the triangular mesh; Based on a relationship between the object triangular mesh corresponding to the triangular mesh formed by the three reference viewpoints used in the interpolation process at the viewpoint before the movement of the listener, and the object triangular mesh corresponding to the triangular mesh including the viewpoint after the movement of the listener, three reference viewpoints to be used in the interpolation process at the viewpoint after the movement of the listener are selected. The information processing device according to claim 12.

14. The object position calculation unit uses, for the interpolation process at the viewpoint before the movement of the listener, three of the reference viewpoints that form the triangular mesh including the viewpoint after the movement of the listener, corresponding to the object triangular mesh having a common side with the object triangular mesh corresponding to the triangular mesh formed by the three reference viewpoints used in the interpolation process at the viewpoint before the movement of the listener. The information processing device according to claim 13.

15. The reference viewpoint information acquisition unit acquires configuration information including the position information and listener direction information of each of a plurality of reference viewpoints including the first reference viewpoint and the second reference viewpoint. The information processing device according to claim 2 .

16. The configuration information includes information indicating the number of the plurality of reference viewpoints and information indicating the number of the objects. The information processing device according to claim 15.

17. An information processing device, Obtaining listener position information of the listener's viewpoint; acquiring position information of a first reference viewpoint and first object position information of an object at the first reference viewpoint, and acquiring position information of a second reference viewpoint and second object position information of the object at the second reference viewpoint; acquiring first object position absolute coordinates and second object position absolute coordinates of the object in a common absolute coordinate space based on the position information of the first reference viewpoint and the first object position information at the first reference viewpoint, and based on the position information of the second reference viewpoint and the second object position information at the second reference viewpoint; calculating absolute coordinate position information of the object at the position of the listener in the common absolute coordinate space by performing an interpolation process based on the listener position information, the first object position absolute coordinate, and the second object position absolute coordinate; Based on the absolute coordinate position information, position information of the object in the viewpoint of the listener is obtained. Information processing methods.

18. Obtaining listener position information of the listener's viewpoint; acquiring position information of a first reference viewpoint and first object position information of an object at the first reference viewpoint, and acquiring position information of a second reference viewpoint and second object position information of the object at the second reference viewpoint; acquiring first object position absolute coordinates and second object position absolute coordinates of the object in a common absolute coordinate space based on the position information of the first reference viewpoint and the first object position information at the first reference viewpoint, and based on the position information of the second reference viewpoint and the second object position information at the second reference viewpoint; calculating absolute coordinate position information of the object at the position of the listener in the common absolute coordinate space by performing an interpolation process based on the listener position information, the first object position absolute coordinate, and the second object position absolute coordinate; Based on the absolute coordinate position information, position information of the object in the viewpoint of the listener is obtained. A program that causes a computer to execute a process including steps.

Images