Information processing device, method, and program

Abstract

The present technology relates to an information processing device and a method, and a program that make it possible to reproduce content based on the intention of the content producer. The information processing device comprises: a listener position information acquisition unit that acquires listener position information; a viewpoint selection unit that selects a plurality of reference viewpoints which form a region including a listening position at a prescribed time; a reference viewpoint information acquisition unit that acquires viewpoint position information regarding a plurality of reference viewpoints and object position information regarding an object at each of the plurality of reference viewpoints; and an object position calculation unit that, if at a time different from the prescribed time, the listening position is outside of the region including the listening position at the prescribed time, calculates position information for the object at the listening position on the basis of the object position information regarding the plurality of reference viewpoints which form a region including the listening position at that different time, or outputs the position information ultimately obtained. The present technology can be applied to an information processing device.

Description

[Technical Field]

[0001] The present technology relates to an information processing device, method, and program, and more particularly to an information processing device, method, and program that enable content playback based on the intentions of a content creator. [Background technology]

[0002] For example, in a free viewpoint space, each object placed in the space using an absolute coordinate system is placed in a fixed position (see, for example, Patent Document 1).

[0003] In this case, the direction of each object as seen from any listening position is uniquely determined based on the listener's coordinate position in absolute space, the direction of their face, and the relationship to the object, and the gain of each object is uniquely determined based on the distance from the listening position, and the sound of each object is reproduced. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2019 / 198540 Summary of the Invention [Problem to be solved by the invention]

[0005] On the other hand, there are artistic aspects of the content and points that we want to emphasize to listeners.

[0006] For example, in the case of music content, there are cases where it is desirable for an object to be closer to the foreground, such as an instrument or player at a certain listening point that you want to emphasize in terms of the content, or a player that you want to emphasize in sports content.

[0007] In light of this, the mere physical relationship between the listener and the object as described above may not adequately convey the interest of the content.

[0008] The present technology has been developed in light of these circumstances, and makes it possible to realize content playback based on the intentions of the content creator while following the listener's free position. [Means for solving the problem]

[0009] An information processing device according to one aspect of the present technology includes a listener position information acquisition unit that acquires listener position information indicating a listening position, a viewpoint selection unit that selects, from among a plurality of reference viewpoints, a plurality of reference viewpoints that form an area that includes the listening position at a predetermined time, a reference viewpoint information acquisition unit that acquires viewpoint position information of the plurality of reference viewpoints and object position information of an object at the reference viewpoint for each of the plurality of reference viewpoints, and an object position calculation unit that, when the listening position is outside the area that includes the listening position at a time different from the predetermined time, calculates position information of the object at the listening position based on the object position information of the plurality of reference viewpoints that form the area that includes the listening position at the different time, or outputs the position information of the object at the listening position that was last determined.

[0010] An information processing method or program according to one aspect of the present technology includes steps of acquiring listener position information indicating a listening position, selecting from among a plurality of reference viewpoints a plurality of reference viewpoints that form an area including the listening position at a predetermined time, acquiring viewpoint position information of the plurality of reference viewpoints and object position information of an object at the reference viewpoint for each of the plurality of reference viewpoints, and, if the listening position is outside the area including the listening position at the predetermined time at a time different from the predetermined time, calculating position information of the object at the listening position based on the object position information of the plurality of reference viewpoints that form the area including the listening position at the different time, or outputting the position information of the object at the listening position that was last determined.

[0011] In one aspect of the present technology, listener position information indicating a listening position is obtained, a plurality of reference viewpoints that form an area including the listening position at a predetermined time are selected from a plurality of reference viewpoints, viewpoint position information of the plurality of reference viewpoints and object position information of the object at the reference viewpoint for each of the plurality of reference viewpoints are obtained, and if the listening position is outside the area including the listening position at the predetermined time at a time different from the predetermined time, position information of the object at the listening position is calculated based on the object position information of the plurality of reference viewpoints that form the area including the listening position at the different time, or the last obtained position information of the object at the listening position is output. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a content playback system. [Figure 2] FIG. 10 is a diagram illustrating an example of system configuration information. [Figure 3] FIG. 10 is a diagram illustrating coordinate transformation. [Figure 4]FIG. 10 is a diagram illustrating a coordinate axis conversion process. [Figure 5] FIG. 10 is a diagram illustrating an interpolation process. [Figure 6] 10A and 10B are diagrams illustrating interpolation of object absolute coordinate position information; [Figure 7] 10A and 10B are diagrams illustrating the internal division ratio of a triangular mesh on the viewpoint side. [Figure 8] FIG. 10 is a diagram illustrating calculation of an object position using an internal division ratio. [Figure 9] 10A and 10B are diagrams illustrating calculation of gain information using an interior division ratio. [Figure 10] FIG. 10 is a diagram illustrating an example of a sequence of a content playback system. [Figure 11] FIG. 10 is a diagram illustrating the relationship between the listening position and a triangular mesh. [Figure 12] FIG. 10 is a diagram illustrating the relationship between the listening position and a triangular mesh. [Figure 13] FIG. 1 is a diagram illustrating the present technology. [Figure 14] FIG. 1 is a diagram illustrating a configuration of a content playback system. [Figure 15] 10 is a flowchart illustrating a system configuration information transmission process and a system configuration information reception process. [Figure 16] 10 is a flowchart illustrating a viewpoint selection information transmission process. [Figure 17] 10 is a flowchart illustrating a providing process. [Figure 18] 10 is a flowchart illustrating a playback audio data generation process. [Figure 19] FIG. 10 is a diagram illustrating the addition of a reference viewpoint. [Figure 20] 10 is a flowchart illustrating a viewpoint selection information transmission process. [Figure 21] 10 is a flowchart illustrating a playback audio data generation process. [Figure 22] FIG. 10 is a diagram illustrating the estimation of transmission delay and the addition of a reference viewpoint. [Figure 23]10 is a flowchart illustrating a viewpoint selection information transmission process. [Figure 24] FIG. 1 illustrates an example of the configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments to which the present technology is applied will be described with reference to the drawings.

[0014] First Embodiment <Configuration example of content playback system> The present technology has the following features F1 to F3.

[0015] (Feature F1) The method is characterized by providing object placement and gain information for multiple reference viewpoints in a free viewpoint space. (Feature F2) The object position and gain information at an arbitrary listening position are calculated based on the object arrangement and gain information at multiple reference viewpoints that surround or enclose the arbitrary listening position. (Feature F3) When obtaining the object position and gain information for an arbitrary listening position, a proportional ratio or a similar element is obtained from a plurality of reference viewpoints that sandwich or surround the arbitrary listening position and the arbitrary listening position, and the object position and gain information are obtained using the proportional ratio or a similar element.

[0016] First, a content playback system to which the present technology is applied will be described.

[0017] The content playback system includes a server and a client that encode, transmit, and decode each piece of data.

[0018] For example, listener position information is transmitted from the client to the server as needed, and some object position information is transmitted from the server to the client based on the listener position information. Then, rendering processing is performed for each object based on the object position information received on the client, and content consisting of the sound of each object is played back.

[0019] Such a content playback system is configured, for example, as shown in FIG.

[0020] That is, the content reproduction system shown in FIG.

[0021] The server 11 includes a configuration information sending unit 21 and an encoded data sending unit 22 .

[0022] The configuration information sending unit 21 sends (transmits) pre-prepared system configuration information to the client 12, and receives viewpoint selection information, etc. sent from the client 12 and supplies it to the encoded data sending unit 22.

[0023] In a content reproduction system, a plurality of listening positions in a predetermined common absolute coordinate space are designated (set) in advance by a content creator as reference viewpoint positions (hereinafter also referred to as reference viewpoint positions).

[0024] Here, the content creator specifies (sets) in advance as a reference viewpoint the position in the common absolute coordinate space that the listener should use as the listening position when the content is played back, and the direction in which the listener should face at that position, i.e., the viewpoint from which the listener should listen to the sound of the content.

[0025] The server 11 prepares system configuration information, which is information relating to each reference viewpoint, and object polar coordinate encoded data for each reference viewpoint.

[0026] Here, the object polar coordinate encoded data for each reference viewpoint is obtained by encoding object polar coordinate position information that indicates the relative position of an object as seen from the reference viewpoint. In the object polar coordinate position information, the position of an object as seen from the reference viewpoint is expressed in polar coordinates. Note that even for the same object, the absolute placement position of the object in the common absolute coordinate space differs for each reference viewpoint.

[0027] The configuration information sending unit 21 sends the system configuration information to the client 12 via a network or the like immediately after the content reproduction system starts operating, that is, immediately after a connection with the client 12 is established, for example.

[0028] The encoded data sending unit 22 sends the object polar coordinate encoded data of each of the multiple reference viewpoints indicated by the viewpoint selection information supplied from the configuration information sending unit 21 to the client 12 via a network or the like.

[0029] Here, the viewpoint selection information is information indicating two or more reference viewpoints selected on the client 12 side, for example.

[0030] Therefore, the encoded data sending unit 22 acquires the object polar coordinate encoded data of the reference viewpoint requested by the client 12 and sends it to the client 12 .

[0031] The client 12 also has a listener position information acquisition unit 41, a viewpoint selection unit 42, a configuration information acquisition unit 43, an encoded data acquisition unit 44, a decoding unit 45, a coordinate conversion unit 46, a coordinate axis conversion processing unit 47, an object position calculation unit 48, and a polar coordinate conversion unit 49.

[0032] The listener position information acquisition unit 41 acquires listener position information indicating the absolute position (listening position) of the listener in the common absolute coordinate space in response to a designation operation by the user (listener), etc., and supplies the information to the viewpoint selection unit 42, the object position calculation unit 48, and the polar coordinate conversion unit 49.

[0033] For example, in the listener position information, the position of the listener in the common absolute coordinate space is expressed by absolute coordinates. Note that hereinafter, the coordinate system of absolute coordinates indicated by the listener position information is also referred to as the common absolute coordinate system.

[0034] The viewpoint selection unit 42 selects two or more reference viewpoints based on the system configuration information supplied from the configuration information acquisition unit 43 and the listener position information supplied from the listener position information acquisition unit 41, and supplies viewpoint selection information indicating the selection result to the configuration information acquisition unit 43.

[0035] For example, the viewpoint selection unit 42 selects, from among a plurality of reference viewpoints, two or more reference viewpoints that form a predetermined range (area) that includes the listening position at a predetermined time.

[0036] Specifically, for example, the viewpoint selection unit 42 selects two reference viewpoints that sandwich the listening position. In this case, the listening position is located on the section connecting the two selected reference viewpoints, that is, in the area (range) between the two reference viewpoints.

[0037] Furthermore, for example, the viewpoint selection unit 42 selects three reference viewpoints that surround the listening position. In this case, the listening position is located within a triangular area (range) formed by the three selected reference viewpoints.

[0038] The configuration information acquisition unit 43 receives system configuration information transmitted from the server 11 and supplies it to the viewpoint selection unit 42 and the coordinate axis transformation processing unit 47, and also transmits the viewpoint selection information supplied from the viewpoint selection unit 42 to the server 11 via a network or the like.

[0039] Here, we will explain an example in which the viewpoint selection unit 42 that selects a reference viewpoint based on listener position information and system configuration information is provided in the client 12, but the viewpoint selection unit 42 may also be provided on the server 11 side.

[0040] The coded data acquisition unit 44 receives the object polar coordinate coded data transmitted from the server 11 and supplies it to the decoding unit 45. That is, the coded data acquisition unit 44 acquires the object polar coordinate coded data from the server 11.

[0041] The decoding unit 45 decodes the object polar coordinate encoded data supplied from the encoded data acquisition unit 44, and supplies the object polar coordinate position information obtained as a result to the coordinate conversion unit 46.

[0042] The coordinate conversion unit 46 performs coordinate conversion on the object polar coordinate position information supplied from the decoding unit 45 , and supplies the resulting object absolute coordinate position information to a coordinate axis conversion processing unit 47 .

[0043] The coordinate conversion unit 46 performs coordinate conversion to convert polar coordinates into absolute coordinates. As a result, object polar coordinate position information, which is polar coordinates indicating the position of an object as seen from the reference viewpoint, is converted into object absolute coordinate position information, which is absolute coordinates indicating the position of an object in an absolute coordinate system with the position of the reference viewpoint as the origin.

[0044] The coordinate axis conversion processing unit 47 performs coordinate axis conversion processing on the object absolute coordinate position information supplied from the coordinate conversion unit 46 based on the system configuration information supplied from the configuration information acquisition unit 43 .

[0045] Here, the coordinate axis transformation process is a process that combines coordinate transformation (coordinate axis transformation) and offset shift, and the coordinate axis transformation process obtains object absolute coordinate position information that indicates the absolute coordinates of the object projected onto the common absolute coordinate space. In other words, the object absolute coordinate position information obtained by the coordinate axis transformation process is the absolute coordinates of the common absolute coordinate system that indicate the absolute position of the object on the common absolute coordinate space.

[0046] The object position calculation unit 48 performs interpolation processing based on the listener position information supplied from the listener position information acquisition unit 41 and the object absolute coordinate position information supplied from the coordinate axis conversion processing unit 47, and supplies the final object absolute coordinate position information obtained as a result to the polar coordinate conversion unit 49.

[0047] The final object absolute coordinate position information here refers to information indicating the position of the object in the common absolute coordinate system when the listener's viewpoint is at the listening position indicated by the listener position information.

[0048] On the other hand, the object absolute coordinate position information output from the coordinate axis conversion processing unit 47 is information indicating the position of the object in the common absolute coordinate system when the listener's viewpoint is at the reference viewpoint.

[0049] The object position calculation unit 48 calculates the absolute position of the object in the common absolute coordinate space corresponding to the listening position, i.e., the absolute coordinates of the common absolute coordinate system, from the listening position indicated by the listener position information and the positions of the multiple reference viewpoints indicated by the viewpoint selection information, and sets this as final object absolute coordinate position information. At this time, the object position calculation unit 48 acquires system configuration information from the configuration information acquisition unit 43 and viewpoint selection information from the viewpoint selection unit 42, as necessary.

[0050] The polar coordinate conversion unit 49 performs polar coordinate conversion on the object absolute coordinate position information supplied from the object position calculation unit 48 based on the listener position information supplied from the listener position information acquisition unit 41, and outputs the resulting polar coordinate position information to a downstream rendering processing unit (not shown).

[0051] The polar coordinate conversion unit 49 performs polar coordinate conversion to convert the object absolute coordinate position information, which is the absolute coordinates in the common absolute coordinate system, into polar coordinate position information, which is the polar coordinates that indicate the relative position of the object as seen from the listening position.

[0052] In the above, an example has been described in which the server 11 prepares object polar coordinate encoded data for each reference viewpoint, but the server 11 may also prepare object absolute coordinate position information that is output from the coordinate axis transformation processing unit 47.

[0053] In such a case, the client 12 is configured so as not to include the coordinate conversion unit 46 and the coordinate axis conversion processing unit 47.

[0054] The coded data acquisition unit 44 then receives the object absolute coordinate coded data transmitted from the server 11 and supplies it to the decoding unit 45. The decoding unit 45 also decodes the object absolute coordinate coded data supplied from the coded data acquisition unit 44, and supplies the resulting object absolute coordinate position information to the object position calculation unit 48.

[0055] Next, the content playback system and the like will be further described.

[0056] First, the process of creating content provided from the server 11 to the client 12 will be described.

[0057] Content production using the polar coordinate system is currently being carried out for fixed-viewpoint-based 3D audio, and the content playback system of this technology has the advantage of being able to utilize such production techniques as is.

[0058] A content creator (hereinafter simply referred to as a creator) sets multiple reference viewpoints in a three-dimensional space from which the creator wants the listener to listen.

[0059] Reference viewpoint information, which is information relating to each reference viewpoint, is composed of reference viewpoint position information, which is the absolute coordinates of the common absolute coordinate system indicating the standing position within the common absolute coordinate space, i.e., the position of the reference viewpoint, and listener orientation information, which indicates the direction of the listener's face, but other data may also be used as components.

[0060] Here, the listener direction information includes, for example, the horizontal rotation angle (horizontal angle) of the listener's face at the reference viewpoint and a vertical angle indicating the vertical direction of the listener's face.

[0061] Next, the creator sets object polar coordinate position information, which expresses the position of each object at each of the multiple reference viewpoints in polar coordinate format, and the gain amount (gain information) for each object at each of those reference viewpoints.

[0062] For example, the object polar coordinate position information consists of a horizontal angle and a vertical angle indicating the position of the object relative to the reference viewpoint, and a radius indicating the distance from the reference viewpoint to the object.

[0063] When the position of the object and the like are set for each of the plurality of reference viewpoints in this way, the following information IFP1 to information IFP5 are obtained as information relating to the reference viewpoints.

[0064] (Information IFP1) Number of objects (Information IFP2) Number of reference viewpoints (Information IFP3) Listener's face orientation at the reference viewpoint (horizontal and vertical angles) (Information IFP4) Absolute coordinate position of the reference viewpoint in absolute space (common absolute coordinate space) (Information IFP5) Polar coordinate position (horizontal angle, vertical angle, radius) and gain amount of each object as seen from information IFP3 and information IFP4

[0065] Here, information IFP3 is the above-mentioned listener-oriented information, and information IFP4 is the above-mentioned reference viewpoint position information.

[0066] The polar coordinate position as information IFP5 is object polar coordinate position information consisting of a horizontal angle, a vertical angle, and a radius, and indicating the relative position of the object with respect to the reference viewpoint. This object polar coordinate position information is equivalent to the polar coordinate encoding information of MPEG (Moving Picture Experts Group)-H, so the MPEG-H encoding method can be used.

[0067] Of these information IFP1 to information IFP5, the information IFP1 to information IFP4 are included in the system configuration information.

[0068] This system configuration information is transmitted to the client 12 prior to transmission of data relating to the object, that is, object polar coordinate encoded data and encoded audio data obtained by encoding the audio data of the object.

[0069] A specific example of the system configuration information is as shown in FIG.

[0070] In FIG. 2, "NumOfObjs" indicates the number of objects that constitute the content, that is, the above-mentioned information IFP1, and "NumfOfRefViewPoint" indicates the number of reference viewpoints, that is, the above-mentioned information IFP2.

[0071] The system configuration information also includes as many pieces of reference viewpoint information as the number of reference viewpoints "NumfOfRefViewPoint".

[0072] That is, "RefViewX[i]", "RefViewY[i]", and "RefViewZ[i]" respectively indicate the X coordinate, Y coordinate, and Z coordinate of the common absolute coordinate system indicating the position of the reference viewpoint, which constitute the reference viewpoint position information of the i-th reference viewpoint as information IFP4.

[0073] Furthermore, "ListenerYaw[i]" and "ListenerPitch[i]" are the horizontal angle (yaw angle) and vertical angle (pitch angle) that constitute the listener direction information of the i-th reference viewpoint as information IFP3.

[0074] Furthermore, in this example, the system configuration information includes information "ObjectOverLapMode[i]" that indicates the playback mode for each object when the listener and the object overlap, i.e., when the listener (listening position) and the object are in the same position.

[0075] The server 11 stores the system configuration information obtained as described above, the object polar coordinate encoded data of each object for each reference viewpoint, and the encoding gain information obtained by encoding the gain information indicating the gain amount.

[0076] For example, in a streaming service using free viewpoint audio, when the content playback system starts operating, the server 11 transmits system configuration information to the client 12 before transmitting the object polar coordinate encoded data. This allows the client 12 to grasp the number of objects constituting the content, the number of reference viewpoints, the positions of the reference viewpoints in the common absolute coordinate space, and so on.

[0077] Next, the client 12 selects a reference viewpoint in accordance with the listener position information, and sends viewpoint selection information indicating the selection result to the server 11.

[0078] Then, the server 11 transmits to the client 12 the object polar coordinate encoded data and encoding gain information of the reference viewpoint requested by the viewpoint selection information.

[0079] On the client 12 side, the object absolute coordinate position information and gain information at the current arbitrary viewpoint of the listener are calculated by interpolation processing or the like based on the object polar coordinate encoded data, encoding gain information, and listener position information at each of the multiple reference viewpoints.

[0080] Here, a specific example of calculation of the final object absolute coordinate position information and gain information at the current arbitrary viewpoint of the listener will be described.

[0081] First, an example will be described in which the viewpoint selection information is information indicating two reference viewpoints on either side of the listener.

[0082] In such a case, the client 12 performs the following processes PC1 to PC4 to obtain the final absolute coordinate position information and gain information of the object from the listener's viewpoint.

[0083] (Processing PC1) In the process PC1, the coordinate conversion unit 46 performs coordinate conversion on the object polar coordinate position information of each object for each reference viewpoint, and generates object absolute coordinate position information.

[0084] For example, as shown in Figure 3, suppose there is one object OBJ11 in a polar coordinate system space with origin O as the reference point. Also, a three-dimensional Cartesian coordinate system (absolute coordinate system) with origin O as the reference point (origin) and x-, y-, and z-axes as the respective axes will be called an xyz coordinate system.

[0085] In this case, the position of object OBJ11 in the polar coordinate system can be expressed by polar coordinates consisting of a horizontal angle θ, which is the angle in the horizontal direction, a vertical angle γ, which is the angle in the vertical direction, and a radius r, which indicates the distance from the origin O to object OBJ11. In this example, the polar coordinates (θ, γ, r) are the object polar coordinate position information of object OBJ11.

[0086] The horizontal angle θ is the horizontal angle starting from the origin O, i.e., the point in front of the listener. In this example, let LN be the line (line segment) connecting the origin O and the object OBJ11, and let LN' be the line obtained by projecting this line LN onto the xy plane. The angle between the y-axis and the line LN' is the horizontal angle θ.

[0087] Furthermore, the vertical angle γ is the angle in the vertical direction starting from the origin O, i.e., in front of the listener, and in this example, the angle between the line LN and the xy plane is the vertical angle γ. Furthermore, the radius r is the distance from the listener (origin O) to the object OBJ11, i.e., the length of the line LN.

[0088] If the position of such an object OBJ11 is expressed by the coordinates (x, y, z) of the xyz coordinate system, that is, by absolute coordinates, it becomes as shown in the following equation (1).

[0089]

number

[0090] In process PC1, by calculating equation (1) based on the object polar coordinate position information, which is a polar coordinate, object absolute coordinate position information, which is absolute coordinates indicating the position of the object in an xyz coordinate system (absolute coordinate system) with the position of the reference viewpoint as the origin O, is calculated.

[0091] In particular, in the process PC1, for each of two reference viewpoints, coordinate conversion is performed on the object polar coordinate position information of each of a plurality of objects at those reference viewpoints.

[0092] (Processing PC2) In process PC2, the coordinate axis conversion processing unit 47 performs coordinate axis conversion processing on the object absolute coordinate position information obtained in process PC1 for each of the two reference viewpoints for each object.

[0093] The object absolute coordinate position information at each of the two reference viewpoints obtained in the above-described process PC1 indicates a position in an xyz coordinate system with each reference viewpoint as the origin O. Therefore, the coordinates (coordinate system) of the object absolute coordinate position information differ for each reference viewpoint.

[0094] Therefore, a coordinate axis conversion process is performed as process PC2 to combine the object absolute coordinate position information at each reference viewpoint into absolute coordinates in one common absolute coordinate system, that is, absolute coordinates in a common absolute coordinate system (common absolute coordinate space).

[0095] To perform this coordinate axis transformation process, reference viewpoint position information and listener direction information for each reference viewpoint are required in addition to object absolute coordinate position information for each object for each reference viewpoint.

[0096] In other words, the coordinate axis transformation process requires the object absolute coordinate position information obtained by process PC1, reference viewpoint position information indicating the position of the reference viewpoint in the common absolute coordinate system, and system configuration information including listener direction information at the reference viewpoint.

[0097] For simplicity of explanation, only the horizontal rotation angle is used as the face direction indicated by the listener direction information, but information on the pitch of the face can also be added.

[0098] Now, assuming that the common absolute coordinate system is an XYZ coordinate system with the X, Y, and Z axes as its respective axes, and that the rotation angle corresponding to the face direction indicated by the listener direction information is φ, then coordinate axis conversion processing is performed as shown in, for example, FIG. 4.

[0099] That is, in the example shown in FIG. 4, the coordinate axis transformation process includes coordinate axis rotation, which rotates the coordinate axes by a rotation angle φ, and a process of shifting the origin of the coordinate axes from the position of the reference viewpoint to the origin position of the common absolute coordinate system; more specifically, a process of shifting the position of the object in accordance with the positional relationship between the reference viewpoint and the origin of the common absolute coordinate system.

[0100] 4, position P21 indicates the position of the reference viewpoint, and arrow Q11 indicates the direction of the listener's face indicated by the listener orientation information at that reference viewpoint. In particular, the X and Y coordinates of position P21 in the common absolute coordinate system (XYZ coordinate system) are (Xref, Yref).

[0101] Position P22 indicates the position of the object when the reference viewpoint is at position P21. Here, the X and Y coordinates of the common absolute coordinate system indicating object position P22 are (Xobj, Yobj), and the x and y coordinates of the xyz coordinate system with the reference viewpoint as the origin indicating object position P22 are (xobj, yobj).

[0102] Furthermore, in this example, the angle φ formed by the X axis of the common absolute coordinate system (XYZ coordinate system) and the x axis of the xyz coordinate system is the rotation angle φ of the coordinate axis transformation determined from the listener direction information.

[0103] Therefore, for example, the coordinate axis X (X coordinate) and the coordinate axis Y (Y coordinate) after the transformation are as shown in the following equation (2).

[0104]

number

[0105] In equation (2), x and y represent the x-axis (x-coordinate) and y-axis (y-coordinate) before conversion, i.e., the xyz coordinate system. Also, the "reference viewpoint X coordinate value" and "reference viewpoint Y coordinate value" in equation (2) represent the X-coordinate and Y-coordinate indicating the position of the reference viewpoint in the XYZ coordinate system (common absolute coordinate system), i.e., the X-coordinate and Y-coordinate constituting the reference viewpoint position information.

[0106] Therefore, in the example of FIG. 4, the X coordinate value Xobj and the Y coordinate value Yobj indicating the position of the object after the coordinate axis transformation process can be calculated from equation (2).

[0107] That is, φ in equation (2) is set to the rotation angle φ calculated from the listener orientation information at position P21, and the X coordinate value Xobj can be obtained by substituting "Xref," "xobj," and "yobj" for "reference viewpoint X coordinate value," "x," and "y" in equation (2), respectively.

[0108] Furthermore, the Y coordinate value Yobj can be obtained by substituting "Yref", "xobj", and "yobj" for "reference viewpoint Y coordinate value", "x", and "y" in equation (2), respectively, with φ in equation (2) being the rotation angle φ calculated from the listener orientation information at position P21.

[0109] Similarly, if two reference viewpoints, A and B, are selected by the viewpoint selection information, the X and Y coordinate values ​​indicating the position of the object after the coordinate axis transformation process for those reference viewpoints are as shown in the following equation (3).

[0110]

number

[0111] In equation (3), xa and ya represent the X and Y coordinate values ​​in the XYZ coordinate system after axis transformation (after coordinate axis transformation processing) for reference viewpoint A, and φa represents the rotation angle of the axis transformation for reference viewpoint A, i.e., the above-mentioned rotation angle φ.

[0112] Therefore, by substituting the x and y coordinates constituting the object absolute coordinate position information at the reference viewpoint A obtained in process PC1 into equation (3), coordinates xa and ya are obtained as the X and Y coordinates indicating the position of the object in the XYZ coordinate system (common absolute coordinate system) at the reference viewpoint A. The absolute coordinates consisting of the coordinates xa and ya obtained in this way and the Z coordinate are the object absolute coordinate position information output from coordinate axis conversion processing unit 47.

[0113] In this example, only the horizontal rotation angle φ is handled, so coordinate axis conversion is not performed on the Z axis (Z coordinate). Therefore, for example, the z coordinate constituting the object absolute coordinate position information obtained in process PC1 can be used as is as the Z coordinate indicating the position of the object in the common absolute coordinate system.

[0114] As with reference viewpoint A, in equation (3), xb and yb indicate the X and Y coordinate values ​​in the XYZ coordinate system after axis transformation (after coordinate axis transformation processing) for reference viewpoint B, and φb indicates the rotation angle (rotation angle φ) of the axis transformation for reference viewpoint B.

[0115] The coordinate axis conversion processing unit 47 performs the above-described coordinate axis conversion processing as processing PC2.

[0116] (Processing PC3) In process PC3, the allocation ratio for the interpolation process is calculated based on the positional relationship between the absolute coordinate positions of each of the two reference viewpoints, i.e., the position indicated by the reference viewpoint position information included in the system configuration information, and an arbitrary listening position sandwiched between the positions of the two reference viewpoints.

[0117] That is, the object position calculation unit 48 performs processing PC3 to determine the allocation ratio (m:n) based on the listener position information supplied from the listener position information acquisition unit 41 and the reference viewpoint position information included in the system configuration information.

[0118] Here, it is assumed that the reference viewpoint position information indicating the position of the first reference viewpoint A is (x1, y1, z1), the reference viewpoint position information indicating the position of the second reference viewpoint B is (x2, y2, z2), and the listener position information indicating the listening position is (x3, y3, z3).

[0119] In this case, the object position calculation unit 48 calculates the proportion ratio (m:n), that is, the proportion ratio m and n, by performing the calculation of the following equation (4).

[0120]

number

[0121] (Processing PC4) Next, the object position calculation unit 48 performs interpolation processing as processing PC4 based on the proportion ratio (m:n) obtained in processing PC3 and the object absolute coordinate position information of each object of the two reference viewpoints supplied from the coordinate axis transformation processing unit 47.

[0122] That is, in process PC4, the object position and gain amount corresponding to an arbitrary listening position are determined by applying the allocation ratio (m:n) obtained in process PC3 to the same object corresponding to the two reference viewpoints obtained in process PC2.

[0123] Here, the object absolute coordinate position information of the predetermined object at the reference viewpoint A obtained by the process PC2 is (xa, ya, za), and the gain amount indicated by the gain information of the predetermined object for the reference viewpoint A is g1.

[0124] Similarly, the object absolute coordinate position information of a predetermined object at the reference viewpoint B obtained by the process PC2 is (xb, yb, zb), and the gain amount indicated by the gain information of the object for the reference viewpoint B is g2.

[0125] Also, (xc, yc, zc) and gain_c are the absolute coordinates indicating the position in the XYZ coordinate system (common absolute coordinate system) of the above-mentioned predetermined object and the gain amount corresponding to any listening position between reference viewpoint A and reference viewpoint B. These absolute coordinates (xc, yc, zc) are the final object absolute coordinate position information output from object position calculation unit 48 to polar coordinate conversion unit 49.

[0126] At this time, the final object absolute coordinate position information (xc, yc, zc) and gain amount gain_c for the predetermined object can be obtained by calculating the following equation (5) using the proportional division ratio (m:n).

[0127]

number

[0128] The positional relationship between the reference viewpoint A, the reference viewpoint B, and the listening position described above, and the positional relationship between the same object at each of the reference viewpoint A, the reference viewpoint B, and the listening position, are as shown in FIG.

[0129] 5, the horizontal and vertical axes respectively represent the X and Y axes of the XYZ coordinate system (common absolute coordinate system). Note that, for simplicity of explanation, only the X-axis and Y-axis directions are shown here.

[0130] In this example, position P51 is a position indicated by reference viewpoint position information (x1, y1, z1) of reference viewpoint A, and position P52 is a position indicated by reference viewpoint position information (x2, y2, z2) of reference viewpoint B.

[0131] Furthermore, a position P53 between reference viewpoint A and reference viewpoint B is a listening position indicated by listener position information (x3, y3, z3).

[0132] In the above equation (4), the proportion ratio (m:n) is found based on the positional relationship between the reference viewpoint A, the reference viewpoint B, and the listening position.

[0133] Furthermore, position P61 is a position indicated by the object absolute coordinate position information (xa, ya, za) at reference viewpoint A, and position P62 is a position indicated by the object absolute coordinate position information (xb, yb, zb) at reference viewpoint B.

[0134] Furthermore, a position P63 located between the positions P61 and P62 is a position indicated by the object absolute coordinate position information (xc, yc, zc) at the listening position.

[0135] By performing the calculation of equation (5), that is, the interpolation process, absolute object coordinate position information indicating the appropriate object position can be obtained for any listening position.

[0136] Note that, in the above, an example has been described in which the object position, i.e., the final object absolute coordinate position information, is determined using the proportional division ratio (m:n). However, the present invention is not limited to this, and the final object absolute coordinate position information may be estimated using machine learning or the like.

[0137] Two-point interpolation using information from two reference viewpoints has been described above.

[0138] Next, a specific example of three-point interpolation using information from three reference viewpoints will be described.

[0139] For example, as shown on the left side of FIG. 6, it is assumed that absolute coordinate position information of an object at an arbitrary listening position F is obtained by interpolation processing.

[0140] In this example, there are three reference viewpoints A, B, and C surrounding listening position F, and it is assumed that information from these reference viewpoints A to C is used to perform interpolation processing.

[0141] In the following, the X and Y coordinates of the listening position F in the common absolute coordinate system, i.e., the XYZ coordinate system, are (x f ,y f )

[0142] Similarly, the X and Y coordinates of the positions of reference viewpoint A, reference viewpoint B, and reference viewpoint C are (x a ,y a ), (x b ,y b ), and (x c ,y c )

[0143] In this case, object position F' at listening position F is determined based on the coordinates of object position A', object position B', and object position C' corresponding to reference viewpoint A, reference viewpoint B, and reference viewpoint C, respectively, as shown on the right side of Figure 6.

[0144] Here, for example, object position A' indicates the position of the object when the viewpoint is at reference viewpoint A, that is, the position of the object in the common absolute coordinate system indicated by the object absolute coordinate position information of reference viewpoint A.

[0145] Moreover, object position F′ indicates the position of the object in the common absolute coordinate system when the listener is at listening position F, that is, the position indicated by the object absolute coordinate position information output by object position calculation section 48.

[0146] In the following, the X and Y coordinates of object position A', object position B', and object position C' are (xa ',y a '), (x b ',y b '), and (x c ',y c '), and the X and Y coordinates of object position F' are (x f ',y f ').

[0147] In the following description, a triangular area surrounded by any three reference viewpoints, such as reference viewpoint A to reference viewpoint C, i.e., a triangular area formed by three reference viewpoints, will also be referred to as a triangular mesh. For example, a triangular mesh formed by reference viewpoint A to reference viewpoint C will be referred to as triangular mesh ABC.

[0148] Since a plurality of reference viewpoints exist in the common absolute coordinate space, a plurality of triangular meshes can be formed in the common absolute coordinate space with the reference viewpoints as vertices.

[0149] Similarly, hereinafter, a triangular area surrounded (formed) by object positions indicated by the object absolute coordinate position information of any three reference viewpoints, such as object position A' to object position C', will also be referred to as a triangular mesh. For example, a triangular mesh formed by object positions A' to C' will be referred to as triangular mesh A'B'C', etc.

[0150] In the above example of two-point interpolation, the listener can move to any position on the line connecting the two reference viewpoints and listen to the sound of the content.

[0151] In contrast, with three-point interpolation, the listener can move to any position within the triangular mesh area surrounded by the three reference viewpoints and listen to the sound of the content. In other words, the listening position can cover the area other than the line segment connecting the two reference viewpoints in the case of two-point interpolation.

[0152] As with the two-point interpolation, when three-point interpolation is performed, the coordinates indicating any position in the common absolute coordinate system (XYZ coordinate system) can be obtained from the coordinates of that position in the xyz coordinate system, listener direction information, and reference viewpoint position information using the above formula (2).

[0153] Here, the Z coordinate value in the XYZ coordinate system is assumed to be the same as the z coordinate value in the xyz coordinate system. However, if the Z coordinate value and z coordinate value are different, the Z coordinate value indicating an arbitrary position can be calculated by adding the Z coordinate value indicating the position of the reference viewpoint in the XYZ coordinate system to the z coordinate value of that arbitrary position.

[0154] Ceva's theorem proves that any listening position within a triangular mesh formed by three reference viewpoints can be uniquely determined as the intersection of line segments from each of the three vertices of the triangular mesh to each of the three internal division points of the three sides that are not adjacent to those vertices, provided that the internal division ratios of each side of the triangular mesh are appropriately determined.

[0155] From the proof formula, this holds true for all triangular meshes, regardless of the shape of the triangular mesh, once the internal division ratio of the three sides of the triangular mesh is determined.

[0156] Therefore, by calculating the internal division ratio of the triangular mesh including the listening position for the viewpoint side, i.e., the reference viewpoint, and applying that internal division ratio to the triangular mesh for the object side, i.e., the object position, it is possible to calculate the appropriate object position for any listening position.

[0157] An example will be described below in which the property of such an internal division ratio is used to determine absolute object coordinate position information that indicates the position of an object when the listener is at an arbitrary listening position.

[0158] In this case, first, the internal division ratio of the sides of the triangular mesh of the reference viewpoint on the XY plane of the XYZ coordinate system, which is a two-dimensional space, is calculated.

[0159] Next, the above-mentioned internal division ratio is applied to the triangular mesh of the object position corresponding to the three reference viewpoints on the XY plane, and the X and Y coordinates of the object position corresponding to the listening position on the XY plane are obtained.

[0160] Furthermore, the Z coordinate of the object corresponding to the listening position is calculated based on a three-dimensional plane containing the positions of the three objects corresponding to the three reference viewpoints in three-dimensional space (XYZ coordinate system) and the X and Y coordinates of the object at the listening position on the XY plane.

[0161] Here, with reference to FIGS. 7 to 9, an example will be described in which the object absolute coordinate position information and gain information indicating the object position F' are obtained by interpolation processing for the listening position F shown in FIG.

[0162] For example, as shown in FIG. 7, first, the X and Y coordinates of the dividing point in the triangular mesh consisting of reference viewpoints A to C, including the listening position F, are found.

[0163] Now, let the intersection of the line passing through the listening position F and the reference viewpoint C with the line segment AB from the reference viewpoint A to the reference viewpoint B be point D. Let the coordinates of point D on the XY plane be (x d ,y d ) In other words, point D is an internal division point on line segment AB (side AB).

[0164] In this case, the relationship shown in the following equation (6) holds for the X and Y coordinates indicating the position of any point on the line segment CF from the reference viewpoint C to the listening position F, and for the X and Y coordinates indicating the position of any point on the line segment AB.

[0165]

number

[0166] Furthermore, point D is the intersection of the line passing through reference viewpoint C and listening position F with line segment AB, so from equation (6), the coordinates of point D on the XY plane (x d ,y d ) can be found, and its coordinates (x d ,y d ) is expressed as the following equation (7).

[0167]

number

[0168] Therefore, as shown in the following equation (8), the coordinates of point D (x d ,y d ), coordinates of reference viewpoint A (x a ,y a ), and the coordinates of reference viewpoint B (x b ,y b ), we can obtain the internal division ratio (m,n) of the line segment AB by point D, that is, the division ratio.

[0169]

number

[0170] Similarly, the intersection of the line passing through the listening position F and the reference viewpoint B with the line segment AC from the reference viewpoint A to the reference viewpoint C is defined as point E, and the coordinates indicating the position of point E on the XY plane are (x e ,y e ) In other words, point E is an internal division point on line segment AC (side AC).

[0171] In this case, the relationship shown in the following equation (9) holds for the X and Y coordinates indicating the position of any point on the line segment BF from the reference viewpoint B to the listening position F, and for the X and Y coordinates indicating the position of any point on the line segment AC.

[0172]

number

[0173] Furthermore, point E is the intersection of the line passing through reference viewpoint B and listening position F with line segment AC, so from equation (9), the coordinates of point E on the XY plane (x e ,y e ) can be found, and its coordinates (x e ,y e ) is expressed as the following equation (10).

[0174]

number

[0175] Therefore, as shown in the following equation (11), the coordinates of point E (x e ,y e ), coordinates of reference viewpoint A (x a ,y a ), and the coordinates of the reference viewpoint C (x c ,y c ), we can obtain the internal division ratio (k, l) of the line segment AC by point E, that is, the division ratio.

[0176]

number

[0177] Next, the ratio of the two sides thus obtained, that is, the internal division ratio (m, n) and the internal division ratio (k, l), are applied to the triangular mesh on the object side as shown in Figure 8, and the coordinates of the object position F' on the XY plane (x f ',y f ') is required.

[0178] Specifically, in this example, the point on the line segment A'B' connecting the object position A' and the object position B', which corresponds to the point D, is set as point D'.

[0179] Similarly, the point on the line segment A'C' connecting the object position A' and the object position C' that corresponds to the point E is set as point E'.

[0180] Furthermore, the intersection of a line passing through object position C' and point D' and a line passing through object position B' and point E' is object position F' corresponding to listening position F.

[0181] Here, let us assume that the internal division ratio of line segment A'B' at point D' is the same as that at point D (m, n). In this case, the coordinates of point D' on the XY plane (x d ',y d As shown in the following equation (12), the internal division ratio (m, n) and the coordinates of the object position A' (x a ',y a '), and the coordinates of object position B' (x b ',y b ') can be obtained based on

[0182]

number

[0183] Also, assume that the internal division ratio of the line segment A'C' at point E' is the same as that at point E (k, l). In this case, the coordinates of point E' on the XY plane (x e ',y e As shown in the following equation (13), the internal division ratio (k, l) and the coordinates of the object position A' (x a ',y a '), and the coordinates of object position C' (x c ',y c ') can be obtained based on

[0184]

number

[0185] Therefore, the relationship shown in the following equation (14) holds for the X and Y coordinates indicating the position of any point on the line segment B'E' from object position B' to point E', and the X and Y coordinates indicating the position of any point on the line segment C'D' from object position C' to point D'.

[0186]

number

[0187] The target object position F' is the intersection of the line segments B'E' and C'D', so from the relationship in equation (14), the coordinates of the object position F' (x f ',y f ') can be obtained.

[0188]

number

[0189] By the above process, the coordinates of the object position F' on the XY plane (x f ',y f ') is obtained.

[0190] Next, the coordinates of the object position F' on the XY plane (x f ',y f ') and the coordinates of object position A' in the XYZ coordinate system (x a ',y a ',z a '), coordinates of object position B' (x b ',y b ',z b '), and the coordinates of object position C' (x c ',y c ',z c ') and the coordinates (x,y,z) of the object position F' in the XYZ coordinate system f ',y f ',z f That is, the Z coordinate z of the object position F' in the XYZ coordinate system is calculated. f ' is required.

[0191] For example, a triangle in three-dimensional space with vertices at object positions A', B', and C' in the XYZ coordinate system (common absolute coordinate space) is obtained, that is, a three-dimensional plane A'B'C' including object positions A', B', and C' is obtained. Then, the X and Y coordinates on the three-dimensional plane A'B'C' are calculated as follows: f ',y f ') is found, and the Z coordinate of that point is z f 'It is said that.

[0192] Specifically, the vector starting from object position A' in the XYZ coordinate system and ending at object position B' is called vector A'B'=(x ab ',y ab ',z ab ').

[0193] Similarly, the vector starting from object position A' in the XYZ coordinate system and ending at object position C' is vector A'C'=(x ac ',y ac ',z ac ').

[0194] These vectors A'B' and A'C' are the coordinates of the object position A' (x a ',y a ',z a '), coordinates of object position B' (x b ',y b ',z b '), and the coordinates of object position C' (x c ',y c ',z c That is, vectors A'B' and A'C' can be obtained by the following equation (16).

[0195]

number

[0196] Furthermore, the normal vector (s, t, u) of the three-dimensional plane A'B'C' is the cross product of the vector A'B' and the vector A'C', and can be calculated by the following equation (17).

[0197]

number

[0198] Therefore, the normal vector (s,t,u) and the coordinates of the object position A' (x a ',y a ',z a '), the plane equation of the three-dimensional plane A'B'C' is given by the following equation (18).

[0199]

number

[0200] Here, the X coordinate x of the object position F' on the 3D plane A'B'C' is f ' and the Y coordinate y f ' has already been found, so we can use the X coordinate x in the X and Y parts of the plane equation (18). f ' and the Y coordinate y f By substituting ', the Z coordinate z is calculated as shown in the following equation (19). f ' can be sought.

[0201]

number

[0202] From the above calculation, the coordinates of the target object position F' (x f ',y f ',z f The object position calculation unit 48 calculates the coordinates (x f ',y f ',z f ') is output as absolute object coordinate position information.

[0203] As with the object absolute coordinate position information, the gain information can also be determined by three-point interpolation.

[0204] That is, the gain information of the object at object position F′ can be obtained by performing an interpolation process based on the gain information of the object when the viewpoint is at each of reference viewpoints A to C.

[0205] For example, as shown in FIG. 9, the gain information G of the object at object position F' in the triangular mesh formed by object positions A', B', and C' is f Think about asking for '.

[0206] Now, when the viewpoint is at the reference viewpoint A, the gain information of the object at the object position A' is G a ', and the gain information of the object at object position B' is G b ', and the gain information of the object at object position C' is G c '.

[0207] In this case, first, the gain information G of the object at point D', which is the internal division point of the line segment A'B' when the viewpoint is at point D. d ' is required.

[0208] Specifically, the gain information G d ' is the internal division ratio (m,n) of the above line segment A'B' and the gain information G of the object position A' a and the gain information G for object position B. b Based on these, it can be obtained by calculating the following equation (20).

[0209]

number

[0210] That is, in equation (20), the gain information G a ' and gain information G b ', the gain information G d ' is required.

[0211] Next, the internal division ratio (o, p) of the line segment C'D' from the object position C' to the point D' by the object position F' and the gain information G c ' and point D' d By performing interpolation based on the gain information G of the object position F', f That is, by performing the calculation of the following equation (21), the gain information G f ' is required.

[0212]

number

[0213] The object position calculation unit 48 outputs the gain information G f ' is output as the gain information of the object corresponding to the listening position F.

[0214] As described above, by performing three-point interpolation, it is possible to obtain absolute object coordinate position information and gain information for any listening position. Note that the following description will be continued assuming that three-point interpolation is basically performed.

[0215] <About listeners and objects> Incidentally, there are two possible examples of reference viewpoints: one that assumes the viewpoint of a listener, and one that assumes the viewpoint of a performer who imagines becoming the object.

[0216] In the latter case, the listener and the object overlap in the reference viewpoint, that is, the listener and the object are in the same position, and therefore the following cases CA1 to CA3 are possible.

[0217] (Case CA1) Prevent listeners from overlapping with objects, or prevent listeners from entering a specific area (Case CA2) The listener becomes one with the object, and the sound generated by the object is output from all channels. (Case CA3) Mutes or attenuates sounds coming from overlapping objects

[0218] For example, in case CA2, it is possible to recreate the sensation of being localized inside the listener's head.

[0219] In addition, in case CA3, by muting or attenuating the sound of the objects, the listener can become the performer, for example, in a karaoke mode. In this case, the listener can get the feeling that they are singing in a surrounding environment, surrounded by accompaniment other than the performer's voice.

[0220] If the content creator has such an intention, the content creator can store an identifier indicating these cases CA1 to CA3 in the coded bitstream transmitted from the server 11 and transmit it to the client 12. For example, such an identifier is information indicating the above-mentioned playback mode.

[0221] <Content playback system operation> The content playback system described above can be applied to a system that distributes free viewpoint audio content using Audio Artistic Intent, for example. In this case, content may be distributed in real time, or archived content prepared in advance may be distributed (archive data transmission).

[0222] As described above, in the content playback system, multiple reference viewpoints created by the content creator are assumed, and information on object placement at these reference viewpoints, i.e., system configuration information and object polar coordinate position information, etc., is created.

[0223] On the other hand, the listener can freely move to positions other than the reference viewpoint.

[0224] If the listener is located at a position other than the reference viewpoint, interpolation is performed based on the absolute object coordinate position information of multiple reference viewpoints surrounding the listener's position, and the absolute object coordinate position information corresponding to the listener's current position is calculated. This makes it possible to reproduce spatial audio from any viewpoint while reflecting the intentions of the content creator.

[0225] However, when transmitting the object polar coordinate encoded data corresponding to the reference viewpoint requested by the listener, delays in the network, etc. may cause delays in the arrival of the object polar coordinate encoded data at the client 12.

[0226] This would result in an event where the client 12 is unable to obtain appropriate object position information corresponding to the current listening position.

[0227] An example of a processing flow (sequence) performed in the content playback system and transmission delays will be described below with reference to FIG.

[0228] For example, on the server 11 side, object polar coordinate encoded data is generated and stored by a polar coordinate system editor for all reference viewpoints, and system configuration information is also generated and stored.

[0229] Then, the server 11 transmits the system configuration information to the client 12 via a network or the like, and the client 12 receives and stores the system configuration information. At this time, the client 12 decodes the received system configuration information and initializes the client system.

[0230] Next, the client 12 selects the reference viewpoint required for the interpolation process based on the listener position information and system configuration information, and requests the transmission of object polar coordinate encoded data by sending viewpoint selection information indicating the selection result to the server 11.

[0231] For example, when selecting reference viewpoints, three reference viewpoints surrounding the listening position or two reference viewpoints sandwiching the listening position are selected. In other words, multiple reference viewpoints that form a range including the listening position, that is, a section sandwiching the listening position, or a region surrounding the listening position are selected.

[0232] Furthermore, in response to viewpoint selection information received from the client 12, the server 11 prepares to transmit the encoded object polar coordinate data of the reference viewpoint required for the interpolation process.

[0233] That is, the server 11 generates a bitstream by reading and multiplexing the object polar coordinate encoded data and encoding gain information of the reference viewpoint indicated by the viewpoint selection information. Then, the server 11 transmits (transmits) the generated bitstream to the client 12.

[0234] The client 12 receives the bitstream transmitted from the server 11, demultiplexes and decodes it, and obtains the object polar coordinate position information and gain information.

[0235] The client 12 converts the object polar coordinate position information into object absolute coordinate position information by performing coordinate transformation, and also performs coordinate axis transformation on the object absolute coordinate position information to expand it into a common absolute coordinate space.

[0236] Furthermore, the client 12 calculates the above-mentioned internal division ratio or proportional division ratio for the interpolation process from the current listener position and the position of the reference viewpoint, and performs interpolation process for the object absolute coordinate position information and gain information, thereby obtaining the object absolute coordinate position information and gain information corresponding to the current listener position.

[0237] Thereafter, the client 12 converts the object absolute coordinate position information into polar coordinate position information by polar coordinate conversion, and performs rendering processing by applying the obtained polar coordinate position information and gain information.

[0238] For example, all objects are rendered using a polar coordinate system defined by MPEG-H, such as Vector Based Amplitude Panning (VBAP), to obtain playback audio data for playing back the sound of the content.

[0239] The above process is performed for a predetermined frame, and when playback audio data is generated, content is played back based on the playback audio data as appropriate. After that, new viewpoint selection information is sent from the client 12 to the server 11 as appropriate, and the above process is repeated.

[0240] In this way, the content playback system calculates object position information and gain information at any listening position by interpolation processing from object position information for each of multiple reference viewpoints. In this way, it is possible to realize object placement based on the intentions of the content creator according to the listening position, rather than simply the physical relationship between the listener and the object. This allows content playback based on the intentions of the content creator to fully convey the enjoyment of the content to the listener.

[0241] However, if there is a large transmission delay in the network between the server 11 and the client 12, the client 12 may not be able to obtain appropriate polar coordinate position information of the object.

[0242] For example, the time when the client 12 generates viewpoint selection information, as shown by arrow Q41, is defined as time Tα, and the listening position at that time Tα is defined as listening position α. ​​Also, the time when the client 12 receives the object polar coordinate encoded data (bitstream), as shown by arrow Q42, is defined as time Tβ, and the listening position at that time Tβ is defined as listening position β.

[0243] In this case, the time from time Tα to time Tβ is a delay time that is made up of the processing time in the server 11, transmission delays on the network, and the like.

[0244] If such a delay time is large, the listener may have moved from listening position α at time Tα to a different listening position β, and the object polar coordinate encoded data received at time Tβ may no longer be appropriate for the listening position β after the move.

[0245] As a specific example, a case where three reference viewpoints surrounding the listening position are selected, that is, a case where three-point interpolation is performed, will be described.

[0246] For example, as shown in FIG. 11, assume that there are a triangular mesh ABC consisting of reference viewpoints A to C and a triangular mesh BCD consisting of reference viewpoints B to D.

[0247] Also, it is assumed that the listening position α at time Tα is the position indicated by the arrow F11.

[0248] In this case, since the listening position α is within the triangular mesh ABC, viewpoint selection information indicating reference viewpoint A, reference viewpoint B, and reference viewpoint C is generated at time Tα. Then, at time Tβ, object polar coordinate encoded data for reference viewpoint A to reference viewpoint C is received.

[0249] In this case, if listening position β at time Tβ is the position indicated by arrow F12, listening position β is within triangular mesh ABC, just like listening position α.

[0250] Therefore, the object polar coordinate encoded data at reference viewpoints A to C is required for the interpolation process to obtain the final object absolute coordinate position information when the listener is at listening position β.

[0251] The client 12 receives the object polar coordinate encoded data required for the interpolation process when the listener is at the listening position β at time Tβ, and can therefore perform the interpolation process appropriately to obtain the reproduced audio data.

[0252] On the other hand, for example, as shown in FIG. 12, it is assumed that listening position α at time Tα is the position indicated by arrow F21, and listening position β at time Tβ is the position indicated by arrow F22.

[0253] In this example, at time Tα, listening position α was within triangular mesh ABC, but the listener then moves, and at time Tβ, listening position β is within triangular mesh BCD.

[0254] Therefore, for the interpolation process when the listener is at the listening position β, the object polar coordinate encoded data of reference viewpoints B to D is required.

[0255] However, what is received at time Tβ is the object polar coordinate encoded data of reference viewpoints A to C. Therefore, the client 12 does not have the object polar coordinate encoded data of reference viewpoint D, and is unable to perform correct interpolation processing for listening position β.

[0256] From the above, if the listening positions α and β are located within the same triangular mesh, the interpolation process can be performed appropriately.

[0257] On the other hand, if listening positions α and β are located in different triangular meshes due to the influence of transmission delays, etc., it becomes impossible to perform interpolation processing for the current position of the listener. In other words, it becomes impossible to perform rendering processing continuously.

[0258] If, at time Tβ, the received object polar coordinate encoded data for reference viewpoints A to C and listener position information indicating the current listening position β are used, interpolation processing and rendering processing will be performed with an incorrect positional relationship (inappropriate positional relationship).

[0259] Therefore, in this technology, when listening positions α and β are located within different triangular meshes, for example, as shown in Fig. 13, the rendering process can be continued by using the polar coordinate position information obtained in the last interpolation process. Note that in Fig. 13, parts corresponding to those in Fig. 11 are assigned the same reference numerals, and their explanation will be omitted where appropriate.

[0260] In the example of Figure 13, listening position α at time Tα is the position indicated by arrow F11, and listening position β at time Tβ is the position indicated by arrow F12. These listening positions α and β are located within the same triangular mesh ABC.

[0261] In this case, similarly to the example in FIG. 11, at time Tβ, the received encoded object polar coordinate data of reference viewpoints A to C is used to perform interpolation processing for listening position β.

[0262] At this time, it is assumed that the object absolute coordinate position information OBJPOS1 is obtained by the interpolation process. At time Tβ, the object absolute coordinate position information OBJPOS1 is further converted to polar coordinates based on the listening position β at time Tβ, and rendering is performed based on the resulting polar coordinate position information OBJPOS1'.

[0263] Also, at time Tβ, viewpoint selection information for when the listener is at listening position β is generated for the next frame, etc., and transmitted to the server 11. In this example, viewpoint selection information indicating reference viewpoints A to C is transmitted at time Tβ.

[0264] Furthermore, suppose that in response to the viewpoint selection information transmitted at time Tβ, the server 11 transmits encoded object polar coordinate data, and the encoded object polar coordinate data is received by the client 12 at time Tγ.

[0265] Also, it is assumed that the listening position γ at time Tγ is the position indicated by the arrow F31.

[0266] In this case, the listening position γ is located in a triangular mesh BCD that is different from the triangular mesh ABC that contains the listening position β.

[0267] Therefore, interpolation processing for listening position γ requires object polar coordinate encoded data for reference viewpoints B to D, but since object polar coordinate encoded data for reference viewpoint D has not been received at time Tγ, interpolation processing cannot be performed.

[0268] Therefore, at time Tγ, the received encoded object polar coordinate data is discarded, and rendering is performed using the polar coordinate position information OBJPOS1' obtained at the immediately preceding time Tβ. In other words, at time Tγ, rendering is performed using the object absolute coordinate position information OBJPOS1 at the immediately preceding time Tβ and the listener position information indicating the listening position β.

[0269] By doing this, it is possible to avoid performing interpolation processing with an inappropriate positional relationship at time Tγ. That is, it is possible to obtain playback audio data with an appropriate positional relationship between the object and the listening position. Therefore, although there may be a slight sense of delay, it is possible to suppress deterioration in the sound quality of the content.

[0270] <Configuration example of content playback system> Here, a more detailed embodiment of the content playback system to which the present technology described above is applied will be described.

[0271] Fig. 14 is a diagram showing an example of the configuration of a content playback system to which the present technology is applied. Note that in Fig. 14, parts corresponding to those in Fig. 1 are given the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0272] The content reproduction system shown in FIG. 14 includes a server 11 that distributes content, and a client 12 that receives the content distributed from the server 11.

[0273] Server 11 also includes configuration information recording unit 101 , configuration information transmission unit 21 , recording unit 102 , and encoded data transmission unit 22 .

[0274] 2, and supplies the recorded system configuration information to the configuration information sending unit 21. Note that a part of the recording unit 102 may be configured as the configuration information recording unit 101.

[0275] The recording unit 102 records, for example, encoded audio data obtained by encoding audio data of objects that make up the content, object polar coordinate encoded data of each object for each reference viewpoint, encoding gain information, and the like.

[0276] The recording unit 102 supplies the recorded encoded audio data, object polar coordinate encoded data, encoding gain information, etc. to the encoded data sending unit 22 in response to a request or the like.

[0277] The client 12 also includes a listener position information acquisition unit 41 , a viewpoint selection unit 42 , a communication unit 111 , a decoding unit 45 , a position calculation unit 112 , and a rendering processing unit 113 .

[0278] The communication unit 111 corresponds to the configuration information acquisition unit 43 and the encoded data acquisition unit 44 shown in FIG. 1, and communicates with the server 11 to send and receive various types of data.

[0279] For example, the communication unit 111 transmits viewpoint selection information supplied from the viewpoint selection unit 42 to the server 11, and receives system configuration information and a bitstream transmitted from the server 11. In other words, the communication unit 111 functions as a reference viewpoint information acquisition unit that acquires, from the server 11, the system configuration information, and the object polar coordinate encoded data and encoding gain information included in the bitstream.

[0280] The position calculation unit 112 generates polar coordinate position information indicating the position of the object based on the object polar coordinate position information supplied from the decoding unit 45 and the system configuration information supplied from the communication unit 111, and supplies the generated information to the rendering processing unit 113.

[0281] Furthermore, the position calculation unit 112 performs gain adjustment on the audio data of the object supplied from the decoding unit 45 , and supplies the audio data after gain adjustment to the rendering processing unit 113 .

[0282] The position calculation unit 112 includes a coordinate conversion unit 46 , a coordinate axis conversion processing unit 47 , an object position calculation unit 48 , and a polar coordinate conversion unit 49 .

[0283] The rendering processing unit 113 performs rendering processing such as VBAP based on the polar coordinate position information and audio data supplied from the polar coordinate conversion unit 49, and generates and outputs playback audio data for playing back the sound of the content.

[0284] <Description of System Configuration Information Transmission Process and System Configuration Information Reception Process> Next, the operation of the content reproduction system shown in FIG. 14 will be described.

[0285] First, the system configuration information transmission process by the server 11 and the system configuration information reception process by the client 12 will be described with reference to the flowchart of FIG.

[0286] For example, when a connection is established between the server 11 and the client 12 for the purpose of distributing predetermined content, the system configuration information transmission process is started and the process of step S11 is performed.

[0287] That is, in step S11, the configuration information sending unit 21 reads the system configuration information of the requested content from the configuration information recording unit 101, and transmits the read system configuration information to the client 12, and the system configuration information transmission process ends.

[0288] For example, the system configuration information is prepared in advance and is sent to the client 12 immediately after the content playback system starts operating, i.e., immediately after a connection between the server 11 and the client 12 is established, and before the encoded audio data, etc. is sent.

[0289] Then, in step S21, the communication unit 111 of the client 12 receives the system configuration information sent from the server 11 and supplies it to the viewpoint selection unit 42, the decoding unit 45, the coordinate axis transformation processing unit 47, and the object position calculation unit 48.

[0290] The timing at which the communication unit 111 acquires the system configuration information from the server 11 may be any timing as long as it is before the start of content playback.

[0291] In step S22, the viewpoint selection unit 42, the decoding unit 45, the coordinate axis transformation processing unit 47, and the object position calculation unit 48 hold the system configuration information supplied from the communication unit 111, and the system configuration information reception process ends.

[0292] In this way, by the client 12 acquiring and storing the system configuration information before playing back the content, it becomes possible to appropriately select a reference viewpoint using the system configuration information.

[0293] <Description of viewpoint selection information transmission process> 15, the client 12 continues to perform a viewpoint selection information transmission process and a playback audio data generation process until the content playback is completed. After the system configuration information transmission process, the server 11 performs a provision process.

[0294] The viewpoint selection information transmission process, playback audio data generation process, and provision process will be described below.

[0295] First, the viewpoint selection information transmission process performed by the client 12 will be described with reference to the flowchart of FIG.

[0296] For example, when the viewpoint selection information transmission process is started, the viewpoint selector 42 starts a polling timer for specifying the timing (polling time) for transmitting the viewpoint selection information.

[0297] For example, when object polar coordinate encoded data is acquired for each frame of content to be played (playback audio data), the time when it is time to acquire object polar coordinate encoded data of the next frame to be played is the polling time. When the polling time arrives, the viewpoint selection unit 42 performs the process of step S51.

[0298] In step S51, the viewpoint selection unit 42 acquires listener position information from the listener position information acquisition unit 41.

[0299] That is, the listener position information acquisition unit 41 acquires listener position information in response to operations by the listener, etc., and outputs the information to the viewpoint selection unit 42, the decoding unit 45, the object position calculation unit 48, and the polar coordinate conversion unit 49. The viewpoint selection unit 42 acquires the listener position information output from the listener position information acquisition unit 41 in this manner.

[0300] In step S52, the viewpoint selection unit 42 selects a plurality of reference viewpoints based on the system configuration information supplied from the communication unit 111 and held therein, and the listener position information acquired from the listener position information acquisition unit 41.

[0301] For example, when three-point interpolation is performed in the object position calculation unit 48, the viewpoint selection unit 42 selects three reference viewpoints that surround the listening position indicated by the listener position information from among the multiple reference viewpoints indicated by the system configuration information. In other words, one triangular mesh that includes the listening position is selected from among multiple triangular meshes, and the three reference viewpoints that make up that triangular mesh are selected.

[0302] Furthermore, for example, when two-point interpolation is performed in object position calculation unit 48, viewpoint selection unit 42 selects two reference viewpoints that sandwich the listening position from among the multiple reference viewpoints indicated by the system configuration information. In other words, the reference viewpoints are selected so that the listening position is located on the line connecting the two selected reference viewpoints.

[0303] In step S53, the viewpoint selection unit 42 generates viewpoint selection information indicating the reference viewpoint selected in step S52, and supplies the generated information to the communication unit 111. For example, the viewpoint selection information may be index information indicating each selected reference viewpoint.

[0304] In step S54, the communication unit 111 transmits the viewpoint selection information supplied from the viewpoint selection unit 42 to the server 11. This results in a request for transmission of object polar coordinate encoded data at the reference viewpoint indicated by the viewpoint selection information.

[0305] In step S55, the client 12 determines whether or not to end the processing it is currently performing. For example, in step S55, it determines to end the processing when the user issues an instruction to end the playback of the content, or when a data end signal indicating that transmission of all content data has been completed is received from the server 11.

[0306] If it is determined in step S55 that the process is not yet finished, the process returns to step S51, and the above-described process is repeated. In this case, the process of step S51 is performed at the timing of the next polling time.

[0307] On the other hand, if it is determined in step S55 that the processing is to be ended, the client 12 ends the session with the server 11, stops the processing being performed by each unit, and ends the viewpoint selection information transmission processing.

[0308] In this way, the client 12 selects a reference viewpoint according to the listening position and transmits viewpoint selection information indicating the selection result to request transmission of appropriate object polar coordinate encoded data. In this way, content can be played back according to the intentions of the content creator, according to the listening position.

[0309] Moreover, in the content playback system, by selecting the reference viewpoint on the client 12 side, it is possible to reduce the processing load on the server 11 compared to when the reference viewpoint is selected by the server 11. Such a reduction in the processing load on the server 11 is particularly useful when the server 11 distributes content to many clients 12 simultaneously.

[0310] <Description of the provision process> Next, the providing process performed by the server 11 will be described with reference to the flowchart of Fig. 17. This providing process is repeated until the reproduction of the content is completed.

[0311] In step S 81 , the configuration information sending unit 21 receives viewpoint selection information sent from the client 12 and supplies it to the coded data sending unit 22 .

[0312] The encoded data sending unit 22 reads out the object polar coordinate encoded data and encoding gain information for each object of the reference viewpoint indicated by the viewpoint selection information supplied from the configuration information sending unit 21 from the recording unit 102, and also reads out the encoded audio data of each object of the content.

[0313] In step S82, the coded data sending unit 22 multiplexes the coded object polar coordinate data, coding gain information, and coded audio data read from the recording unit 102 to generate a bit stream.

[0314] In step S83, the coded data sending unit 22 sends the generated bit stream to the client 12, and the providing process ends. This completes the distribution of the content to the client 12.

[0315] In this way, the server 11 generates a bitstream including object polar coordinate encoded data and encoding gain information according to the viewpoint selection information, and transmits it to the client 12. In this way, content playback based on the intentions of the content creator can be realized for each client 12.

[0316] <Description of playback audio data generation process> When the server 11 performs the provision process and transmits the bitstream, the client 12 performs the playback audio data generation process.

[0317] The playback audio data generation process performed by the client 12 will be described below with reference to the flowchart of FIG.

[0318] In step S 111 , the communication unit 111 receives the bit stream transmitted from the server 11 and supplies it to the decoding unit 45 .

[0319] In step S112, the decoding unit 45 extracts the object polar coordinate encoded data, the encoding gain information, and the encoded audio data from the bitstream supplied from the communication unit 111 and performs decoding.

[0320] In step S113, the decoding unit 45 determines whether or not the listener is located within the triangular mesh based on the listener position information, the system configuration information, and the decoding result of the object polar coordinate encoded data.

[0321] That is, for example, at the timing when a bitstream is received, the decoding unit 45 acquires new listener position information indicating the listener's position at the current time from the listener position information acquisition unit 41. The current time here refers to a time after the time when viewpoint selection information was last transmitted to the server 11.

[0322] Furthermore, it is possible to identify which reference viewpoint's object polar coordinate encoded data is included in the bit stream from the results of demultiplexing the bit stream and decoding the object polar coordinate encoded data.

[0323] In the following description, the reference viewpoint corresponding to the object polar coordinate encoded data included in the received bitstream will also be referred to as the "received reference viewpoint." In other words, the bitstream includes the object polar coordinate encoded data of the received reference viewpoint.

[0324] For example, when three-point interpolation is performed, the decoding unit 45 determines that the listener is within a triangular mesh (hereinafter also referred to as a receiving triangular mesh) formed from three receiving reference viewpoints when the current listening position is included within the triangular mesh.

[0325] Therefore, for example, in the state shown in FIG. 11, it is determined in step S113 that the listener is inside the triangular mesh, and in the state shown in FIG. 12, it is determined in step S113 that the listener is not inside the triangular mesh.

[0326] If it is determined in step S113 that the listener is within the triangular mesh, then the process proceeds to step S114.

[0327] In this case, the decoding unit 45 supplies the object polar coordinate position information obtained by decoding to the coordinate conversion unit 46, supplies the gain information obtained by decoding to the object position calculation unit 48, and supplies the audio data obtained by decoding to the polar coordinate conversion unit 49.

[0328] When two-point interpolation is performed, if the current listening position is located between two reception reference viewpoints, the process proceeds to step S114.

[0329] In step S114, the coordinate conversion unit 46 performs coordinate conversion on the object polar coordinate position information of each object supplied from the decoding unit 45, and supplies the resulting object absolute coordinate position information to the coordinate axis conversion processing unit 47.

[0330] For example, in step S114, for each reference viewpoint, the above-mentioned formula (1) is calculated based on the object polar coordinate position information for each object, and the object absolute coordinate position information is calculated.

[0331] In step S 115 , the coordinate axis conversion processing unit 47 performs coordinate axis conversion processing on the object absolute coordinate position information supplied from the coordinate conversion unit 46 based on the system configuration information supplied from the communication unit 111 .

[0332] The coordinate axis transformation processing unit 47 performs coordinate axis transformation processing on each object for each reference viewpoint, and supplies the resulting object absolute coordinate position information indicating the position of the object in the common absolute coordinate system to the object position calculation unit 48. For example, in step S115, a calculation similar to that of the above-mentioned equation (3) is performed to calculate the object absolute coordinate position information.

[0333] In step S116, the object position calculation unit 48 performs interpolation processing based on the system configuration information supplied from the communication unit 111, the listener position information supplied from the listener position information acquisition unit 41, the object absolute coordinate position information supplied from the coordinate axis transformation processing unit 47, and the gain information supplied from the decoding unit 45.

[0334] For example, at the timing when a bitstream is received, the object position calculation unit 48 acquires new listener position information indicating the position of the listener at the current time from the listener position information acquisition unit 41.

[0335] Then, the object position calculation unit 48 performs the above-described three-point interpolation as an interpolation process for each object, and calculates the final object absolute coordinate position information and gain information.

[0336] Specifically, the object position calculation unit 48 performs calculations similar to the above-mentioned equations (6) to (11) based on the reference viewpoint position information included in the system configuration information and the listener position information, to determine the internal division ratios (m, n) and (k, l).

[0337] Then, the object position calculation unit 48 performs three-point interpolation processing by performing calculations similar to the above-mentioned equations (12) to (21) based on the determined internal division ratios (m, n) and internal division ratios (k, l) and the object absolute coordinate position information and gain information of each reference viewpoint.

[0338] Furthermore, for example, when two-point interpolation is performed, the object position calculation unit 48 determines the apportionment ratio (m:n) by performing a calculation similar to the above-mentioned equation (4) based on the reference viewpoint position information included in the system configuration information and the listener position information.

[0339] Then, the object position calculation unit 48 performs a two-point interpolation process by performing a calculation similar to the above-mentioned equation (5) based on the calculated proportion ratio (m:n) and the object absolute coordinate position information and gain information of the two reference viewpoints.

[0340] In two-point or three-point interpolation, the interpolation process may be performed by weighting the object absolute coordinate position information and gain information of the desired reference viewpoint.

[0341] When the interpolation process is performed in this manner and the final object absolute coordinate position information and gain information are obtained, the object position calculation unit 48 supplies the obtained object absolute coordinate position information and gain information to the polar coordinate conversion unit 49 .

[0342] In step S117, the polar coordinate conversion unit 49 performs polar coordinate conversion on the object absolute coordinate position information supplied from the object position calculation unit 48 based on the listener position information supplied from the listener position information acquisition unit 41, thereby generating polar coordinate position information.

[0343] The listener position information used at this time is assumed to be that acquired from the listener position information acquisition unit 41 at the timing when the bit stream is received, for example.

[0344] Furthermore, the polar coordinate conversion unit 49 performs gain adjustment on the audio data of each object supplied from the decoding unit 45 based on the gain information of each object supplied from the object position calculation unit 48 .

[0345] The polar coordinate conversion unit 49 supplies the polar coordinate position information obtained by the polar coordinate conversion and the audio data of each object obtained by the gain adjustment to the rendering processing unit 113, and then the process proceeds to step S119.

[0346] If it is determined in step S113 that the listener is not within the triangular mesh, the process proceeds to step S118.

[0347] In this case, the listening position at the current time is located outside the receiving triangular mesh consisting of the receiving reference viewpoints that surround the listening position at the specified time, which are different from the current time, i.e., selected by the viewpoint selection unit 42 at a specified time before the current time.

[0348] If it is determined that the listener is not within the triangular mesh, the decoding unit 45 discards the object polar coordinate position information and gain information obtained by decoding, and supplies the audio data obtained by decoding to the polar coordinate conversion unit 49.

[0349] In step S118, the polar coordinate conversion unit 49 supplies (outputs) the last calculated polar coordinate position information, that is, the polar coordinate position information generated in the last (immediately preceding) step S117, as is to the rendering processing unit 113. In other words, in the subsequent rendering process, the same polar coordinate position information as the previous one is used.

[0350] In addition, the polar coordinate conversion unit 49 performs gain adjustment on the audio data of each object supplied from the decoding unit 45 based on the gain information of each object generated in the last (immediately before) step S116 and supplied from the object position calculation unit 48.

[0351] The polar coordinate conversion unit 49 then supplies the audio data of each object obtained by gain adjustment to the rendering processing unit 113, after which the process proceeds to step S119. In this case, the gain adjustment of the latest audio data is performed based on the same gain information as that used in the previous gain adjustment.

[0352] After the processing of step S117 or step S118 is performed, the processing of step S119 is then performed.

[0353] In step S119, the rendering processing unit 113 performs rendering processing such as VBAP based on the polar coordinate position information of each object and the audio data supplied from the polar coordinate conversion unit 49, and outputs the playback audio data obtained as a result.

[0354] For example, a speaker or the like downstream of the rendering processing unit 113 reproduces the sound of the content based on the reproduced audio data.

[0355] In step S120, the client 12 determines whether or not to end the processing it is currently performing. For example, in step S120, it is determined to end the processing when the user issues an instruction to end the playback of the content, or when a data end signal indicating that transmission of all content data has been completed is received from the server 11.

[0356] If it is determined in step S120 that the process is not yet finished, the process returns to step S111, and the above-described process is repeated. In this case, the communication unit 111 enters a state of waiting to receive a bitstream, and when a new bitstream is transmitted from the server 11, the process of step S111 is newly performed.

[0357] On the other hand, if it is determined in step S120 that the process is to be ended, the client 12 ends the session with the server 11, stops the processes being performed by each unit, and ends the playback audio data generation process.

[0358] In addition, before the rendering process, the rendering processing unit 113 or the polar coordinate conversion unit 49 may perform processing on the audio data of the object according to the playback mode based on the listener position information and the information indicating the playback mode included in the system configuration information.

[0359] In such a case, for example, the audio data of an object located at a position overlapping with the listening position is subjected to attenuation processing such as gain adjustment, or the audio data is muted by being replaced with zero data. Also, for example, the audio data of an object located at a position overlapping with the listening position is made to be output as sound from all channels (speakers).

[0360] In this way, the client 12 performs interpolation processing based on the information on each reference viewpoint included in the received bitstream, and obtains the object absolute coordinate position information and gain information for each object.

[0361] This allows for object placement based on the intentions of the content creator, according to the listening position, rather than simply the physical relationship between the listener and the object. This allows content playback based on the intentions of the content creator, and fully conveys the enjoyment and realism of the content to the listener.

[0362] Furthermore, if the current listening position is not included in the receiving triangular mesh, the client 12 can prevent interpolation processing for inappropriate positional relationships by simply using the same polar coordinate position information and gain information as used previously, thereby achieving higher quality content playback.

[0363] Second Embodiment <Selecting a reference viewpoint> Furthermore, for example, when content is transmitted in real time by server 11, in addition to the reference viewpoint of the triangular mesh including the listening position, other reference viewpoints may also be selected, and object polar coordinate encoding data for those reference viewpoints may be requested.

[0364] In such a case, the reference viewpoint is selected as shown in FIG.

[0365] In the example shown in FIG. 19, listening position α at time Tα is the position indicated by arrow F51, and listening position β at time Tβ is the position indicated by arrow F52.

[0366] In this example, at time Tα, the listening position α is located within the triangular mesh ABC, so the viewpoint selector 42 normally selects one of the reference viewpoints A to C.

[0367] However, at time Tβ, the listener has already moved to a position (listening position β) within the triangular mesh BCD, and therefore new object polar coordinate encoded data for the reference viewpoint D is required to perform the interpolation process.

[0368] Therefore, if the listening position α at time Tα is near the boundary (side) of the triangular mesh ABC that contains the listening position α, that is, near another triangular mesh adjacent to the triangular mesh ABC, additional reference viewpoints that constitute the other triangular meshes adjacent to the triangular mesh ABC may also be selected.

[0369] In the following, the additionally selected reference viewpoints will also be referred to as additional reference viewpoints. Also, the triangular mesh selected when generating the viewpoint selection information, i.e., the triangular mesh consisting of three selected reference viewpoints that are not additional reference viewpoints, will also be referred to as selected triangular mesh.

[0370] Whether or not to select an additional reference viewpoint, i.e., whether or not to request object polar coordinate encoded data of the additional reference viewpoint, may be determined based on at least one of the following: delay time due to transmission, etc.; distance from the listening position to the edge of the selected triangular mesh (the positional relationship between the selected triangular mesh and the listening position); speed of movement of the listener; and direction of movement of the listener.

[0371] Here, the delay time can be, for example, the time from transmitting viewpoint selection information to receiving a bitstream for the last processed frame, etc. Furthermore, the moving speed and moving direction of the listener can be calculated from the listener position information at each time.

[0372] For example, an additional reference viewpoint may be selected when a circle of a predetermined radius centered at the listening position α intersects with an edge of the selected triangular mesh, in other words, when the distance from the listening position α to the edge of the selected triangular mesh is equal to or less than a predetermined threshold. The radius of the circle (predetermined threshold) may be determined based on, for example, the moving speed and direction of the listener, delay time, etc.

[0373] Specifically, in the example of FIG. 19, it is assumed that the distance from the listening position α indicated by the arrow F51 to the side BC of the selected triangular mesh ABC is equal to or less than a predetermined threshold.

[0374] In this case, the reference viewpoint D of the triangular mesh BCD, which has the side BC as a side (has a side BC in common with the selected triangular mesh ABC) and is adjacent to the selected triangular mesh ABC, is selected as the additional reference viewpoint.

[0375] Then, viewpoint selection information indicating the additional reference viewpoint D in addition to the reference viewpoints A to C is generated.

[0376] Then, at time Tβ, object polar coordinate encoded data for reference viewpoints A to D is received, so even if the listener moves to triangular mesh BCD, interpolation processing can be performed based on the received object polar coordinate encoded data.

[0377] As another example, the range that the listener can reach (move) until the time when the next encoded object polar coordinate data is expected to be received, that is, the estimated value (estimated time) of time Tβ, may be calculated based on the listener's moving speed, moving direction, delay time, etc. In such a case, an additional reference viewpoint may be selected when the calculated range (area) intersects with an edge of the selected triangular mesh.

[0378] The number of selected additional reference viewpoints may be one or more. For example, reference viewpoints constituting two or more triangular meshes adjacent to the triangular mesh containing the current listening position may be selected as additional reference viewpoints.

[0379] In this way, by selecting an additional reference viewpoint as appropriate depending on the situation, it is possible to obtain reproduced audio data in which the object and the listening position have an appropriate positional relationship without causing a sense of delay, although the amount of data transmission may increase. Note that the selection of an additional reference viewpoint as described above may also be performed on the server 11 side.

[0380] <Description of viewpoint selection information transmission process> Furthermore, when an additional reference viewpoint is selected as needed, the client 12 performs viewpoint selection information transmission processing shown in Fig. 20. Hereinafter, the viewpoint selection information transmission processing performed by the client 12 will be described with reference to the flowchart in Fig. 20.

[0381] The processes in steps S151 and S152 are the same as those in steps S51 and S52 in FIG. 16, and therefore will not be described further.

[0382] In step S153, the viewpoint selection unit 42 determines whether to select an additional reference viewpoint based on the result of the reference viewpoint selection in step S152, the delay time of transmission, etc., and the moving speed and direction of the listener.

[0383] For example, as described with reference to FIG. 19, the viewpoint selection unit 42 determines to select an additional reference viewpoint when the distance from the listening position to the side of the selected triangular mesh is equal to or less than a predetermined threshold.

[0384] If it is determined in step S153 that an additional reference viewpoint is not to be selected, the process of step S154 is not performed, and the process then proceeds to step S155.

[0385] On the other hand, if it is determined in step S153 that an additional reference viewpoint is to be selected, the viewpoint selection unit 42 selects the additional reference viewpoint in step S154.

[0386] At this time, the viewpoint selection unit 42 selects an additional reference viewpoint based on at least one of the result of the reference viewpoint selection in step S152 (the positional relationship between the listening position at the current time and the selected triangular mesh), the direction and speed of movement of the listener, and the delay time of transmission, etc.

[0387] For example, as described with reference to Fig. 19, the viewpoint selection unit 42 selects, as an additional reference viewpoint, a reference viewpoint that constitutes a triangular mesh that has a common side with the selected triangular mesh, i.e., a triangular mesh that is adjacent to the selected triangular mesh. Note that two or more reference viewpoints may be selected as additional reference viewpoints.

[0388] If the process of step S154 has been performed or if it is determined in step S153 that an additional reference viewpoint is not to be selected, the viewpoint selection unit 42 generates viewpoint selection information and supplies it to the communication unit 111 in step S155.

[0389] For example, if it is determined in step S153 that an additional reference viewpoint is not to be selected, the viewpoint selection unit 42 generates viewpoint selection information indicating the reference viewpoint selected in step S152.

[0390] On the other hand, when the process of step S154 is performed, the viewpoint selection unit 42 generates viewpoint selection information indicating the reference viewpoint selected in step S152 and the additional reference viewpoint selected in step S154.

[0391] Once the viewpoint selection information is generated in this manner, the processing of steps S156 and S157 is then carried out and the viewpoint selection information transmission processing is completed. However, since these processing steps are the same as the processing steps S54 and S55 in FIG. 16, their explanation will be omitted.

[0392] In this way, the client 12 generates viewpoint selection information by appropriately selecting additional reference viewpoints. By doing so, even if the listener moves, it is possible to prevent the positional relationship between the listening position at the time of receiving the bitstream and the reception reference viewpoint from becoming inappropriate, making it impossible to perform interpolation processing, etc.

[0393] <Description of playback audio data generation process> Furthermore, when the viewpoint selection information transmission process described with reference to FIG. 20 is performed, the server 11 performs the provision process described with reference to FIG.

[0394] In this case, in step S82, a bitstream is generated that includes not only the reference viewpoint of the triangular mesh including the listening position, but also object polar coordinate encoded data and encoding gain information for the additional reference viewpoint as appropriate, according to the viewpoint selection information.

[0395] When such a bitstream is transmitted, the client 12 performs the playback audio data generation process shown in FIG.

[0396] The playback audio data generation process performed by the client 12 will be described below with reference to the flowchart in Fig. 21. Note that the processes from step S181 to step S183 are similar to the processes from step S111 to step S113 in Fig. 18, and therefore their description will be omitted where appropriate.

[0397] Here, in step S183, if the current listening position is included in a triangular mesh formed from three reference viewpoints that are not additional reference viewpoints among the reference viewpoints indicated by the viewpoint selection information, i.e., the three reference viewpoints selected in step S152 of Figure 20, it is determined that the listener is within the triangular mesh.

[0398] If it is determined in step S183 that the listener is inside the triangular mesh, then steps S185 to S188 are performed, and the process proceeds to step S 190. Note that the processes of steps S185 to S188 are the same as the processes of steps S114 to S117 in Fig. 18, and therefore a description thereof will be omitted.

[0399] Also, if it is determined in step S183 that the listener is not within the triangular mesh, in step S184 the decoding unit 45 determines whether there is information about the triangular mesh that includes the listening position at the time the bitstream was received (current time).

[0400] Here, it is determined that triangular mesh information is present when there is an additional reference viewpoint among the receiving reference viewpoints and the current listening position is included within the triangular mesh formed from the three receiving reference viewpoints including that additional reference viewpoint.

[0401] If it is determined in step S184 that there is information about a triangular mesh that includes the listening position, the processes of steps S185 to S188 are performed, and the process proceeds to step S190.

[0402] In this case, the processes of steps S185 to S188 are performed using the object polar coordinate position information and gain information of the three reception reference viewpoints including the additional reference viewpoint that form a triangular mesh including the current listening position.

[0403] Furthermore, if it is determined in step S184 that there is no information about a triangular mesh that includes the listening position, the process proceeds to step S189.

[0404] In step S189, the same process as in step S118 in Fig. 18 is performed, and the process proceeds to step S190. That is, the polar coordinate position information generated last is supplied to the rendering processing unit 113 as is.

[0405] After the processing of step S188 or step S189 is performed, the processing of steps S190 and S191 is performed, and the playback audio data generation processing ends. However, since these processing steps are similar to steps S119 and S120 in FIG. 18, their description will be omitted.

[0406] In this way, the client 12 performs interpolation processing using the object polar coordinate position information and gain information of the additional reference viewpoint as needed. In this way, even if the listener moves, it is possible to prevent the interpolation processing from becoming impossible.

[0407] Third Embodiment <Selecting a reference viewpoint> Incidentally, the content distributed by the server 11 may be archive content in which object polar coordinate encoded data and encoded audio data for all playback times (frames) are created in advance before the content is distributed. In such cases, the server 11 can extract and transmit data for any playback time.

[0408] Therefore, taking into consideration delay times such as network delays, data may be transmitted with a playback time that matches the expected reception time at the client 12. By doing so, it is possible to transmit to the client 12 object polar coordinate position information and gain information that are close to the actual playback time at the client 12.

[0409] Specifically, for example, as shown in FIG. 22, listening position α at time Tα is the position indicated by arrow F61, and listening position β at time Tβ is the position indicated by arrow F62.

[0410] In this example, at time Tα, the listening position α is located within the triangular mesh ABC, so the viewpoint selection unit 42 selects reference viewpoints A to C.

[0411] In addition, the viewpoint selection unit 42 calculates the delay time of transmission, etc., from the measurement results of the time from sending viewpoint selection information for the last processed frame, etc. to receiving the bit stream, and estimates the time Tβ, which is the time when the data is received, based on this delay time.

[0412] Furthermore, the viewpoint selection unit 42 takes the estimated time Tβ into consideration and also selects an additional reference viewpoint when the listening position α is near the boundary (side) of the triangular mesh ABC that includes the listening position α, for example, as in the case of Fig. 19. In this example, the reference viewpoint D is selected as the additional reference viewpoint.

[0413] Then, the viewpoint selection unit 42 generates viewpoint selection information indicating the selected reference viewpoints A to D, including requested time information indicating the playback time (frame) such as the presentation timestamp corresponding to the estimated time Tβ, more specifically, the time Tβ.

[0414] The server 11 then transmits encoded audio data corresponding to time Tα, and object polar coordinate encoded data and encoding gain information for reference viewpoints A to D corresponding to time Tβ indicated by the requested time information. The encoded audio data corresponding to time Tα here refers to the encoded audio data with the next playback time following the encoded audio data with the predetermined playback time last transmitted by the server 11.

[0415] By generating such viewpoint selection information, the client 12 can obtain object polar coordinate encoded data and encoding gain information at time Tβ for reference viewpoints B to D that form a triangular mesh BCD that includes the listening position β.

[0416] Therefore, at the actual time Tβ, interpolation and rendering processes are performed as in the example of Figure 19, and reproduced audio data can be obtained in which the object and the listening position have an appropriate positional relationship without causing any sense of delay.

[0417] <Description of viewpoint selection information transmission process> Furthermore, when viewpoint selection information is generated as shown in Fig. 22, the client 12 performs viewpoint selection information transmission processing shown in Fig. 23. Below, the viewpoint selection information transmission processing performed by the client 12 will be described with reference to the flowchart in Fig. 23.

[0418] When the viewpoint selection information transmission process is started, listener position information is acquired in step S221, and a delay time (transmission delay) is estimated in step S222. Note that the process of step S221 is similar to the process of step S151 in Fig. 20, and therefore a description thereof will be omitted.

[0419] In step S 222 , the viewpoint selection unit 42 estimates the delay time from the current time until the next reception of the bitstream from the server 11 .

[0420] Specifically, the measurement result of the time from when the process of step S227 is last performed until the bitstream is received (until when step S181 in FIG. 21 is last performed) is set as the delay time, etc. Furthermore, the viewpoint selection unit 42 also estimates the estimated reception time (expected acquisition time) of the next bitstream from the estimated delay time.

[0421] Once the delay time is estimated, the processes of steps S223 to S225 are performed, but these processes are similar to the processes of steps S152 to S154 in Fig. 20, and therefore their description will be omitted. Note that in steps S223 to S225, the delay time estimated in step S222 may also be taken into consideration.

[0422] In step S226, the viewpoint selection unit 42 generates viewpoint selection information including requested time information indicating a playback time corresponding to the estimated reception time of the bitstream, and supplies the generated viewpoint selection information to the communication unit 111. This results in a request to transmit object polar coordinate encoded data and encoding gain information at the playback time corresponding to the estimated reception time.

[0423] In this case, for example, if the process of step S225 is not performed, viewpoint selection information indicating the reference viewpoint selected in step S223 is generated.

[0424] On the other hand, when the process of step S225 is performed, viewpoint selection information indicating the reference viewpoint selected in step S223 and the additional reference viewpoint selected in step S225 is generated.

[0425] Once the viewpoint selection information is generated, the processing of steps S227 and S228 is performed, and the viewpoint selection information transmission processing ends. However, since these processing steps are similar to the processing steps S156 and S157 in FIG. 20, their description will be omitted.

[0426] In this way, the client 12 generates viewpoint selection information including requested time information. In this way, the client 12 can obtain object polar coordinate encoded data and encoding gain information of the object at the playback time corresponding to the reception time of the bitstream. This makes it possible to obtain playback audio data in which the object and the listening position have an appropriate positional relationship, without causing a sense of delay.

[0427] In this embodiment, the server 11 also performs the provision processing described with reference to FIG.

[0428] However, in this case, in step S83, the encoded audio data with the next playback time after the encoded audio data with the playback time transmitted in the last (immediately preceding) step S83 is stored in the bitstream. Also, the bitstream stores the object polar coordinate encoded data and encoding gain information with the playback time indicated by the requested time information.

[0429] Furthermore, in this embodiment, when a bitstream is transmitted from the server 11, the client 12 performs the playback audio data generation process described with reference to Fig. 21. In this case, in steps S185 to S188, interpolation is performed based on the object polar coordinate position information and gain information at the playback time indicated by the request time information.

[0430] <Example of computer configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0431] FIG. 24 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes using a program.

[0432] In the computer, a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, and a RAM (Random Access Memory) 503 are interconnected by a bus 504.

[0433] An input / output interface 505 is further connected to the bus 504. An input unit 506, an output unit 507, a recording unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.

[0434] The input unit 506 includes a keyboard, a mouse, a microphone, an image sensor, etc. The output unit 507 includes a display, a speaker, etc. The recording unit 508 includes a hard disk, a nonvolatile memory, etc. The communication unit 509 includes a network interface, etc. The drive 510 drives a removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0435] In a computer configured as described above, the CPU 501 performs the above-described series of processes by, for example, loading a program recorded in the recording unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executing it.

[0436] The program executed by the computer (CPU 501) can be provided by being recorded on a removable recording medium 511 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0437] In a computer, a program can be installed in the recording unit 508 via the input / output interface 505 by inserting a removable recording medium 511 into the drive 510. The program can also be received by the communication unit 509 via a wired or wireless transmission medium and installed in the recording unit 508. Alternatively, the program can be installed in the ROM 502 or the recording unit 508 in advance.

[0438] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0439] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0440] For example, this technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0441] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by multiple devices.

[0442] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0443] Furthermore, the present technology can also be configured as follows.

[0444] (1) a listener position information acquisition unit that acquires listener position information indicating a listening position; a viewpoint selection unit that selects, from among a plurality of reference viewpoints, a plurality of reference viewpoints that form an area including the listening position at a predetermined time; a reference viewpoint information acquisition unit that acquires viewpoint position information of the plurality of reference viewpoints and object position information of an object at the reference viewpoint for each of the plurality of reference viewpoints; If, at a time different from the predetermined time, the listening position is outside the area including the listening position at the predetermined time, calculating position information of the object at the listening position based on the object position information of the plurality of reference viewpoints that form the area including the listening position at the different times; or Output the position information of the object at the listening position that was last determined. an object position calculation unit; An information processing device comprising: (2) The reference viewpoint is a viewpoint that is preset by the content creator. An information processing device according to (1). (3) a rendering processing unit that performs rendering processing based on the position information of the object at the listening position and the audio data of the object; An information processing device according to (1) or (2). (4) the reference viewpoint information acquisition unit further acquires gain information of the object at the reference viewpoint for each of the plurality of reference viewpoints; When the listening position at the different time is outside the area including the listening position at the predetermined time, the object position calculation unit calculating gain information at the listening position based on gain information of a plurality of reference viewpoints that form the area including the listening position at the different times, and adjusting the gain of the audio data based on the calculated gain information; or or The gain of the audio data is adjusted based on the gain information at the listening position that was last obtained. (3) An information processing device according to the present invention. (5) the reference viewpoint information acquisition unit acquires the object position information of a plurality of the reference viewpoints that form the area including the listening position at the predetermined time; If, at the different time, the listening position is outside the area including the listening position at the predetermined time, the object position calculation unit outputs the position information of the object at the listening position that was last calculated. An information processing device according to any one of (1) to (4). (6) the viewpoint selection unit selects a plurality of reference viewpoints that form the area including the listening position at the predetermined time and a reference viewpoint that forms another area adjacent to the area, the reference viewpoint information acquisition unit acquires the object position information of the plurality of reference viewpoints selected by the viewpoint selection unit, When the listening position is within the other area at the different time, the object position calculation unit calculates the position information of the object at the listening position based on the object position information of the plurality of reference viewpoints that form the other area. An information processing device according to any one of (1) to (4). (7) The viewpoint selection unit selects the reference viewpoint that forms the other area based on at least one of the positional relationship between the listening position and the area at the predetermined time, the moving direction and moving speed of the listener, and the delay time until the object position information is acquired. (6) An information processing device according to (6). (8) the viewpoint selection unit estimates an expected acquisition time of the object position information based on the delay time; The reference viewpoint information acquisition unit acquires the object position information at a playback time corresponding to the assumed acquisition time of the plurality of reference viewpoints selected by the viewpoint selection unit. (7) An information processing device according to (7). (9) The object position calculation unit calculates the position information of the object at the listening position by interpolation processing based on the listener position information, the viewpoint position information of the plurality of reference viewpoints, and the object position information of the plurality of reference viewpoints. An information processing device according to any one of (1) to (8). (10) The object position calculation unit performs the interpolation process by weighting the object position information of the plurality of reference viewpoints. (9) An information processing device according to (9). (11) The object position calculation unit performs the interpolation process based on the viewpoint position information and the object position information of the two reference viewpoints sandwiching the listening position. The information processing device according to (9) or (10). (12) The object position calculation unit performs the interpolation process based on the viewpoint position information and the object position information of the three reference viewpoints surrounding the listening position. The information processing device according to (9) or (10). (13) The object position calculation unit calculates the gain information at the listening position by interpolation based on the listener position information, the viewpoint position information of the plurality of reference viewpoints, and the gain information of the plurality of reference viewpoints. (4) An information processing device according to the present invention. (14) The object position calculation unit calculates the position information of the object at the listening position by interpolation processing based on the listener position information, the viewpoint position information of the plurality of reference viewpoints, the object position information of the plurality of reference viewpoints, and listener direction information that indicates the direction of the listener's face at the reference viewpoint, the listener direction information being set for each of the plurality of reference viewpoints. An information processing device according to any one of (1) to (13). (15) The reference viewpoint information acquisition unit acquires configuration information including the viewpoint position information and the listener direction information of each of the plurality of reference viewpoints. (14) An information processing device according to (14). (16) The configuration information includes information indicating the number of the plurality of reference viewpoints and information indicating the number of the objects. (15) An information processing device according to (15). (17) The information processing device Obtaining listener position information indicating the listening position; selecting, from among a plurality of reference viewpoints, a plurality of reference viewpoints that form an area including the listening position at a predetermined time; Obtaining viewpoint position information of the plurality of reference viewpoints and object position information of the object at the reference viewpoint for each of the plurality of reference viewpoints; If, at a time different from the predetermined time, the listening position is outside the area including the listening position at the predetermined time, calculating position information of the object at the listening position based on the object position information of the plurality of reference viewpoints that form the area including the listening position at the different times; or Output the position information of the object at the listening position that was last determined. Information processing methods. (18) Obtaining listener position information indicating the listening position; selecting, from among a plurality of reference viewpoints, a plurality of reference viewpoints that form an area including the listening position at a predetermined time; Obtaining viewpoint position information of the plurality of reference viewpoints and object position information of the object at the reference viewpoint for each of the plurality of reference viewpoints; If, at a time different from the predetermined time, the listening position is outside the area including the listening position at the predetermined time, calculating position information of the object at the listening position based on the object position information of the plurality of reference viewpoints that form the area including the listening position at the different times; or Output the position information of the object at the listening position that was last determined. A program that causes a computer to perform a process. [Explanation of symbols]

[0445] 11 server, 12 client, 41 listener position information acquisition unit, 42 viewpoint selection unit, 45 decoding unit, 111 communication unit, 112 position calculation unit, 113 rendering processing unit

Claims

1. a listener position information acquisition unit that acquires listener position information indicating a listening position; a viewpoint selection unit that selects, from among a plurality of reference viewpoints, a plurality of reference viewpoints that form an area including the listening position at a predetermined time; a reference viewpoint information acquisition unit that acquires viewpoint position information of the plurality of reference viewpoints and object position information of an object at the reference viewpoint for each of the plurality of reference viewpoints; If, at a time different from the predetermined time, the listening position is outside the area including the listening position at the predetermined time, calculating position information of the object at the listening position based on the object position information of the plurality of reference viewpoints that form the area including the listening position at the different times; or Output the position information of the object at the listening position that was last determined. an object position calculation unit; An information processing device comprising:

2. The reference viewpoint is a viewpoint that is preset by the content creator. The information processing device according to claim 1 .

3. a rendering processing unit that performs rendering processing based on the position information of the object at the listening position and the audio data of the object; The information processing device according to claim 1 .

4. the reference viewpoint information acquisition unit further acquires gain information of the object at the reference viewpoint for each of the plurality of reference viewpoints; When the listening position at the different time is outside the area including the listening position at the predetermined time, the object position calculation unit calculating gain information at the listening position based on gain information of a plurality of reference viewpoints that form the area including the listening position at the different times, and adjusting the gain of the audio data based on the calculated gain information; or or The gain of the audio data is adjusted based on the gain information at the listening position that was last obtained. The information processing device according to claim 3 .

5. the reference viewpoint information acquisition unit acquires the object position information of a plurality of the reference viewpoints that form the area including the listening position at the predetermined time; If, at the different time, the listening position is outside the area including the listening position at the predetermined time, the object position calculation unit outputs the position information of the object at the listening position that was last calculated. The information processing device according to claim 1 .

6. the viewpoint selection unit selects a plurality of reference viewpoints that form the area including the listening position at the predetermined time and a reference viewpoint that forms another area adjacent to the area, the reference viewpoint information acquisition unit acquires the object position information of the plurality of reference viewpoints selected by the viewpoint selection unit, When the listening position is within the other area at the different time, the object position calculation unit calculates the position information of the object at the listening position based on the object position information of the plurality of reference viewpoints that form the other area. The information processing device according to claim 1 .

7. The viewpoint selection unit selects the reference viewpoint that forms the other area based on at least one of a positional relationship between the listening position and the area at the predetermined time, a moving direction and moving speed of the listener, and a delay time until the object position information is acquired. The information processing device according to claim 6 .

8. the viewpoint selection unit estimates an expected acquisition time of the object position information based on the delay time; The reference viewpoint information acquisition unit acquires the object position information at a playback time corresponding to the assumed acquisition time of the plurality of reference viewpoints selected by the viewpoint selection unit. The information processing device according to claim 7 .

9. The object position calculation unit calculates the position information of the object at the listening position by interpolation processing based on the listener position information, the viewpoint position information of the plurality of reference viewpoints, and the object position information of the plurality of reference viewpoints. The information processing device according to claim 1 .

10. The object position calculation unit performs the interpolation process by weighting the object position information of the plurality of reference viewpoints. The information processing device according to claim 9 .

11. The object position calculation unit performs the interpolation process based on the viewpoint position information and the object position information of the two reference viewpoints sandwiching the listening position. The information processing device according to claim 9 .

12. The object position calculation unit performs the interpolation process based on the viewpoint position information and the object position information of the three reference viewpoints surrounding the listening position. The information processing device according to claim 9 .

13. The object position calculation unit calculates the gain information at the listening position by interpolation based on the listener position information, the viewpoint position information of the plurality of reference viewpoints, and the gain information of the plurality of reference viewpoints. The information processing device according to claim 4 .

14. The object position calculation unit calculates the position information of the object at the listening position by interpolation processing based on the listener position information, the viewpoint position information of the plurality of reference viewpoints, the object position information of the plurality of reference viewpoints, and listener direction information that indicates the direction of the listener's face at the reference viewpoint, the listener direction information being set for each of the plurality of reference viewpoints. The information processing device according to claim 1 .

15. The reference viewpoint information acquisition unit acquires configuration information including the viewpoint position information and the listener direction information of each of the plurality of reference viewpoints. The information processing device according to claim 14.

16. The configuration information includes information indicating the number of the plurality of reference viewpoints and information indicating the number of the objects. The information processing device according to claim 15.

17. The information processing device Obtaining listener position information indicating the listening position; selecting, from among a plurality of reference viewpoints, a plurality of reference viewpoints that form an area including the listening position at a predetermined time; Obtaining viewpoint position information of the plurality of reference viewpoints and object position information of the object at the reference viewpoint for each of the plurality of reference viewpoints; If, at a time different from the predetermined time, the listening position is outside the area including the listening position at the predetermined time, calculating position information of the object at the listening position based on the object position information of the plurality of reference viewpoints that form the area including the listening position at the different times; or Output the position information of the object at the listening position that was last determined. Information processing methods.

18. Obtaining listener position information indicating the listening position; selecting, from among a plurality of reference viewpoints, a plurality of reference viewpoints that form an area including the listening position at a predetermined time; Obtaining viewpoint position information of the plurality of reference viewpoints and object position information of the object at the reference viewpoint for each of the plurality of reference viewpoints; If, at a time different from the predetermined time, the listening position is outside the area including the listening position at the predetermined time, calculating position information of the object at the listening position based on the object position information of the plurality of reference viewpoints that form the area including the listening position at the different times; or Output the position information of the object at the listening position that was last determined. A program that causes a computer to perform a process.

Images