Information processing device, method, and program

Abstract

The present technology relates to an information processing device, a method, and a program that make it possible to reproduce content based on the intention of content producer. The information processing device: acquires listener position information; acquires position information for a first reference viewpoint, object position information for a first object at the first reference viewpoint, position information for a second reference viewpoint, and object position information for the first object at the second reference viewpoint; acquires object position information for a second object; and calculates position information for the first object at the viewpoint of a listener on the basis of listener position information, the position information for the first reference viewpoint, the object position information at the first reference viewpoint, the position information for the second reference viewpoint, and the object position information at the second reference viewpoint. The present technology can be applied to an information processing device.

Description

[Technical Field]

[0001] The present technology relates to an information processing device, method, and program, and in particular to an information processing device, method, and program that enable content playback based on the intentions of a content creator. [Background technology]

[0002] For example, in a free viewpoint space, each object placed in the space using an absolute coordinate system is placed in a fixed position (see, for example, Patent Document 1).

[0003] In this case, the direction of each object as seen from any listening position is uniquely determined based on the listener's coordinate position in absolute space, the direction of their face, and the relationship to the object, and the gain of each object is uniquely determined based on the distance from the listening position, and the sound of each object is reproduced. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2019 / 198540 Summary of the Invention [Problem to be solved by the invention]

[0005] On the other hand, there are artistic aspects of the content and points that we want to emphasize to listeners.

[0006] For example, in the case of music content, it may be desirable for an object to be closer to the foreground, such as an instrument or player at a certain listening point that you want to emphasize in terms of the content, or a player that you want to emphasize in sports content.

[0007] Furthermore, there may be cases where it is desired to realize different placements for each object, such as always placing the object at a fixed position relative to the listener.

[0008] Given this, it is possible that the interest of the content may not be fully conveyed through a simple physical relationship between the listener and the object as described above.

[0009] The present technology has been made in light of such circumstances, and makes it possible to realize content playback based on the intentions of the content creator. [Means for solving the problem]

[0010] An information processing device according to one aspect of the present technology includes: a listener position information acquisition unit that acquires listener position information of a listener's viewpoint; a reference viewpoint information acquisition unit that acquires position information of a first reference viewpoint, object position information of a first object at the first reference viewpoint, position information of a second reference viewpoint, and object position information of the first object at the second reference viewpoint, and acquires object position information of a second object; and an object position calculation unit that calculates position information of the first object at the listener's viewpoint based on the listener position information, the position information of the first reference viewpoint, the object position information of the first object at the first reference viewpoint, the position information of the second reference viewpoint, and the object position information of the first object at the second reference viewpoint.

[0011] An information processing method or program according to one aspect of the present technology includes steps of acquiring listener position information of a listener's viewpoint, acquiring position information of a first reference viewpoint, object position information of a first object at the first reference viewpoint, position information of a second reference viewpoint, and object position information of the first object at the second reference viewpoint, acquiring object position information of a second object, and calculating position information of the first object at the listener's viewpoint based on the listener position information, the position information of the first reference viewpoint, and the object position information of the first object at the first reference viewpoint, the position information of the second reference viewpoint, and the object position information of the first object at the second reference viewpoint.

[0012] In one aspect of the present technology, listener position information of a listener's viewpoint is acquired, position information of a first reference viewpoint, object position information of a first object at the first reference viewpoint, position information of a second reference viewpoint, and object position information of the first object at the second reference viewpoint are acquired, and object position information of a second object is acquired, and position information of the first object at the listener's viewpoint is calculated based on the listener position information, the position information of the first reference viewpoint, and the object position information of the first object at the first reference viewpoint, the position information of the second reference viewpoint, and the object position information of the first object at the second reference viewpoint. [Brief explanation of the drawings]

[0013] [Figure 1] 10A and 10B are diagrams illustrating an absolute coordinate-based interpolation object and a polar coordinate-based fixed object. [Figure 2] FIG. 1 illustrates an example of the configuration of a content playback system. [Figure 3]FIG. 10 is a diagram illustrating a polar coordinate-based interpolation object. [Figure 4] FIG. 1 illustrates an example of the configuration of a content playback system. [Figure 5] FIG. 10 is a diagram illustrating an absolute coordinate base fixed object. [Figure 6] FIG. 1 illustrates an example of the configuration of a content playback system. [Figure 7] 10A and 10B are diagrams illustrating interpolation of object absolute coordinate position information; [Figure 8] 10A and 10B are diagrams illustrating the internal division ratio of a triangular mesh on the viewpoint side. [Figure 9] FIG. 10 is a diagram illustrating calculation of an object position. [Figure 10] 10A and 10B are diagrams illustrating calculation of object polar coordinate position information by interpolation. [Figure 11] FIG. 10 is a diagram illustrating an example of system configuration information. [Figure 12] FIG. 10 is a diagram illustrating an example of a bitstream format. [Figure 13] FIG. 10 is a diagram illustrating an example of a bitstream format. [Figure 14] FIG. 10 is a diagram illustrating an example of metadata for a polar coordinate-based fixed object. [Figure 15] FIG. 10 illustrates example metadata for a polar coordinate-based interpolation object. [Figure 16] FIG. 10 is a diagram illustrating an example of metadata of an absolute coordinate-based fixed object. [Figure 17] FIG. 1 illustrates an example of the configuration of a content playback system. [Figure 18] 10 is a flowchart illustrating a providing process. [Figure 19] 10 is a flowchart illustrating a playback audio data generation process. [Figure 20] 10 is a flowchart illustrating a polar coordinate position information generation process. [Figure 21] FIG. 1 illustrates an example of the configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments to which the present technology is applied will be described with reference to the drawings.

[0015] First Embodiment About this technology This technology provides multiple types of objects with different coordinate representations, such as origin positions and coordinate formats, allowing for more flexible object placement and content playback based on the content creator's intentions.

[0016] For example, in free viewpoint audio using Artistic Intent, any object position can be generated by interpolation using absolute coordinates for multiple reference viewpoints or polar coordinate data that takes into account the listener's orientation.

[0017] On the other hand, if you want to place an object that is always fixed in position relative to the listener, the above method cannot express an object placement that is independent of the listener because it takes into account the listener's orientation.

[0018] Furthermore, if it is desired to place an object at an absolute position within a free viewpoint space independent of the listener's position, the above method requires reverse calculation of the relative position information of an absolute immovable object at each reference viewpoint, which is undesirable in terms of the amount of computational processing and accuracy.

[0019] Therefore, this technology can handle four types of objects: absolute coordinate-based interpolation objects, polar coordinate-based fixed objects, polar coordinate-based interpolation objects, and absolute coordinate-based fixed objects. This makes it possible to play back content based on the intentions of the content creator.

[0020] For example, in free viewpoint audio using the Audio Artistic Intent of the present technology, multiple reference viewpoints created by the content creator are assumed, and information regarding the object placement at each of these reference viewpoints is created in advance.

[0021] On the other hand, the listener can freely move to positions other than the reference viewpoint.

[0022] When the listener is at a position different from the reference viewpoint, the position information of the object corresponding to the current position of the listener is calculated by performing an interpolation process based on the position information of the object at multiple reference viewpoints surrounding the listener's position.

[0023] Therefore, for example, as shown by arrow Q11 in Figure 1, when a listener moves from position P11 in the free viewpoint space to position P11', the position of the same object also moves from position P12 to position P12' as the listener moves.

[0024] This makes it possible to reproduce spatial audio from any viewpoint while reflecting the intentions of the content creator.

[0025] Hereinafter, such an object will be referred to as an absolute coordinate-based interpolation object.

[0026] The position of the absolute coordinate-based interpolation object in the free viewpoint space is determined for each of a plurality of reference viewpoints. When the listener is located at a position different from the reference viewpoint, the position of the absolute coordinate-based interpolation object is determined by an interpolation process based on the positions of the absolute coordinate-based interpolation object determined for a plurality of reference viewpoints surrounding the listener's position.

[0027] Therefore, the position of the absolute coordinate-based interpolation object changes depending on the position and orientation of the listener in the free viewpoint space.

[0028] In systems that handle such absolute coordinate-based interpolation objects, for example, when considering an audio augmented reality (AR) system or a guidance support system that uses the direction of sound arrival, an object that is always to be positioned fixedly relative to the listener, independent of the listener's position or orientation in the free viewpoint space, is required. Hereinafter, such an object will be referred to as a polar coordinate-based fixed object.

[0029] For example, when the listener moves from position P11 to position P11' in the free viewpoint space as shown by arrow Q12 in Figure 1, the polar coordinate-based fixed object moves accordingly, for example, from position P13 to position P13'. In this case, from the listener's perspective, the polar coordinate-based fixed object will always be in the same position, such as the front left, before and after the movement.

[0030] The representation using the above-mentioned absolute coordinate-based interpolated object is premised on mapping the object to absolute coordinates, taking into account the orientation of the listener.

[0031] Therefore, it is not possible to handle objects that always remain in the same position relative to the listener. In cases where an object needs to be placed in a fixed position relative to the listener, this technology makes it possible to handle objects for such purposes by combining absolute coordinate-based interpolated objects and polar coordinate-based fixed objects.

[0032] A content playback system capable of handling absolute coordinate-based interpolated objects and polar coordinate-based fixed objects in this way may be configured as shown in FIG. 2, for example.

[0033] The content playback system shown in FIG.

[0034] The server 11 includes a configuration information sending unit 21 and an encoded data sending unit 22 .

[0035] The configuration information sending unit 21 sends (transmits) pre-prepared system configuration information to the client 12, and receives viewpoint selection information, etc. sent from the client 12 and supplies it to the encoded data sending unit 22.

[0036] In a content reproduction system, a plurality of listening positions in a predetermined common absolute coordinate space are designated (set) in advance by a content creator as reference viewpoint positions (hereinafter also referred to as reference viewpoint positions).

[0037] Here, the content creator specifies (sets) in advance as a reference viewpoint the position in the common absolute coordinate space that the listener should use as the listening position when the content is played back, and the direction in which the listener should face at that position, i.e., the viewpoint from which the listener should listen to the sound of the content.

[0038] The server 11 is provided in advance with system configuration information, which is information relating to each reference viewpoint, and object polar coordinate encoded data indicating the position of an absolute coordinate-based interpolation object for each reference viewpoint.

[0039] Here, the object polar coordinate encoded data of the absolute coordinate-based interpolation object for each reference viewpoint is obtained by encoding the object polar coordinate position information that indicates the relative position of the absolute coordinate-based interpolation object as seen from the reference viewpoint.

[0040] This object polar coordinate position information expresses the relative position of the absolute coordinate-based interpolation object as seen from the reference viewpoint, i.e., with the reference viewpoint as the base, in polar coordinates. Note that even for the same absolute coordinate-based interpolation object, the absolute placement position of the absolute coordinate-based interpolation object in the common absolute coordinate space differs for each reference viewpoint.

[0041] The configuration information sending unit 21 sends the system configuration information to the client 12 via a network or the like immediately after the content playback system starts operating, that is, immediately after a connection with the client 12 is established, for example. The system configuration information may be resent to the client 12 at an appropriate timing after the connection is established.

[0042] The encoded data sending unit 22 selects two or more reference viewpoints from among the multiple reference viewpoints based on the viewpoint selection information supplied from the configuration information sending unit 21, and sends the object polar coordinate encoded data of the absolute coordinate-based interpolation object at each selected reference viewpoint to the client 12 via a network or the like.

[0043] Here, the viewpoint selection information is information indicating a reference viewpoint selected on the client 12 side, for example.

[0044] Therefore, the encoded data sending unit 22 acquires the object polar coordinate encoded data of the absolute coordinate-based interpolation object for the reference viewpoint requested by the client 12 and sends it to the client 12 .

[0045] In the following, the explanation will be continued assuming that three reference viewpoints are selected (specified) by viewpoint selection information.

[0046] Furthermore, the server 11 prepares object polar coordinate encoded data of a polar coordinate-based fixed object separately from the object polar coordinate encoded data of an absolute coordinate-based interpolated object.

[0047] The object polar coordinate encoded data of a polar coordinate-based fixed object is obtained by encoding object polar coordinate position information that indicates the relative position of the polar coordinate-based fixed object as seen by the listener, i.e., with the listener as the reference point. This object polar coordinate position information is position information expressed in polar coordinates.

[0048] The object polar coordinate position information of a polar coordinate-based fixed object differs from the object polar coordinate position information of an absolute coordinate-based interpolated object in that it uses the listener's position as the origin (reference) rather than the reference viewpoint as the base (origin), or more specifically, the listener's position and orientation as the base.

[0049] Even if the position or orientation of the listener changes, the object polar coordinate position information of the polar coordinate-based fixed object does not change, so one piece of object polar coordinate encoded data is prepared for one polar coordinate-based fixed object.

[0050] The encoded data sending unit 22 acquires the object polar coordinate encoded data of the polar coordinate-based fixed object and sends it to the client 12 .

[0051] The client 12 has a listener position information acquisition unit 41, a viewpoint selection unit 42, a configuration information acquisition unit 43, an encoded data acquisition unit 44, a decoding unit 45, a coordinate conversion unit 46, a coordinate axis conversion processing unit 47, an object position calculation unit 48, and a polar coordinate conversion unit 49.

[0052] The listener position information acquisition unit 41 acquires listener position information indicating the absolute position (listening position) of the listener in the common absolute coordinate space in response to a designation operation by the user (listener), etc., and supplies the information to the viewpoint selection unit 42, the object position calculation unit 48, and the polar coordinate conversion unit 49.

[0053] For example, in the listener position information, the position of the listener in the common absolute coordinate space is expressed by absolute coordinates. Note that hereinafter, the coordinate system of absolute coordinates indicated by the listener position information will also be referred to as the common absolute coordinate system.

[0054] The viewpoint selection unit 42 selects three reference viewpoints surrounding the listening position based on the system configuration information supplied from the configuration information acquisition unit 43 and the listener position information supplied from the listener position information acquisition unit 41, and supplies viewpoint selection information indicating the selection result to the configuration information acquisition unit 43.

[0055] The configuration information acquisition unit 43 receives system configuration information transmitted from the server 11 and supplies it to the viewpoint selection unit 42 and the coordinate axis transformation processing unit 47, and also transmits the viewpoint selection information supplied from the viewpoint selection unit 42 to the server 11 via a network or the like.

[0056] Here, we will explain an example in which the viewpoint selection unit 42 that selects a reference viewpoint based on listener position information and system configuration information is provided in the client 12, but the viewpoint selection unit 42 may also be provided on the server 11 side.

[0057] The coded data acquisition unit 44 receives the object polar coordinate coded data transmitted from the server 11 and supplies it to the decoding unit 45. That is, the coded data acquisition unit 44 acquires the object polar coordinate coded data from the server 11.

[0058] The decoding unit 45 decodes the object polar coordinate encoded data supplied from the encoded data acquisition unit 44 .

[0059] The decoding unit 45 supplies the object polar coordinate position information of the absolute coordinate-based interpolation object obtained by decoding to the coordinate conversion unit 46 .

[0060] Furthermore, the decoding unit 45 outputs the object polar coordinate position information of the polar coordinate-based fixed object obtained by decoding to a rendering processing unit (not shown) as polar coordinate position information.

[0061] The coordinate conversion unit 46 performs coordinate conversion on the object polar coordinate position information supplied from the decoding unit 45 , and supplies the resulting object absolute coordinate position information to a coordinate axis conversion processing unit 47 .

[0062] The coordinate conversion unit 46 performs coordinate conversion to convert polar coordinates into absolute coordinates. As a result, object polar coordinate position information, which is polar coordinates indicating the position of the absolute coordinate-based interpolation object as seen from the reference viewpoint, is converted into object absolute coordinate position information, which is absolute coordinates indicating the position of the absolute coordinate-based interpolation object in an absolute coordinate system with the position of the reference viewpoint as the origin.

[0063] The coordinate axis conversion processing unit 47 performs coordinate axis conversion processing on the object absolute coordinate position information supplied from the coordinate conversion unit 46 based on the system configuration information supplied from the configuration information acquisition unit 43 .

[0064] Here, the coordinate axis transformation process is a process that combines coordinate transformation (coordinate axis transformation) and offset shift, and the coordinate axis transformation process obtains object absolute coordinate position information that indicates the absolute coordinates of the absolute coordinate-based interpolation object projected onto the common absolute coordinate space. In other words, the object absolute coordinate position information obtained by the coordinate axis transformation process is the absolute coordinates (position information expressed in absolute coordinates) of the common absolute coordinate system that indicates the absolute position of the absolute coordinate-based interpolation object on the common absolute coordinate space.

[0065] The object position calculation unit 48 performs interpolation processing based on the listener position information supplied from the listener position information acquisition unit 41 and the object absolute coordinate position information supplied from the coordinate axis conversion processing unit 47, and supplies the final object absolute coordinate position information obtained as a result to the polar coordinate conversion unit 49. The final object absolute coordinate position information here refers to information that indicates the position of the absolute coordinate-based interpolated object in the common absolute coordinate system when the listener's viewpoint is at the listening position indicated by the listener position information.

[0066] The object position calculation unit 48 calculates the absolute position of the absolute coordinate-based interpolation object in the common absolute coordinate space corresponding to the listening position, i.e., the absolute coordinates of the common absolute coordinate system, from the listening position indicated by the listener position information and the positions of the three reference viewpoints indicated by the viewpoint selection information, and sets this as final object absolute coordinate position information. At this time, the object position calculation unit 48 acquires system configuration information from the configuration information acquisition unit 43 and viewpoint selection information from the viewpoint selection unit 42, as necessary.

[0067] The polar coordinate conversion unit 49 performs polar coordinate conversion on the object absolute coordinate position information supplied from the object position calculation unit 48 based on the listener position information supplied from the listener position information acquisition unit 41, and outputs the resulting polar coordinate position information to a downstream rendering processing unit (not shown).

[0068] The polar coordinate conversion unit 49 performs polar coordinate conversion to convert the object absolute coordinate position information, which is the absolute coordinate of the common absolute coordinate system, into polar coordinate position information, which is the polar coordinate indicating the relative position of the object (absolute coordinate-based interpolated object) as seen from the listening position.

[0069] For example, rendering processing such as VBAP (Vector Based Amplitude Panning) requires polar coordinate position information indicating the relative position of an object with respect to the listening position as the object position information.

[0070] Therefore, for absolute coordinate-based interpolated objects, object absolute coordinate position information for the listener's viewpoint is obtained by interpolation processing, and the object absolute coordinate position information is converted into polar coordinate position information and supplied to the rendering processing unit.

[0071] In contrast, the object polar coordinate position information of the polar coordinate-based fixed object is already in polar coordinates as seen from the listener's viewpoint, so no interpolation processing or conversion to polar coordinates is performed, and it is supplied to the rendering processing unit as is.

[0072] Such a polar coordinate-based fixed object makes it possible to realize object placement that is independent of the position and orientation of the listener.

[0073] As described above, by providing two different types of objects, an absolute coordinate-based interpolation object and a polar coordinate-based fixed object, it is possible to realize content playback based on the intentions of the content creator.

[0074] Incidentally, polar coordinate-based fixed objects are fixed objects that are independent of the position in the listener's free viewpoint space. However, depending on the intention of the content creator, there may be cases where a fixed object that is independent of the listener's orientation is intended to be placed in a different position for each reference viewpoint.

[0075] In such a case, for example, as shown in Figure 3, the object arrangement centered on the listener differs for each viewpoint, but the rendering process at that viewpoint can be considered as a fixed object relative to the listener. Note that in Figure 3, parts corresponding to those in Figure 1 are assigned the same reference numerals, and their explanation will be omitted where appropriate.

[0076] For example, in the example of FIG. 3, when the listener is at position P11 in the free viewpoint space, a fixed object that does not depend on the listener's orientation is at position P21.

[0077] As long as the listener is in the same position, this object will always be in the same position (direction) as seen by the listener, for example, to the left and front of the listener, regardless of the direction the listener is facing.

[0078] In this way, when the listener moves from position P11 to position P11', the object that was at position P21 moves to position P21'. Furthermore, as long as the listener is at position P11', the object will always be in the same position as seen by the listener, regardless of the listener's orientation.

[0079] In this case, the relative position of the object seen by the listener will be different when the listener is at position P11 and when the listener is at position P11'.

[0080] Hereinafter, such an object will be referred to as a polar coordinate-based interpolation object.

[0081] In the example of FIG. 3, by preparing an absolute coordinate-based interpolation object and a polar coordinate-based interpolation object as shown in FIG. 1, content playback based on the intentions of the content creator is realized.

[0082] When an absolute coordinate-based interpolation object and a polar coordinate-based interpolation object are prepared, the content playback system may be configured as shown in Fig. 4. In Fig. 4, parts corresponding to those in Fig. 2 are given the same reference numerals, and their explanation will be omitted as appropriate.

[0083] In the content reproduction system shown in FIG. 4, the server 11 has a configuration information sending unit 21 and an encoded data sending unit 22, similar to the example shown in FIG.

[0084] In addition to the object polar coordinate encoded data of the absolute coordinate-based interpolation object described above, the server 11 also prepares object polar coordinate encoded data of the polar coordinate-based interpolation object for each reference viewpoint.

[0085] The object polar coordinate encoded data of the polar coordinate-based interpolation object for each reference viewpoint is obtained by encoding the object polar coordinate position information indicating the relative position of the polar coordinate-based interpolation object as seen from the reference viewpoint.

[0086] The object polar coordinate position information of the polar coordinate-based interpolation object is expressed in the same coordinates as the object polar coordinate position information of the absolute coordinate-based interpolation object.

[0087] That is, both the object polar coordinate position information of the polar coordinate-based interpolation object and the object polar coordinate position information of the absolute coordinate-based interpolation object are polar coordinates that indicate relative positions as seen from the reference viewpoint, with the reference viewpoint as the origin.

[0088] In this way, the object polar coordinate position information of the polar coordinate-based interpolation object is expressed in the same coordinates as the object polar coordinate position information of the absolute coordinate-based interpolation object.

[0089] However, in the client 12, a polar coordinate-based interpolation object is realized by applying processing different from that for an absolute coordinate-based interpolation object to the object polar coordinate position information of the polar coordinate-based interpolation object.

[0090] The encoded data sending unit 22 sends to the client 12 the object polar coordinate encoded data of the absolute coordinate-based interpolation object and the object polar coordinate encoded data of the polar coordinate-based interpolation object for the three reference viewpoints indicated by the viewpoint selection information.

[0091] 4, the client 12 further includes an object position calculation unit 71 in addition to the components shown in FIG.

[0092] In the decoding unit 45 of the client 12, the object polar coordinate position information of the absolute coordinate-based interpolation object obtained by decoding is supplied to the coordinate conversion unit 46, as in the case of FIG.

[0093] The decoding unit 45 also supplies the object polar coordinate position information of the polar coordinate-based interpolation object at each reference viewpoint obtained by decoding to the object position calculation unit 71 .

[0094] The object position calculation unit 71 performs interpolation processing based on the listener position information supplied from the listener position information acquisition unit 41 and the object polar coordinate position information of the polar coordinate-based interpolated object supplied from the decoding unit 45. At this time, the object position calculation unit 71 acquires system configuration information from the configuration information acquisition unit 43 and viewpoint selection information from the viewpoint selection unit 42 as necessary.

[0095] As a result, object polar coordinate position information of the polar coordinate-based interpolation object for the position of the listener (listening position) is obtained as polar coordinate position information.

[0096] This polar coordinate position information is polar coordinates that indicate the relative position of the polar coordinate-based interpolation object as seen from the listening position when the listener's viewpoint is at the listening position indicated by the listener position information.

[0097] The object position calculation unit 71 outputs the polar coordinate position information of the polar coordinate-based interpolated object obtained by the interpolation process to a rendering processing unit (not shown) at a subsequent stage.

[0098] In the object position calculation unit 71, the object polar coordinate position information of each reference viewpoint is not converted into absolute coordinates, but is used directly, i.e., in the polar coordinate state, for interpolation processing, and polar coordinate position information indicating the relative position as seen from the listening position is generated.

[0099] In this way, the polar coordinate-based interpolation object is processed differently from the absolute coordinate-based interpolation object. As a result, although the position seen by the listener varies depending on the listening position, it is possible to realize a polar coordinate-based interpolation object that always exists fixedly in the same position (direction) as seen by the listener at the same listening position, regardless of the listener's orientation (viewpoint).

[0100] Furthermore, in applications such as recreating a live concert venue, objects representing background noise present in the venue are placed at absolute positions within the free viewpoint space, independent of the listener's position, as shown in Figure 5. Note that in Figure 5, parts corresponding to those in Figure 1 are given the same reference numerals, and their explanation will be omitted.

[0101] In this example, in addition to the absolute coordinate-based interpolated object and the polar coordinate-based fixed object, there is also an object in the free viewpoint space that is always at the same position in the free viewpoint space regardless of the listener's viewpoint. Hereinafter, such an object that is always placed at the same position (fixed position) in the free viewpoint space (common absolute coordinate space) will be referred to as an absolute coordinate-based fixed object.

[0102] 5, when the listener moves from position P11 to position P11', the absolute coordinate-based interpolated object moves from position P12 to position P12', and the polar coordinate-based fixed object moves from position P13 to position P13'. However, even if the listener moves from position P11 to position P11', the absolute coordinate-based fixed object remains at position P31.

[0103] It is possible to express the behavior of such an absolute coordinate-based fixed object using an absolute coordinate-based interpolated object. However, in such cases, the object's polar coordinate position information is calculated by back-calculating the relative position information of the absolute immovable object from each reference viewpoint, which has many disadvantages, such as unnecessary calculations and a loss in accuracy.

[0104] Therefore, in this technology, an absolute coordinate-based fixed object is separately prepared as an object with a fixed absolute coordinate position, and by combining the absolute coordinate-based fixed object with other absolute coordinate-based interpolation objects or polar coordinate-based fixed objects, the amount of calculation is reduced and it is also advantageous in terms of accuracy.

[0105] Specifically, for example, when an absolute coordinate-based interpolation object and an absolute coordinate-based fixed object are prepared, the content playback system is configured as shown in Fig. 6. Note that in Fig. 6, parts corresponding to those in Fig. 2 are given the same reference numerals, and their explanation will be omitted as appropriate.

[0106] In the example of Fig. 6, the server 11 and the client 12 have the same configuration as in Fig. 2. However, in the server 11, in addition to the object polar coordinate encoded data of the absolute coordinate-based interpolated object described above, object absolute coordinate encoded data of the absolute coordinate-based fixed object is also prepared.

[0107] The object absolute coordinate encoded data of the absolute coordinate-based fixed object is obtained by encoding the object absolute coordinate position information, which is the absolute coordinates that indicate the absolute position of the absolute coordinate-based fixed object in the common absolute coordinate space (common absolute coordinate system).

[0108] The object absolute coordinate position information of this absolute coordinate-based fixed object corresponds to the object absolute coordinate position information of the absolute coordinate-based interpolated object obtained by the object position calculation unit 48 of the client 12.

[0109] The encoded data sending unit 22 sends to the client 12 the object polar coordinate encoded data of the absolute coordinate-based interpolated object and the object absolute coordinate encoded data of the absolute coordinate-based fixed object for the three reference viewpoints indicated by the viewpoint selection information.

[0110] Therefore, the decoding unit 45 of the client 12 decodes the object polar coordinate encoded data and the object absolute coordinate encoded data.

[0111] The decoding unit 45 then supplies the object polar coordinate position information of the absolute coordinate-based interpolated object obtained by decoding to the coordinate conversion unit 46, and supplies the object absolute coordinate position information of the absolute coordinate-based fixed object obtained by decoding to the polar coordinate conversion unit 49.

[0112] An absolute coordinate-based fixed object is always placed in a fixed position in the common absolute coordinate space regardless of the viewpoint of the listener, and therefore does not require the interpolation process that is required for an absolute coordinate-based interpolated object.

[0113] The object absolute coordinate position information of the absolute coordinate-based fixed object can be treated in the same way as the object absolute coordinate position information of the absolute coordinate-based interpolated object obtained by the object position calculation unit 48, and is therefore supplied to the polar coordinate conversion unit 49 as is after decoding.

[0114] Based on the listener position information from the listener position information acquisition unit 41, the polar coordinate conversion unit 49 performs polar coordinate conversion not only on the object absolute coordinate position information supplied from the object position calculation unit 48 but also on the object absolute coordinate position information supplied from the decoding unit 45.

[0115] The polar coordinate conversion unit 49 outputs the polar coordinate position information of the absolute coordinate-based interpolated object and the polar coordinate position information of the absolute coordinate-based fixed object obtained as a result of the polar coordinate conversion to a rendering processing unit (not shown) at a subsequent stage.

[0116] Although the above describes specific combinations of absolute coordinate-based interpolation objects, polar coordinate-based fixed objects, polar coordinate-based interpolation objects, and absolute coordinate-based fixed objects as examples, any combination of these objects is possible.

[0117] <Example of interpolation processing> Here, specific examples of the interpolation processing performed by the object position calculation unit 48 and the interpolation processing performed by the object position calculation unit 71 will be described.

[0118] First, an example of interpolation processing based on object absolute coordinate position information at each reference viewpoint, which is performed in the object position calculation unit 48, will be described.

[0119] For example, as shown on the left side of FIG. 7, it is assumed that absolute coordinate position information of an object at an arbitrary listening position F is obtained by interpolation processing.

[0120] In this example, there are three reference viewpoints A, B, and C surrounding listening position F, and it is assumed that information from these reference viewpoints A to C is used to perform interpolation processing.

[0121] In the following, the X and Y coordinates of the listening position F in the common absolute coordinate system, i.e., the XYZ coordinate system, are (x f ,y f )

[0122] Similarly, the X and Y coordinates of the positions of reference viewpoint A, reference viewpoint B, and reference viewpoint C are (x a ,y a ), (x b ,y b ), and (x c ,y c )

[0123] In this case, as shown on the right side of Figure 7, object position F' at listening position F is determined based on the coordinates of object position A', object position B', and object position C' corresponding to reference viewpoint A, reference viewpoint B, and reference viewpoint C, respectively.

[0124] Here, for example, object position A' indicates the position of the object when the viewpoint is at reference viewpoint A, that is, the position of the absolute coordinate-based interpolated object in the common absolute coordinate system indicated by the object absolute coordinate position information of reference viewpoint A.

[0125] Furthermore, object position F' indicates the position of the absolute coordinate-based interpolated object in the common absolute coordinate system when the listener is at listening position F, that is, the position indicated by the object absolute coordinate position information output by object position calculation unit 48.

[0126] In the following, the X and Y coordinates of object position A', object position B', and object position C' are (x a ',y a '), (x b ',y b '), and (x c ',y c '), and the X and Y coordinates of object position F' are (x f ',y f ').

[0127] In the following, a triangular area surrounded by any three reference viewpoints, such as reference viewpoint A to reference viewpoint C, i.e., a triangular area formed by three reference viewpoints, will also be referred to as a triangular mesh.

[0128] Since a plurality of reference viewpoints exist in the common absolute coordinate space, a plurality of triangular meshes can be formed in the common absolute coordinate space with the reference viewpoints as vertices.

[0129] Similarly, in the following, a triangular area surrounded (formed) by object positions indicated by the object absolute coordinate position information of any three reference viewpoints, such as object position A' to object position C', will also be referred to as a triangular mesh.

[0130] Coordinates indicating any position in the common absolute coordinate system (XYZ coordinate system) can be obtained from the coordinates of that position in the xyz coordinate system and information about the reference viewpoint included in the system configuration information, more specifically, the position of the reference viewpoint and the orientation of the listener at the reference viewpoint.

[0131] The xyz coordinate system here is an absolute coordinate system with the position of the reference viewpoint as its origin (reference point). For simplicity of explanation, it is assumed that the Z coordinate value in the XYZ coordinate system is the same as the z coordinate value in the xyz coordinate system.

[0132] Ceva's theorem proves that any listening position within a triangular mesh formed by three reference viewpoints can be uniquely determined as the intersection of line segments from each of the three vertices of the triangular mesh to each of the three internal division points of the three sides that are not adjacent to those vertices, provided that the internal division ratios of each side of the triangular mesh are appropriately determined.

[0133] From the proof formula, this holds true for all triangular meshes, regardless of the shape of the triangular mesh, once the internal division ratio of the three sides of the triangular mesh is determined.

[0134] Therefore, by calculating the internal division ratio of the triangular mesh including the listening position for the viewpoint side, i.e., the reference viewpoint, and applying that internal division ratio to the triangular mesh for the object side, i.e., the object position, it is possible to calculate the appropriate object position for any listening position.

[0135] Here, with reference to FIGS. 8 and 9, an example will be described in which, for the listening position F shown in FIG. 7, absolute object coordinate position information indicating the object position F' is obtained by interpolation processing.

[0136] For example, as shown in FIG. 8, first, the X and Y coordinates of the internal division point in the triangular mesh consisting of reference viewpoints A to C, including the listening position F, are found.

[0137] Now, let the intersection of the line passing through the listening position F and the reference viewpoint C with the line segment AB from the reference viewpoint A to the reference viewpoint B be point D. Let the coordinates of point D on the XY plane be (x d ,y d ) In other words, point D is an internal division point on line segment AB (side AB).

[0138] In this case, the relationship shown in the following equation (1) holds for the X and Y coordinates indicating the position of any point on the line segment CF from the reference viewpoint C to the listening position F, and for the X and Y coordinates indicating the position of any point on the line segment AB.

[0139]

number

[0140] Furthermore, point D is the intersection of the line passing through reference viewpoint C and listening position F with line segment AB, so from equation (1), the coordinates of point D on the XY plane (x d ,y d ) can be found, and its coordinates (x d ,y d ) is expressed as the following equation (2).

[0141]

number

[0142] Therefore, as shown in the following equation (3), the coordinates of point D (x d ,y d ), coordinates of reference viewpoint A (x a ,y a ), and the coordinates of reference viewpoint B (x b ,y b ), we can obtain the internal division ratio (m,n) of the line segment AB by point D, that is, the division ratio.

[0143]

number

[0144] Similarly, the intersection of the line passing through the listening position F and the reference viewpoint B with the line segment AC from the reference viewpoint A to the reference viewpoint C is defined as point E, and the coordinates indicating the position of point E on the XY plane are (x e ,y e ) In other words, point E is an internal division point on line segment AC (side AC).

[0145] In this case, the relationship shown in the following equation (4) holds for the X and Y coordinates indicating the position of any point on the line segment BF from the reference viewpoint B to the listening position F, and for the X and Y coordinates indicating the position of any point on the line segment AC.

[0146]

number

[0147] Furthermore, point E is the intersection of the line passing through reference viewpoint B and listening position F with line segment AC, so from equation (4), the coordinates of point E on the XY plane (x e ,y e ) can be found, and its coordinates (x e ,y e ) is expressed as the following equation (5).

[0148]

number

[0149] Therefore, as shown in the following equation (6), the coordinates of point E (x e ,y e ), coordinates of reference viewpoint A (x a ,y a ), and the coordinates of the reference viewpoint C (x c ,y c ), we can obtain the internal division ratio (k, l) of the line segment AC by point E, that is, the division ratio.

[0150]

number

[0151] Next, the ratio of the two sides thus obtained, that is, the internal division ratio (m, n) and the internal division ratio (k, l), are applied to the triangular mesh on the object side as shown in Figure 9, and the coordinates of the object position F' on the XY plane (x f ',y f') is required.

[0152] Specifically, in this example, the point on the line segment A'B' connecting the object position A' and the object position B', which corresponds to the point D, is set as point D'.

[0153] Similarly, the point on the line segment A'C' connecting the object position A' and the object position C' that corresponds to the point E is set as point E'.

[0154] Furthermore, the intersection of a line passing through object position C' and point D' and a line passing through object position B' and point E' is object position F' corresponding to listening position F.

[0155] Here, let us assume that the internal division ratio of line segment A'B' at point D' is the same as that at point D (m, n). In this case, the coordinates of point D' on the XY plane (x d ',y d As shown in the following equation (7), the internal division ratio (m, n) and the coordinates of the object position A' (x a ',y a '), and the coordinates of object position B' (x b ',y b ') can be obtained based on

[0156]

number

[0157] Also, assume that the internal division ratio of the line segment A'C' at point E' is the same as that at point E (k, l). In this case, the coordinates of point E' on the XY plane (x e ',y e As shown in the following equation (8), the internal division ratio (k, l) and the coordinates of the object position A' (x a ',y a '), and the coordinates of object position C' (x c ',y c ') can be obtained based on

[0158]

number

[0159] Therefore, the relationship shown in the following equation (9) holds for the X and Y coordinates indicating the position of any point on the line segment B'E' from object position B' to point E', and the X and Y coordinates indicating the position of any point on the line segment C'D' from object position C' to point D'.

[0160]

number

[0161] The target object position F' is the intersection of the line segments B'E' and C'D', so from the relationship in equation (9), the coordinates of the object position F' (x f ',y f ') can be obtained.

[0162]

number

[0163] By the above process, the coordinates of the object position F' on the XY plane (x f ',y f ') is obtained.

[0164] Next, the coordinates of the object position F' on the XY plane (x f ',y f ') and the coordinates of object position A' in the XYZ coordinate system (x a ',y a ',z a '), coordinates of object position B' (x b ',y b ',z b '), and the coordinates of object position C' (x c ',y c ',z c') and the coordinates (x,y,z) of the object position F' in the XYZ coordinate system f ',y f ',z f That is, the Z coordinate z of the object position F' in the XYZ coordinate system is calculated. f ' is required.

[0165] For example, a triangle in three-dimensional space with vertices at object positions A', B', and C' in the XYZ coordinate system (common absolute coordinate space) is obtained, that is, a three-dimensional plane A'B'C' including object positions A', B', and C' is obtained. Then, the X and Y coordinates on the three-dimensional plane A'B'C' are calculated as follows: f ',y f ') is found, and the Z coordinate of that point is z f 'It is said that.

[0166] Specifically, the vector starting from object position A' in the XYZ coordinate system and ending at object position B' is called vector A'B'=(x ab ',y ab ',z ab ').

[0167] Similarly, the vector starting from object position A' in the XYZ coordinate system and ending at object position C' is vector A'C'=(x ac ',y ac ',z ac ').

[0168] These vectors A'B' and A'C' are the coordinates of the object position A' (x a ',y a ',z a '), coordinates of object position B' (x b ',y b ',z b '), and the coordinates of object position C' (x c ',y c ',z cThat is, vectors A'B' and A'C' can be obtained by the following equation (11).

[0169]

number

[0170] Furthermore, the normal vector (s, t, u) of the three-dimensional plane A'B'C' is the cross product of the vector A'B' and the vector A'C', and can be calculated by the following equation (12).

[0171]

number

[0172] Therefore, the normal vector (s,t,u) and the coordinates of the object position A' (x a ',y a ',z a '), the plane equation of the three-dimensional plane A'B'C' is given by the following equation (13).

[0173]

number

[0174] Here, the X coordinate x of the object position F' on the 3D plane A'B'C' is f ' and the Y coordinate y f ' has already been found, so we can use the X coordinate x in the X and Y parts of the plane equation (13). f ' and the Y coordinate y f By substituting ', the Z coordinate z is calculated as shown in the following equation (14). f ' can be sought.

[0175]

number

[0176] From the above calculation, the coordinates of the target object position F' (xf ',y f ',z f The object position calculation unit 48 calculates the coordinates (x f ',y f ',z f ') is output as absolute object coordinate position information.

[0177] Next, the interpolation process performed by the object position calculation unit 71 will be described.

[0178] For example, as shown in FIG. 10, consider the case where object polar coordinate position information of a polar coordinate-based interpolated object at listening position F surrounded by three reference viewpoints A, B, and C is obtained by interpolation processing.

[0179] In FIG. 10, parts corresponding to those in FIG. 8 are given the same symbols (reference numerals), and the description thereof will be omitted as appropriate.

[0180] That is, in the example shown in FIG. 10, the coordinates of point D (x d ,y d ), the coordinates of point E (x e ,y e ), and the coordinates of listening position F (x f ,y f ), and the internal division ratios (m, n) and (k, l) are assumed to be obtained.

[0181] Furthermore, the object polar coordinate position information of the polar coordinate-based interpolation object at reference viewpoint A, reference viewpoint B, and reference viewpoint C, respectively, is assumed to be (Az(a),El(a),Rad(a)), (Az(b),El(b),Rad(b)), and (Az(c),El(c),Rad(c)).

[0182] Here, for example, Az(a), El(a), and Rad(a) are the horizontal angle, vertical angle, and radius that make up the polar coordinates.

[0183] In this case, the object polar coordinate position information (Az(a),El(a),Rad(a)) and (Az(b),El(b),Rad(b)) of the reference viewpoint A and the reference viewpoint B, respectively, and the internal division ratio (m, n) are used to calculate the object polar coordinate position information (Az(d),El(d),Rad(d)) of the polar coordinate-based interpolation object at point D using the following equation (15):

[0184]

number

[0185] Furthermore, the object polar coordinate position information of point D (Az(d),El(d),Rad(d)) and the object polar coordinate position information of reference viewpoint C (Az(c),El(c),Rad(c)) and the coordinates of reference viewpoint C (x c ,y c ), the coordinates of point D (x d ,y d ), and the coordinates of listening position F (x f ,y f ), the object polar coordinate position information (Az(f), El(f), Rad(f)) of the polar coordinate-based interpolation object at the listening position F is calculated using the following equation (16).

[0186]

number

[0187] In this way, the object position calculation unit 71 performs interpolation processing based on the object polar coordinate position information at the three reference viewpoints using equation (16) while keeping the polar coordinates, and calculates the object polar coordinate position information of the polar coordinate-based interpolated object at the listening position.

[0188] The interpolation processing performed by the object position calculation unit 71 is not limited to the example described with reference to FIG. 10, and may be realized by any other processing, such as interpolation processing using vector calculations or calculation processing using a neural network.

[0189] <Example of system configuration information> FIG. 11 shows an example of a bitstream format of system configuration information when it is possible to handle the absolute coordinate-based interpolation object, polar coordinate-based fixed object, polar coordinate-based interpolation object, and absolute coordinate-based fixed object described above.

[0190] 11, "NumOfObjs" indicates the number of objects that make up the content. The number of objects here refers to the total number of absolute coordinate-based interpolation objects, polar coordinate-based fixed objects, polar coordinate-based interpolation objects, and absolute coordinate-based fixed objects.

[0191] Also, "NumfOfRefViewPoint" indicates the number of reference viewpoints.

[0192] The system configuration information includes reference viewpoint information consisting of reference viewpoint position information and listener direction information, the number of which is "NumfOfRefViewPoint", the number of reference viewpoints.

[0193] The reference viewpoint position information is the absolute coordinates in a common absolute coordinate system that indicate the position of the reference viewpoint. In this example, it includes the X coordinate "RefViewX[i]", the Y coordinate "RefViewY[i]", and the Z coordinate "RefViewZ[i]" that indicate the position of the reference viewpoint in the common absolute coordinate system.

[0194] The listener direction information is the desired face direction of the listener at the reference viewpoint, that is, the horizontal rotation angle (horizontal angle) of the listener's face, which indicates the expected face direction of the listener at the reference viewpoint.

[0195] In this example, the listener orientation information includes the horizontal angle of the listener's face, "RefYaw[i]." Note that the listener orientation information may include not only the horizontal angle of the listener's face (yaw angle) but also the vertical angle (pitch angle) indicating the vertical orientation of the listener's face.

[0196] Furthermore, the system configuration information includes "ObjectOverLapMode[i]," which indicates the playback mode when the listener and the object overlap, i.e., when the listener (listening position) and the object are in the same position, for each object (the number of objects, "NumOfObjs"). Also, "ProhibitRadius" indicates the normalized value of the distance from the object to the listener when the space is normalized to 1.0.

[0197] "InterpolationMode" indicates the interpolation mode permitted to the client 12. "NonInterpolatePolarObjFlag" is a flag indicating whether or not a polar coordinate-based fixed object exists. That is, the value "1" of the flag "NonInterpolatePolarObjFlag" indicates that a polar coordinate-based fixed object exists, and the value "0" indicates that a polar coordinate-based fixed object does not exist.

[0198] Furthermore, when the value of the flag "NonInterpolatePolarObjFlag" is "1", the system configuration information stores "NumOfObjs_NIPO" which indicates the number of polar coordinate-based fixed objects.

[0199] "NonInterpolateCartesianObjFlag" is a flag that indicates whether an absolute coordinate-based fixed object exists. That is, the value "1" of the flag "NonInterpolateCartesianObjFlag" indicates that an absolute coordinate-based fixed object exists, and the value "0" indicates that an absolute coordinate-based fixed object does not exist.

[0200] Furthermore, when the value of the flag "NonInterpolateCartesianObjFlag" is "1", the system configuration information stores "NumOfObjs_NICO" which indicates the number of absolute coordinate-based fixed objects.

[0201] "InterpolatePolarObjFlag" is a flag that indicates whether a polar coordinate-based interpolation object exists. That is, a value of "1" for the flag "InterpolatePolarObjFlag" indicates that a polar coordinate-based interpolation object exists, and a value of "0" indicates that a polar coordinate-based interpolation object does not exist.

[0202] Furthermore, when the value of the flag "InterpolatePolarObjFlag" is "1", the system configuration information stores "NumOfObjs_IPO" which indicates the number of polar coordinate-based interpolation objects.

[0203] "NumOfAncBytes" indicates the size of the extended information area, and "AncByteData[i]" indicates the extended area byte data.

[0204] For example, the server 11 transmits to the client 12 the system configuration information having the configuration shown in FIG.

[0205] <Bitstream format example> FIG. 12 shows an example of a bitstream format for transmitting information indicating the position of each object collectively, i.e., object polar coordinate encoded data and object absolute coordinate encoded data, under the condition that the position of each object does not change over time, i.e., the object does not move.

[0206] In this example, "fva_structure_info_polar()" indicates the system configuration information. Note that the system configuration information does not necessarily need to be included if it is sent separately.

[0207] The bitstream contains the metadata "object_metadata()" of the absolute coordinate-based interpolation object for the reference points, for the number of reference viewpoints "NumfOfRefViewPoint".

[0208] This metadata "object_metadata()" includes the object polar coordinate position information of the above-mentioned absolute coordinate-based interpolation object, more specifically, the object polar coordinate encoded data and the gain information (gain amount) of the absolute coordinate-based interpolation object.

[0209] In particular, in this example, since each piece of information is transmitted collectively, the metadata "object_metadata()" of the absolute coordinate-based interpolation object is stored for all reference viewpoints.

[0210] Furthermore, if the value of the flag "NonInterpolatePolarObjFlag" included in the system configuration information is "1", the bitstream stores the metadata "object_metadata_nonintpPolar()" of the polar coordinate-based fixed object.

[0211] Similarly, if the value of the flag "NonInterpolateCartesianObjFlag" included in the system configuration information is "1", the bitstream stores the metadata "object_metadata_nonintpCarte()" of the absolute coordinate-based fixed object.

[0212] Furthermore, if the value of the flag "InterpolatePolarObjFlag" included in the system configuration information is "1", the bitstream stores the metadata "object_metadata_intpPolar()" of the polar coordinate-based interpolation object.

[0213] In this example, as in the case of the absolute coordinate-based interpolation object, the metadata "object_metadata_intpPolar()" of the polar coordinate-based interpolation object for the reference points is stored for the number of reference viewpoints "NumfOfRefViewPoint".

[0214] In FIG. 12, an example has been described in which it is assumed that the position of each object does not change over time.

[0215] In contrast, Figure 13 shows an example of a bitstream format when transmitting object polar coordinate encoded data or object absolute coordinate encoded data of each object in frame units of the audio data of each object, corresponding to the temporal change in the position of each object.

[0216] In Fig. 13, "fva_structure_info_polar_present" indicates a configuration information present flag that indicates whether or not system configuration information is included in this bitstream, and in particular, a value of "1" of the configuration information present flag indicates that system configuration information is included (stored). In contrast, a value of "0" of the configuration information present flag indicates that system configuration information is not included.

[0217] If the value of the configuration information present flag "fva_structure_info_polar_present" is "1", the bitstream contains the system configuration information "fva_structure_info_polar()".

[0218] 13, the system configuration information does not necessarily have to be included, and the system configuration information may be transmitted at equal or unequal intervals, etc. In other words, the system configuration information may be transmitted in some frames and not in other frames.

[0219] Also, in this example, since transmission is performed on a frame-by-frame basis, the metadata "object_metadata()" of the absolute coordinate-based interpolation object is stored only for the three reference viewpoints indicated by the viewpoint selection information received (acquired) from the client 12.

[0220] Furthermore, as in the example of Figure 12, if the value of the flag "NonInterpolatePolarObjFlag" included in the system configuration information is "1", the bitstream stores the metadata "object_metadata_nonintpPolar()" of the polar coordinate-based fixed object.

[0221] Furthermore, if the value of the flag "NonInterpolateCartesianObjFlag" included in the system configuration information is "1", the bitstream stores the metadata "object_metadata_nonintpCarte()" of the absolute coordinate-based fixed object.

[0222] On the other hand, if the value of the flag "InterpolatePolarObjFlag" included in the system configuration information is "1", the bitstream stores the metadata of the polar coordinate-based interpolation object "object_metadata_intpPolar()" only for the three reference viewpoints indicated by the viewpoint selection information.

[0223] Next, examples of the metadata of the polar coordinate-based fixed object, the metadata of the polar coordinate-based interpolated object, and the metadata of the absolute coordinate-based fixed object will be described with reference to FIGS.

[0224] FIG. 14 shows an example of the bitstream format of the metadata "object_metadata_nonintpPolar()" of the polar coordinate-based fixed object shown in FIG. 12 or 13.

[0225] In this example, the object polar coordinate position information (object polar coordinate encoded data) and gain amount (gain information) of the polar coordinate base fixed objects are stored for the number of polar coordinate base fixed objects "NumOfObjs_NIPO" included in the system configuration information.

[0226] That is, "PosAzi[i]", "PosEle[i]", and "PosRad[i]" indicate the horizontal angle, vertical angle, and radius that constitute the object polar coordinate position information of the polar coordinate-based fixed object. Also, "Gain[i]" indicates the gain amount for gain adjustment of the audio data of the polar coordinate-based fixed object, more specifically, the encoding gain information obtained by encoding the gain information.

[0227] FIG. 15 shows an example of the bitstream format of the metadata "object_metadata_intpPolar()" of the polar coordinate-based interpolation object shown in FIG. 12 or 13.

[0228] In this example, the object polar coordinate position information (object polar coordinate encoded data) and gain amount (gain information) of the polar coordinate based interpolation objects are stored for the number of polar coordinate based interpolation objects "NumOfObjs_IPO" included in the system configuration information.

[0229] That is, "PosAzi[i]", "PosEle[i]", and "PosRad[i]" indicate the horizontal angle, vertical angle, and radius that constitute the object polar coordinate position information of the polar coordinate-based interpolation object. Also, "Gain[i]" indicates the gain amount for gain adjustment of the audio data of the polar coordinate-based interpolation object, more specifically, the encoding gain information.

[0230] FIG. 16 shows an example of the bitstream format of the metadata "object_metadata_nonintpCarte()" of the absolute coordinate-based fixed object shown in FIG. 12 or 13.

[0231] In this example, the object absolute coordinate position information (object absolute coordinate encoded data) and gain amount (gain information) of the absolute coordinate base fixed objects are stored for the number of absolute coordinate base fixed objects "NumOfObjs_NICO" included in the system configuration information.

[0232] That is, "PosX[i]", "PosY[i]", and "PosZ[i]" indicate the X coordinate, Y coordinate, and Z coordinate of the common absolute coordinate system (XYZ coordinate system) that constitute the object absolute coordinate position information of the absolute coordinate-based fixed object. Also, "Gain[i]" indicates the gain amount for gain adjustment of the audio data of the absolute coordinate-based fixed object, more specifically, encoding gain information.

[0233] <Configuration example of content playback system> Here, a more detailed embodiment of the content playback system to which the present technology described above is applied will be described.

[0234] Fig. 17 is a diagram showing an example of the configuration of a content playback system to which the present technology is applied. Note that in Fig. 17, parts corresponding to those in Fig. 4 are given the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0235] The content reproduction system shown in FIG. 17 includes a server 11 that distributes content, and a client 12 that receives the content distributed from the server 11.

[0236] The server 11 also includes a configuration information recording unit 101 , a configuration information transmitting unit 21 , a recording unit 102 , an encoded data transmitting unit 22 , and a transmission buffer 103 .

[0237] 11 prepared in advance, and supplies the recorded system configuration information to configuration information transmission unit 21. Note that a part of recording unit 102 may be configured as configuration information recording unit 101.

[0238] The recording unit 102 records, for example, encoded audio data obtained by encoding audio data of objects that make up the content, object polar coordinate encoded data, object absolute coordinate encoded data, encoding gain information, etc. for each object for each reference viewpoint.

[0239] The recording unit 102 supplies the recorded encoded audio data, object polar coordinate encoded data, object absolute coordinate encoded data, encoding gain information, etc. to the encoded data sending unit 22 in response to a request or the like.

[0240] The transmission buffer 103 temporarily stores the encoded audio data, the object polar coordinate encoded data, the object absolute coordinate encoded data, the encoding gain information, and the like, which are supplied from the encoded data sending unit 22 .

[0241] The client 12 also includes a listener position information acquisition unit 41 , a viewpoint selection unit 42 , a communication unit 111 , a decoding unit 45 , a position calculation unit 112 , and a rendering processing unit 113 .

[0242] The communication unit 111 corresponds to the configuration information acquisition unit 43 and the encoded data acquisition unit 44 shown in FIG. 2 and the like, and communicates with the server 11 to send and receive various types of data.

[0243] For example, the communication unit 111 transmits viewpoint selection information supplied from the viewpoint selection unit 42 to the server 11, and receives system configuration information and a bitstream transmitted from the server 11. In other words, the communication unit 111 functions as a reference viewpoint information acquisition unit that acquires, from the server 11, the system configuration information, the object polar coordinate encoded data and object absolute coordinate encoded data included in the bitstream, and encoding gain information.

[0244] The position calculation unit 112 generates polar coordinate position information indicating the positions of various types of objects based on the object polar coordinate position information and object absolute coordinate position information supplied from the decoding unit 45 and the system configuration information supplied from the communication unit 111, and supplies the generated information to the rendering processing unit 113.

[0245] Furthermore, the position calculation unit 112 performs gain adjustment on the audio data of each type of object supplied from the decoding unit 45 , and supplies the gain-adjusted audio data to the rendering processing unit 113 .

[0246] The position calculation unit 112 includes a coordinate conversion unit 46 , a coordinate axis conversion processing unit 47 , an object position calculation unit 48 , a polar coordinate conversion unit 49 , and an object position calculation unit 71 .

[0247] The rendering processing unit 113 performs rendering processing such as VBAP based on the polar coordinate position information and audio data supplied from the polar coordinate conversion unit 49, the object position calculation unit 71, and the decoding unit 45, and generates and outputs playback audio data for playing back the sound of the content.

[0248] <Description of the provision process> Next, the operations of the server 11 and the client 12 shown in FIG. 17 will be described.

[0249] For example, after the server 11 has been initialized, if a request to open a network session is received from the client 12, the server 11 performs processing to open a network session with the client 12 in response to the request.

[0250] Then, when the server 11 receives information from the client 12 indicating that a session is to be started and further receives a request from the client 12 to transmit system configuration information, the server 11 starts a providing process, which is a process for providing content.

[0251] The providing process by the server 11 will be described below with reference to the flowchart of FIG.

[0252] In step S 11 , the configuration information sending unit 21 reads the system configuration information of the requested content from the configuration information recording unit 101 , and transmits the read system configuration information to the client 12 .

[0253] For example, the system configuration information is transmitted to the client 12 via a network or the like immediately after a session is opened, i.e., immediately after a connection between the server 11 and the client 12 is established, and before encoded audio data or the like is transmitted.

[0254] When the system configuration information is transmitted, the client 12 transmits viewpoint selection information indicating three reference viewpoints according to the listener's position.

[0255] In step S 12 , the configuration information transmitter 21 receives the viewpoint selection information transmitted from the client 12 , and supplies the viewpoint selection information and system configuration information to the coded data transmitter 22 .

[0256] In step S13, the coded data sending unit 22 loads the data of the absolute coordinate-based interpolation object that constitutes the content into the transmission buffer 103 based on the viewpoint selection information and system configuration information supplied from the configuration information sending unit 21.

[0257] That is, for each of the three reference viewpoints indicated by the viewpoint selection information, the coded data sending unit 22 reads out the object polar coordinate coded data and the coding gain information for each absolute coordinate-based interpolation object from the recording unit 102, and supplies and stores them in the transmission buffer 103. The coded data sending unit 22 also reads out the coded audio data of each absolute coordinate-based interpolation object from the recording unit 102, and supplies and stores them in the transmission buffer 103.

[0258] In step S14, the coded data sending unit 22 determines whether or not there is a polar coordinate-based fixed object as an object constituting the content, based on the system configuration information. Here, if the value of the flag "NonInterpolatePolarObjFlag" in the system configuration information is "1", it is determined that there is a polar coordinate-based fixed object.

[0259] If it is determined in step S14 that there is a polar coordinate-based fixed object, in step S15 the coded data sending unit 22 loads the data of the polar coordinate-based fixed object that constitutes the content into the transmission buffer 103 based on the system configuration information.

[0260] That is, the encoded data sending unit 22 reads out the object polar coordinate encoded data, encoding gain information, and encoded audio data of each polar coordinate-based fixed object that constitutes the content from the recording unit 102, and supplies them to the transmission buffer 103 for storage.

[0261] After the process of step S15 is performed, the process proceeds to step S16.

[0262] On the other hand, if it is determined in step S14 that there is no polar coordinate base fixed object, the process of step S15 is not performed, and the process then proceeds to step S16.

[0263] In step S16, the coded data sending unit 22 determines whether or not there is an absolute coordinate-based fixed object as an object constituting the content, based on the system configuration information. Here, if the value of the flag "NonInterpolateCartesianObjFlag" in the system configuration information is "1", it is determined that there is an absolute coordinate-based fixed object.

[0264] If it is determined in step S16 that there is an absolute coordinate-based fixed object, in step S17 the coded data sending unit 22 loads the data of the absolute coordinate-based fixed object that constitutes the content into the transmission buffer 103 based on the system configuration information.

[0265] That is, the coded data sending unit 22 reads out the object absolute coordinate coded data, coding gain information, and coded audio data of each absolute coordinate-based fixed object that constitutes the content from the recording unit 102, and supplies them to the transmission buffer 103 for storage.

[0266] After the process of step S17 is performed, the process proceeds to step S18.

[0267] On the other hand, if it is determined in step S16 that there is no absolute coordinate base fixed object, the process of step S17 is not performed, and the process then proceeds to step S18.

[0268] In step S18, the coded data sending unit 22 determines whether or not a polar coordinate-based interpolation object exists as an object constituting the content, based on the system configuration information. Here, if the value of the flag "InterpolatePolarObjFlag" in the system configuration information is "1," it is determined that a polar coordinate-based interpolation object exists.

[0269] If it is determined in step S18 that there is a polar coordinate-based interpolation object, then the process proceeds to step S19.

[0270] In step S19, the coded data sending unit 22 loads the data of the polar coordinate-based interpolation object that constitutes the content into the transmission buffer 103 based on the viewpoint selection information and the system configuration information.

[0271] That is, for each of the three reference viewpoints indicated by the viewpoint selection information, the coded data sending unit 22 reads out the object polar coordinate coded data and coding gain information for each polar coordinate based interpolation object from the recording unit 102, and supplies and stores them in the transmission buffer 103. The coded data sending unit 22 also reads out the coded audio data of each polar coordinate based interpolation object from the recording unit 102, and supplies and stores them in the transmission buffer 103.

[0272] After the process of step S19 is performed, the process proceeds to step S20.

[0273] On the other hand, if it is determined in step S18 that there is no polar coordinate-based interpolation object, the process of step S19 is not performed, and the process then proceeds to step S20.

[0274] In step S20, the coded data sending unit 22 multiplexes the data of each object loaded into the transmission buffer 103 in the processes of steps S13 to S19 to generate a bitstream. In this case, the system configuration information may also be multiplexed. As a result, a bitstream in the format shown in Fig. 13 is generated, for example.

[0275] In step S21, the coded data sending unit 22 sends the generated bit stream to the client 12. In this way, the content is distributed to the client 12.

[0276] In step S22, the coded data sending unit 22 determines whether or not to end the process.

[0277] For example, when a request to stop the transmission of the content is received from the client 12, or when the transmission of all the data of the content is completed, it is determined that the processing is to be ended.

[0278] If it is determined in step S22 that the process is not yet finished, the process returns to step S12, and the above-described process is repeated.

[0279] On the other hand, if it is determined in step S22 that the processing is to be ended, each unit of the server 11 stops the processing it is performing, and the providing processing ends. For example, when transmission of all content data has been completed, the server 11 transmits information to the client 12 indicating that data transmission has been completed, and the providing processing ends.

[0280] In this way, the server 11 generates a bitstream including the objects that make up the content, that is, the data of the required types of objects, out of the data of the four types of objects, and transmits it to the client 12. In this way, content playback based on the intentions of the content creator can be realized.

[0281] <Description of playback audio data generation process> After initialization, the client 12 requests the server 11 to open a network session, and when a response is received from the server 11, the client 12 transmits a request to send system configuration information via the communication unit 111.

[0282] When the system configuration information is transmitted from the server 11 in response to the transmission request, the client 12 starts the playback audio data generation process.

[0283] The playback audio data generation process performed by the client 12 will be described below with reference to the flowchart of FIG.

[0284] In step S51, the communication unit 111 receives the system configuration information transmitted from the server 11 and supplies it to the viewpoint selection unit 42, the coordinate axis transformation processing unit 47, the object position calculation unit 48, and the object position calculation unit 71. At this time, the system configuration information may be decoded by the decoding unit 45 as necessary.

[0285] In step S52, the listener position information acquisition unit 41 acquires listener position information in response to operations by the listener, etc., and supplies it to the viewpoint selection unit 42, object position calculation unit 48, object position calculation unit 71, and polar coordinate conversion unit 49.

[0286] In step S53, the viewpoint selection unit 42 selects three reference viewpoints based on the system configuration information supplied from the communication unit 111 and the listener position information supplied from the listener position information acquisition unit 41, and supplies viewpoint selection information indicating the selection result to the communication unit 111.

[0287] For example, in step S53, three reference viewpoints surrounding the listening position indicated by the listener position information are selected from among a plurality of reference viewpoints indicated by the system configuration information.

[0288] Furthermore, the communication unit 111 requests the server 11 to start transmitting the bitstream.

[0289] In step S54, the communication unit 111 transmits the viewpoint selection information supplied from the viewpoint selection unit 42 to the server 11.

[0290] Then, the server 11 transmits the bit stream generated in step S20 of FIG.

[0291] In step S 55 , the communication unit 111 receives the bit stream transmitted from the server 11 and supplies it to the decoding unit 45 .

[0292] In step S56, the client 12 performs a polar coordinate position information generation process to generate polar coordinate position information for each object.

[0293] Here, the polar coordinate position information generation process corresponding to step S56 will be described with reference to the flowchart of FIG.

[0294] In step S81, the decoding unit 45 extracts and decodes the data of the absolute coordinate-based interpolation object from the bit stream supplied from the communication unit 111.

[0295] The decoding unit 45 supplies the object polar coordinate position information of the absolute coordinate-based interpolated object obtained by decoding to the coordinate conversion unit 46, and also supplies the gain information of the absolute coordinate-based interpolated object obtained by decoding to the object position calculation unit 48.

[0296] Furthermore, the decoding unit 45 supplies the audio data of the absolute coordinate-based interpolation object obtained by decoding to the polar coordinate conversion unit 49 .

[0297] In step S82, the coordinate conversion unit 46 performs coordinate conversion on the object polar coordinate position information of the absolute coordinate-based interpolation object supplied from the decoding unit 45, and supplies the resulting object absolute coordinate position information to the coordinate axis conversion processing unit 47.

[0298] This provides object absolute coordinate position information that indicates the position of the absolute coordinate-based interpolation object in an absolute coordinate system that has the position of the reference viewpoint as its origin.

[0299] In step S83, the coordinate axis conversion processing unit 47 performs coordinate axis conversion processing on the object absolute coordinate position information supplied from the coordinate conversion unit 46 based on the system configuration information supplied from the communication unit 111.

[0300] The coordinate axis conversion processing unit 47 performs coordinate axis conversion processing on the object absolute coordinate position information of the absolute coordinate-based interpolation object for each reference viewpoint, and supplies the resulting object absolute coordinate position information indicating the position of the absolute coordinate-based interpolation object in the common absolute coordinate system to the object position calculation unit 48. The coordinate axis conversion processing uses reference viewpoint information for the reference viewpoint included in the system configuration information, i.e., reference viewpoint position information and listener orientation information.

[0301] In step S84, the object position calculation unit 48 performs interpolation processing based on the system configuration information supplied from the communication unit 111, the listener position information supplied from the listener position information acquisition unit 41, the object absolute coordinate position information supplied from the coordinate axis transformation processing unit 47, and the gain information supplied from the decoding unit 45.

[0302] For example, the object position calculation unit 48 performs calculations similar to the above-mentioned equations (1) to (6) based on the reference viewpoint position information included in the system configuration information and the listener position information, to determine the internal division ratios (m, n) and (k, l).

[0303] Then, the object position calculation unit 48 performs calculations similar to the above-mentioned equations (7) to (14) based on the determined internal division ratios (m, n) and (k, l) and the object absolute coordinate position information of each reference viewpoint, thereby determining the final object absolute coordinate position information of the absolute coordinate-based interpolated object through interpolation processing.

[0304] Furthermore, the object position calculation unit 48 performs interpolation processing on the gain information in the same manner as the object absolute coordinate position information, and obtains final gain information of the absolute coordinate-based interpolated object.

[0305] The object position calculation unit 48 supplies the final object absolute coordinate position information and gain information of the absolute coordinate-based interpolated object obtained by the interpolation process to the polar coordinate conversion unit 49.

[0306] In step S85, the polar coordinate conversion unit 49 performs polar coordinate conversion on the object absolute coordinate position information of the absolute coordinate-based interpolation object supplied from the object position calculation unit 48, based on the listener position information supplied from the listener position information acquisition unit 41, to generate polar coordinate position information.

[0307] This provides polar coordinate position information, which is polar coordinates indicating the relative position of the absolute coordinate-based interpolation object as seen from the listening position.

[0308] Furthermore, the polar coordinate conversion unit 49 performs gain adjustment on the audio data of the absolute coordinate-based interpolation object supplied from the decoding unit 45 based on the gain information of the absolute coordinate-based interpolation object supplied from the object position calculation unit 48 .

[0309] The polar coordinate conversion unit 49 supplies the polar coordinate position information and audio data of the absolute coordinate-based interpolation object obtained in this manner to the rendering processing unit 113.

[0310] In step S86, the decoding unit 45 determines whether the bit stream supplied from the communication unit 111 contains data of a polar coordinate base fixed object.

[0311] For example, in step S86, if the value of the flag "NonInterpolatePolarObjFlag" in the system configuration information included in the bitstream or appropriately supplied from the communication unit 111 is "1", it is determined that data of a polar coordinate-based fixed object is present.

[0312] If it is determined in step S86 that data of the polar coordinate base fixed object is present, the decoding unit 45 extracts and decodes the data of the polar coordinate base fixed object from the bit stream supplied from the communication unit 111 in step S87.

[0313] The decoding unit 45 adjusts the gain of the audio data of the polar coordinate base fixed object obtained by decoding, as appropriate, based on the gain information of the polar coordinate base fixed object obtained by decoding.

[0314] The decoding unit 45 also supplies the object polar coordinate position information of the polar coordinate-based fixed object obtained by decoding and the gain-adjusted audio data of the polar coordinate-based fixed object to the rendering processing unit 113. In this case, the object polar coordinate position information of the polar coordinate-based fixed object is supplied to the rendering processing unit 113 as the polar coordinate position information of the polar coordinate-based fixed object as is.

[0315] After the process of step S87 is performed, the process proceeds to step S88.

[0316] On the other hand, if it is determined in step S86 that there is no data for the polar coordinate base fixed object, then the process proceeds to step S88.

[0317] In step S88, the decoding unit 45 determines whether or not the bit stream supplied from the communication unit 111 contains data of an absolute coordinate base fixed object.

[0318] For example, in step S88, if the value of the flag "NonInterpolateCartesianObjFlag" in the system configuration information included in the bitstream or appropriately supplied from the communication unit 111 is "1", it is determined that data of an absolute coordinate-based fixed object is present.

[0319] If it is determined in step S88 that there is data of the absolute coordinate base fixed object, the decoding unit 45 extracts and decodes the data of the absolute coordinate base fixed object from the bit stream supplied from the communication unit 111 in step S89.

[0320] The decoding unit 45 supplies the object absolute coordinate position information of the absolute coordinate base fixed object obtained by decoding, gain information, and audio data to the polar coordinate conversion unit 49 .

[0321] In step S90, the polar coordinate conversion unit 49 performs polar coordinate conversion on the object absolute coordinate position information of the absolute coordinate-based fixed object supplied from the decoding unit 45, based on the listener position information supplied from the listener position information acquisition unit 41, to generate polar coordinate position information.

[0322] This provides polar coordinate position information, which is polar coordinates indicating the relative position of the absolute coordinate-based fixed object as seen from the listener (listening position).

[0323] Furthermore, the polar coordinate conversion unit 49 performs gain adjustment on the audio data of the absolute coordinate base fixed object supplied from the decoding unit 45 based on the gain information of the absolute coordinate base fixed object supplied from the decoding unit 45 .

[0324] The polar coordinate conversion unit 49 supplies the polar coordinate position information of the absolute coordinate base fixed object thus obtained and the audio data to the rendering processing unit 113 .

[0325] After the process of step S90 is performed, the process proceeds to step S91.

[0326] On the other hand, if it is determined in step S88 that there is no data of the absolute coordinate base fixed object, then the process proceeds to step S91.

[0327] In step S91, the decoding unit 45 determines whether or not the bit stream supplied from the communication unit 111 contains data of a polar coordinate-based interpolation object.

[0328] For example, in step S91, if the value of the flag "InterpolatePolarObjFlag" in the system configuration information included in the bitstream or appropriately supplied from the communication unit 111 is "1", it is determined that data of the polar coordinate-based interpolation object is present.

[0329] If it is determined in step S91 that data of the polar coordinate base interpolation object is present, the decoding unit 45 extracts and decodes the data of the polar coordinate base interpolation object from the bit stream supplied from the communication unit 111 in step S92.

[0330] The decoding unit 45 supplies the object polar coordinate position information, gain information, and audio data of the polar coordinate-based interpolation object obtained by decoding to the object position calculation unit 71 .

[0331] In step S93, the object position calculation unit 71 performs interpolation processing based on the system configuration information supplied from the communication unit 111, the listener position information supplied from the listener position information acquisition unit 41, and the object polar coordinate position information and gain information of the polar coordinate-based interpolation object supplied from the decoding unit 45.

[0332] For example, the object position calculation unit 71 performs calculations similar to the above equations (1) to (3) based on the reference viewpoint position information and listener position information included in the system configuration information to find the internal division ratio (m, n).

[0333] Then, the object position calculation unit 71 performs calculations similar to the above-mentioned equations (15) and (16) based on the determined internal division ratio (m, n), the reference viewpoint position information, the object polar coordinate position information of the polar coordinate-based interpolation object at each reference viewpoint, and the listener position information, thereby determining the polar coordinate position information of the polar coordinate-based interpolation object through interpolation processing.

[0334] In addition, the object position calculation unit 71 performs interpolation processing on the gain information in the same way as the polar coordinate position information, and adjusts the gain of the audio data of the polar coordinate-based interpolated object based on the final gain information of the polar coordinate-based interpolated object obtained as a result.

[0335] The object position calculation unit 71 supplies the rendering processing unit 113 with the polar coordinate position information of the polar coordinate-based interpolated object obtained by the interpolation processing and gain adjustment, and the audio data.

[0336] When the process of step S93 is performed, the polar coordinate position information generation process ends, and then the process proceeds to step S57 in FIG.

[0337] Also, if it is determined in step S91 that there is no data for the polar coordinate-based interpolation object, the processes of steps S92 and S93 are not performed, the polar coordinate position information generation process ends, and then the process proceeds to step S57 in Figure 19.

[0338] Returning to the explanation of the flowchart in Figure 19, in step S57, the rendering processing unit 113 performs rendering processing such as VBAP based on the supplied polar coordinate position information and audio data of each object, and outputs the resulting playback audio data.

[0339] In step S57, rendering processing is performed based on the polar coordinate position information and audio data of the absolute coordinate-based interpolated object and the absolute coordinate-based fixed object supplied from the polar coordinate conversion unit 49, the polar coordinate position information and audio data of the polar coordinate-based fixed object supplied from the decoding unit 45, and the polar coordinate position information and audio data of the polar coordinate-based interpolated object supplied from the object position calculation unit 71.

[0340] Furthermore, for example, in a speaker or the like downstream of the rendering processing unit 113, the sound of the content is reproduced based on the reproduced audio data.

[0341] In step S58, the client 12 determines whether or not to end the processing it is currently performing. For example, in step S58, it is determined that the processing should end when a listener issues a command to stop playback of the content, or when all data of the content has been received and played back.

[0342] If it is determined in step S58 that the process is not yet finished, the process returns to step S52, and the above-described process is repeated.

[0343] On the other hand, if it is determined in step S58 that the process is to be ended, the client 12 ends the session with the server 11, stops the processes being performed by each unit, and ends the playback audio data generation process.

[0344] In this way, the client 12 performs appropriate processing on each type of object contained in the received bitstream to generate playback audio data. This allows content playback based on the intentions of the content creator, and the enjoyment of the content can be fully conveyed to the listener.

[0345] As described above, according to the present technology, by combining objects whose positions change depending on the listener's position and objects whose positions are fixed and listener-centered and do not change regardless of the listener's position, data of each type of object can be coded and efficiently transmitted and played back.

[0346] This makes it possible to realize object position representations determined by interpolation at arbitrary positions based on the content creator's intended object placement in 3D space using Audio Artistic Intent. It is also possible to realize objects that are always fixed regardless of the listener's orientation, and objects that are positioned at absolute positions within the free viewpoint space.

[0347] This will enable applications such as acoustic AR and guidance support systems based on the direction of sound arrival, as well as the realization of a world in which Artistic Intent content is reproduced using objects that are appropriately positioned through interpolation processing within a free viewpoint space that contains fixed background noise.

[0348] <Example of computer configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0349] FIG. 21 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes using a program.

[0350] In the computer, a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, and a RAM (Random Access Memory) 503 are interconnected by a bus 504.

[0351] An input / output interface 505 is further connected to the bus 504. An input unit 506, an output unit 507, a recording unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.

[0352] The input unit 506 includes a keyboard, a mouse, a microphone, an image sensor, etc. The output unit 507 includes a display, a speaker, etc. The recording unit 508 includes a hard disk, a nonvolatile memory, etc. The communication unit 509 includes a network interface, etc. The drive 510 drives a removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0353] In a computer configured as described above, the CPU 501 performs the above-described series of processes by, for example, loading a program recorded in the recording unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executing it.

[0354] The program executed by the computer (CPU 501) can be provided by being recorded on a removable recording medium 511 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0355] In a computer, a program can be installed in the recording unit 508 via the input / output interface 505 by inserting a removable recording medium 511 into the drive 510. The program can also be received by the communication unit 509 via a wired or wireless transmission medium and installed in the recording unit 508. Alternatively, the program can be installed in the ROM 502 or the recording unit 508 in advance.

[0356] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0357] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0358] For example, this technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0359] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by multiple devices.

[0360] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0361] Furthermore, the present technology can also be configured as follows.

[0362] (1) a listener position information acquisition unit that acquires listener position information of a listener's viewpoint; a reference viewpoint information acquisition unit that acquires position information of a first reference viewpoint, object position information of a first object at the first reference viewpoint, position information of a second reference viewpoint, and object position information of the first object at the second reference viewpoint, and also acquires object position information of a second object; an object position calculation unit that calculates position information of the first object in the listener's viewpoint based on the listener position information, the position information of the first reference viewpoint, the object position information of the first object in the first reference viewpoint, the position information of the second reference viewpoint, and the object position information of the first object in the second reference viewpoint; An information processing device comprising: (2) The first reference viewpoint and the second reference viewpoint are viewpoints that are preset by a content creator. The information processing device described in (1). (3) The first reference viewpoint and the second reference viewpoint are viewpoints selected based on the listener position information. An information processing device according to (1) or (2). (4) The object position calculation unit calculates position information of the first object from the viewpoint of the listener by interpolation processing. An information processing device according to any one of (1) to (3). (5) the object position information of the first object at the first reference viewpoint is coordinate information indicating a relative position of the first object with respect to the first reference viewpoint; The object position information of the first object at the second reference viewpoint is coordinate information indicating a relative position of the first object with respect to the second reference viewpoint. (4) An information processing device according to the present invention. (6) The object position information of the first object is information indicating a position expressed in polar coordinates. (5) An information processing device according to the present invention. (7) The object position calculation unit performs the interpolation process based on the object position information of the first object expressed in polar coordinates. (6) An information processing device according to (6). (8) The object position calculation unit converts the object position information of the first object expressed in polar coordinates into object absolute coordinate position information that indicates an absolute position of the first object in a common absolute coordinate space expressed in absolute coordinates, and performs the interpolation process based on the object absolute coordinate position information. An information processing device according to any one of (4) to (6). (9) the reference viewpoint information acquisition unit acquires, for three or more reference viewpoints including the first reference viewpoint and the second reference viewpoint, the position information of the reference viewpoints and the object position information of the first object at the reference viewpoint; The object position calculation unit performs the interpolation process based on the listener position information, the position information of each of three of the plurality of reference viewpoints, and the object position information of the first object at each of the three reference viewpoints. (4) An information processing device according to the present invention. (10) The object position information of the second object is coordinate information indicating the relative position of the second object with respect to the position of the listener. An information processing device according to any one of (1) to (9). (11) the object position information of the second object is coordinate information indicating an absolute position of the second object in a common absolute coordinate space; The object position calculation unit converts the object position information of the second object into position information of the second object from the viewpoint of the listener. An information processing device according to any one of (1) to (9). (12) The object position calculation unit calculates position information of the first object in the listener's viewpoint based on the listener position information, the position information of the first reference viewpoint, the object position information of the first object at the first reference viewpoint, listener direction information indicating a facial direction of the listener set at the first reference viewpoint, the position information of the second reference viewpoint, the object position information of the first object at the second reference viewpoint, and the listener direction information at the second reference viewpoint. An information processing device according to any one of (1) to (11). (13) The reference viewpoint information acquisition unit acquires configuration information including the position information and the listener direction information of each of three or more reference viewpoints including the first reference viewpoint and the second reference viewpoint. (12) An information processing device according to (12). (14) The configuration information includes information indicating the number of the plurality of reference viewpoints, and information indicating the number of the first objects and the second objects. (13) An information processing device according to (13). (15) The information processing device Obtaining listener position information of the listener's viewpoint; acquiring position information of a first reference viewpoint, object position information of a first object at the first reference viewpoint, position information of a second reference viewpoint, object position information of the first object at the second reference viewpoint, and object position information of a second object; calculating position information of the first object at the listener's viewpoint based on the listener position information, the position information of the first reference viewpoint, the object position information of the first object at the first reference viewpoint, the position information of the second reference viewpoint, and the object position information of the first object at the second reference viewpoint; Information processing methods. (16) Obtaining listener position information of the listener's viewpoint; acquiring position information of a first reference viewpoint, object position information of a first object at the first reference viewpoint, position information of a second reference viewpoint, object position information of the first object at the second reference viewpoint, and object position information of a second object; calculating position information of the first object at the listener's viewpoint based on the listener position information, the position information of the first reference viewpoint, the object position information of the first object at the first reference viewpoint, the position information of the second reference viewpoint, and the object position information of the first object at the second reference viewpoint; A program that causes a computer to perform a process. [Explanation of symbols]

[0363] 11 server, 12 client, 21 configuration information transmission unit, 22 encoded data transmission unit, 41 listener position information acquisition unit, 42 viewpoint selection unit, 45 decoding unit, 48 object position calculation unit, 71 object position calculation unit, 111 communication unit, 112 position calculation unit, 113 rendering processing unit

Claims

1. a listener position information acquisition unit that acquires listener position information of a listener's viewpoint; a reference viewpoint information acquisition unit that acquires position information of a first reference viewpoint, object position information of a first object at the first reference viewpoint, position information of a second reference viewpoint, and object position information of the first object at the second reference viewpoint, and also acquires object position information of a second object; an object position calculation unit that calculates position information of the first object in the listener's viewpoint based on the listener position information, the position information of the first reference viewpoint, the object position information of the first object in the first reference viewpoint, the position information of the second reference viewpoint, and the object position information of the first object in the second reference viewpoint; Equipped with the object position information of the second object is coordinate information indicating a relative position of the second object with respect to a position of the listener; or The object position information of the second object is coordinate information indicating an absolute position of the second object in a common absolute coordinate space, and the object position calculation unit converts the object position information indicating the absolute position of the second object in the common absolute coordinate space into position information of the second object from the viewpoint of the listener. Information processing device.

2. The first reference viewpoint and the second reference viewpoint are viewpoints that are set in advance by a content creator. The information processing device according to claim 1 .

3. The first reference viewpoint and the second reference viewpoint are viewpoints selected based on the listener position information. The information processing device according to claim 1 .

4. The object position calculation unit calculates position information of the first object from the viewpoint of the listener by interpolation processing. The information processing device according to claim 1 .

5. the object position information of the first object at the first reference viewpoint is coordinate information indicating a relative position of the first object with respect to the first reference viewpoint; The object position information of the first object at the second reference viewpoint is coordinate information indicating a relative position of the first object with respect to the second reference viewpoint. The information processing device according to claim 4 .

6. The object position information of the first object is information indicating a position expressed in polar coordinates. The information processing device according to claim 5 .

7. The object position calculation unit performs the interpolation process based on the object position information of the first object expressed in polar coordinates. The information processing device according to claim 6 .

8. The object position calculation unit converts the object position information of the first object expressed in polar coordinates into object absolute coordinate position information that indicates an absolute position of the first object in the common absolute coordinate space, expressed in absolute coordinates, and performs the interpolation process based on the object absolute coordinate position information. The information processing device according to claim 4 .

9. the reference viewpoint information acquisition unit acquires, for three or more reference viewpoints including the first reference viewpoint and the second reference viewpoint, the position information of the reference viewpoints and the object position information of the first object at the reference viewpoints; The object position calculation unit performs the interpolation process based on the listener position information, the position information of each of three of the reference viewpoints, and the object position information of the first object at each of the three reference viewpoints. The information processing device according to claim 4 .

10. The object position calculation unit calculates position information of the first object in the viewpoint of the listener based on the listener position information, the position information of the first reference viewpoint, the object position information of the first object at the first reference viewpoint, listener direction information indicating a facial direction of the listener set at the first reference viewpoint, the position information of the second reference viewpoint, the object position information of the first object at the second reference viewpoint, and the listener direction information at the second reference viewpoint. The information processing device according to claim 1 .

11. The reference viewpoint information acquisition unit acquires configuration information including the position information and the listener direction information of each of three or more reference viewpoints including the first reference viewpoint and the second reference viewpoint. The information processing device according to claim 10.

12. The configuration information includes information indicating the number of the plurality of reference viewpoints, and information indicating the number of the first objects and the second objects. The information processing device according to claim 11.

13. The information processing device Obtaining listener position information of the listener's viewpoint; acquiring position information of a first reference viewpoint, object position information of a first object at the first reference viewpoint, position information of a second reference viewpoint, object position information of the first object at the second reference viewpoint, and object position information of a second object; calculating position information of the first object at the listener's viewpoint based on the listener position information, the position information of the first reference viewpoint, the object position information of the first object at the first reference viewpoint, the position information of the second reference viewpoint, and the object position information of the first object at the second reference viewpoint; the object position information of the second object is coordinate information indicating a relative position of the second object with respect to a position of the listener; or The object position information of the second object is coordinate information indicating an absolute position of the second object in a common absolute coordinate space, and the object position information indicating the absolute position of the second object in the common absolute coordinate space is converted into position information of the second object from the viewpoint of the listener. Information processing methods.

14. Obtaining listener position information of the listener's viewpoint; acquiring position information of a first reference viewpoint, object position information of a first object at the first reference viewpoint, position information of a second reference viewpoint, object position information of the first object at the second reference viewpoint, and object position information of a second object; Calculating position information of the first object at the listener's viewpoint based on the listener position information, the position information of the first reference viewpoint, the object position information of the first object at the first reference viewpoint, the position information of the second reference viewpoint, and the object position information of the first object at the second reference viewpoint. Have the computer execute the process, the object position information of the second object is coordinate information indicating a relative position of the second object with respect to a position of the listener; or The object position information of the second object is coordinate information indicating an absolute position of the second object in a common absolute coordinate space, and the object position information indicating the absolute position of the second object in the common absolute coordinate space is converted into position information of the second object from the viewpoint of the listener. program.

Images