Information processing apparatus and method, and program

Abstract

The present technology relates to an information processing apparatus and method and a program that make it possible to appropriately display content. The information processing apparatus is provided with a position determination unit that, based on position information indicating a position of a user in a space, and prohibition space information for identifying a prohibition space that includes a predetermined target point and prohibits the user from intruding, identifies whether the position of the user indicated by the position information is located within the prohibition space, and, when the position of the user is within the prohibition space, changes the position of the user to a position that is located outside the prohibition space and is different from the position indicated by the position information. The present technology is applicable to an information processing apparatus.

Description

Technical Field

[0001] This technology relates to information processing apparatus, methods, and procedures, and more specifically, to information processing apparatus, methods, and procedures capable of appropriately performing content-related displays. Background Technology

[0002] In existing technologies, free-viewpoint technology (also known as 6 degrees of freedom (DoF)) has been utilized in various types of content such as games and music. In free-viewpoint content, users (listeners) can not only rotate their heads in three-dimensional space along vertical, horizontal, and diagonal directions, but also move to any position in space to view the content.

[0003] For example, as a technology related to free-viewpoint audio, technologies that enable content playback based on the content creator's intent have been proposed (see, for example, Patent Document 1).

[0004] In Patent Document 1, 3DoF content is created at multiple control viewpoints (referred to as control viewpoints (CVPs)) set by the creator on a free-viewpoint audio system. For example, a CVP is the desired listening position during content playback.

[0005] Specifically, in Patent Document 1, the target point (hereinafter also referred to as TP) is set as the common point (location) of all CVPs in three-dimensional space, and 3DoF content is created with TP as the midplane. That is, 3DoF content is generated assuming that the listener at the CVP is facing (looking at) TP.

[0006] Reference List

[0007] Patent documents

[0008] PTL 1: WO2023 / 085140 Summary of the Invention

[0009] Technical issues

[0010] In Patent Document 1, a situation is considered in which, on the playback side of the content, the listener in the three-dimensional space (i.e., the free viewpoint space) moves while continuously viewing the TP.

[0011] In this case, based on the listener's position and the TP's position in three-dimensional space, the listener's facial orientation displayed by the playback system using a three-dimensional user interface (UI) can be uniquely determined through a calculation formula.

[0012] However, when the listener and TP are in the same position, that is, when the listener is at TP, it becomes impossible to perform calculations according to the formula, potentially leading to the inappropriate display of the UI (UI representation).

[0013] Given this situation, the purpose of this technology is to enable appropriate display that is relevant to the content.

[0014] Solution to the problem

[0015] An information processing apparatus according to one aspect of the present technology includes: a position determination unit configured to, based on position information indicating the position of a user in space and prohibited space information for specifying a prohibited space including a predetermined target point and wherein the user is prohibited from entering, identify whether the position of the user indicated by the position information is within the prohibited space, and when the user's position is within the prohibited space, change the user's position to a position outside the prohibited space and different from the position indicated by the position information.

[0016] An information processing method or procedure according to one aspect of the present technology includes: identifying whether the user's position indicated by the location information is within the prohibited space based on location information representing the user's position in space and prohibited space information for specifying a prohibited space including a predetermined target point and wherein the user is prohibited from entering; and when the user's position is within the prohibited space, changing the user's position to a position outside the prohibited space and different from the position indicated by the location information.

[0017] In one aspect of this technology, based on location information representing the user's position in space and prohibited space information for specifying a prohibited space including a predetermined target point and in which the user is prohibited from entering, it is identified whether the user's position represented by the location information is within the prohibited space. When the user's position is within the prohibited space, the user's position is changed to a position outside the prohibited space and different from the position represented by the location information. Attached Figure Description

[0018] [ Figure 1 [This is a diagram showing the orientation of the listener's face.]

[0019] [ Figure 2 [] is a diagram showing the prohibited space.

[0020] [ Figure 3 The image below is an example diagram showing configuration information.

[0021] [ Figure 4 [This is a diagram showing the listener entering a prohibited space.]

[0022] [ Figure 5 [] is a diagram showing an example of server configuration.

[0023] [ Figure 6 [] is a flowchart illustrating the encoding process.

[0024] [ Figure 7 [] is a diagram showing an example of client configuration.

[0025] [ Figure 8 [ ] is a flowchart illustrating the output audio data generation process.

[0026] [ Figure 9 [] is a flowchart illustrating the metadata generation process.

[0027] [ Figure 10 [] is a diagram showing vector composition.

[0028] [ Figure 11 [] is a diagram showing vector composition.

[0029] [ Figure 12 [ ] is a diagram showing the contribution rate of each vector during vector composition.

[0030] [ Figure 13 [] is a flowchart illustrating the metadata generation process.

[0031] [ Figure 14 [ ] is a diagram showing an example of a computer configuration. Detailed Implementation

[0032] In the following description, embodiments employing this technology are described with reference to the accompanying drawings.

[0033] <First Implementation Method>

[0034] <This technology>

[0035] This technology enables free-viewpoint content with artistic quality.

[0036] For example, this technology has the following features T1 to T4.

[0037] (Feature T1)

[0038] In three-dimensional space, the space (region) centered on the target point (TP) where the presence of the listener is prohibited is pre-set as a prohibited space.

[0039] (Feature T2)

[0040] When a listener attempts to enter a prohibited space, a signal indicating that entry is not permitted is generated, a substitute listener position is determined, and object metadata for that substitute listener position is generated.

[0041] (Feature T3)

[0042] When generating object metadata for the alternative listener location, the alternative listener location is set to the position closest to the listener location on the boundary surface of the forbidden space, and a forced movement to that alternative listener location is performed.

[0043] (Feature T4)

[0044] When the listener's location coincides with the TP, the object metadata generated for the listener's location just before the coincidence with the TP is used.

[0045] The technology is now described below.

[0046] In audio playback systems employing this technology, users can rotate their heads vertically, horizontally, or diagonally within a space and move to any position within that space to view the content. Such content is referred to as 6DoF content, free-viewpoint content, etc.

[0047] The following content may consist solely of audio, or it may consist of video and accompanying audio; however, in the following text, such content will be collectively referred to as "content" without specific distinction. Furthermore, in the following text, audio objects will be simply referred to as "objects".

[0048] In the content production process, in order to improve the quality of art (music), objects can be intentionally placed in locations different from their visually perceptible positions, without being constrained by the physical arrangement of the objects.

[0049] Specifically, for example, in the case of a band performance, the objects (audio objects) corresponding to the singers or guitars are not physically placed at the individual singers' or guitars' positions, but rather placed at positions based on music quality. Note that "based on music quality" means making the music easier to listen to as a piece of music.

[0050] As an example, instead of placing the mono object directly at the physical position of the guitarist in three-dimensional space, the object is placed on the left and right sides in front of the user based on musical knowledge. By placing the extended object in the left and right front positions from the user's perspective in the manner described above, an audio representation that surrounds the listener (user) can be achieved.

[0051] As mentioned above, the creator generates free-viewpoint content by placing various objects in space based on music quality.

[0052] In three-dimensional space (hereinafter also referred to as free viewpoint space), the content creator sets the positions used as multiple control viewpoints (hereinafter also referred to as control viewpoints (CVP)) and the position used as a target point (hereinafter also referred to as target point (TP)).

[0053] A CVP is the location of a viewpoint to be represented in free viewpoint space. For example, the desired location of the listening position during content playback is defined as a CVP. In the following text, the i-th CVP will be specifically referred to as CVPi.

[0054] The target point (TP) is the reference position used for interpolation processing of the object's location information, and it is assumed that the virtual listener at each CVP faces TP. Furthermore, the actual listener (user) in the free viewpoint space can move freely to any position, but is also assumed to always face TP. In other words, it is assumed that the listener can move within the free viewpoint space while facing TP.

[0055] The positions of CVP and TP are represented by absolute coordinates that indicate their absolute positions within the free viewpoint space. For example, when the coordinate system representing the absolute coordinates of the absolute positions within the free viewpoint space is called the common absolute coordinate system, the common absolute coordinate system is defined as an orthogonal coordinate system with a predetermined position within the free viewpoint space as the origin O and mutually orthogonal X-axis, Y-axis, and Z-axis as axes.

[0056] Additionally, the placement of objects is set for each CVP. For example, at each CVP, a polar coordinate space centered on the CVP's ​​location is formed (hereinafter also referred to as the CVP polar coordinate space), and each object is placed within the CVP polar coordinate space.

[0057] More specifically, for example, the position in the CVP polar coordinate space is represented by the coordinates (polar coordinates) of a polar coordinate system consisting of mutually orthogonal x-axis, y-axis and z-axis (hereinafter also referred to as the CVP polar coordinate system), where the origin O'' is located in the CVP, that is, the absolute position of the CVP in the free viewpoint space (common absolute coordinate system).

[0058] Specifically, in this paper, the direction from the position of CVP toward TP is defined as the positive direction of the y-axis, that is, the forward direction of the virtual listener located at CVP; the axis of the left and right direction viewed by the virtual listener at CVP is defined as the x-axis; and the axis of the up and down direction viewed by the virtual listener at CVP is defined as the z-axis.

[0059] Incidentally, the listener can move to any location within the free viewpoint space.

[0060] During content playback, the playback system can display a three-dimensional user interface (hereinafter also referred to as the free viewpoint space UI) based on the listener's position and orientation in free viewpoint space.

[0061] For example, a free-viewpoint space UI is an image associated with free-viewpoint space, such as an image of a 3D space simulating free-viewpoint space. Note that a free-viewpoint space UI can be an image (video) of the content to be played back.

[0062] As a specific example, a free-viewpoint space UI can be an image of the entire free-viewpoint space, showing the listener's position and facial orientation, or an image of the free-viewpoint space viewed from the listener's position along the direction of their face. In either case, the free-viewpoint space UI is generated based on the listener's position and orientation within the free-viewpoint space.

[0063] Furthermore, it is assumed that the listener in the free viewpoint space always moves while looking at the TP. In this case, the listener's position and facial orientation are used for the generation and display of the free viewpoint space UI, and the listener's facial orientation can be uniquely determined.

[0064] For example, suppose a predetermined position on a platform within free viewpoint space is set as TP, and by means of... Figure 1 The coordinates (TpX, TpY, TpZ) in the common absolute coordinate system shown represent the position of TP. Figure 1 In this context, O is the origin of the common absolute coordinate system.

[0065] Furthermore, assume that the listener is located at position O'' in the free viewpoint space, and that the listener's position (position O'') in the free viewpoint space is represented by coordinates (LpX, LpY, LpZ) in a common absolute coordinate system.

[0066] Here, the polar coordinate space with the listener's position (position O'') as its origin (hereinafter also referred to as origin O'') is called the listener polar coordinate space. Furthermore, it is assumed that the position in the listener polar coordinate space is represented by the coordinates (polar coordinates) of the polar coordinate system (hereinafter also referred to as the listener polar coordinate system), which includes mutually orthogonal x-axis, y-axis and z-axis with position O'' as the origin.

[0067] In this case, the listener's orientation (facial orientation) in the free viewpoint space can be represented by the horizontal angle Yaw, which indicates orientation in the horizontal direction, and the vertical angle Pitch, which indicates orientation in the vertical direction.

[0068] Here, when we assume that the X'Y' plane is the plane that includes the listener (i.e., the origin O'' of the listener's polar coordinate system) and is parallel to the XY plane that includes the X-axis and Y-axis of the common absolute coordinate system, the line LN11 is the line obtained by projecting the y-axis of the listener's polar coordinate system onto the X'Y' plane. Furthermore, the line LN12 is a line parallel to the Y-axis of the common absolute coordinate system on the X'Y' plane.

[0069] In this case, the angle between the surface of line LN11 and line LN12 is the horizontal angle Yaw, which represents the listener's orientation in the horizontal direction, and the angle between the y-axis and the surface of line LN11 is the vertical angle Pitch, which represents the listener's orientation in the vertical direction.

[0070] Based on the coordinates of TP (TpX, TpY, TpZ) and the coordinates of the listener's position (LpX, LpY, LpZ), the horizontal angle Yaw and the vertical angle Pitch can be determined by using the following formulas (1) and (2).

[0071] [Mathematical Expression 1]

[0072] [Mathematical Expression 2]

[0073] However, when TP and the listener position have the same value in the parts used to obtain Tmp1 in formula (1) and Tmp2 in formula (2), the denominators (numerical values) of Tmp1 and Tmp2 become zero, and therefore the values ​​of Tmp1 and Tmp2 become infinity.

[0074] As a result, subsequent calculations become impossible, and the horizontal angle (Yaw) and vertical angle (Pitch) cannot be obtained, which in turn makes it impossible to generate and display a free viewpoint space UI. In other words, it becomes impossible to properly display a free viewpoint space UI.

[0075] Therefore, in this technology, a forbidden space is defined, which is a space within the free viewpoint space that the listener (user) is prohibited from entering, that is, a space that the user cannot enter.

[0076] By defining and prohibiting entry into this forbidden space, situations where the listener's orientation cannot be calculated can be prevented, and the free viewpoint space UI display can always be implemented appropriately. Furthermore, by defining the forbidden space, the content creator's intentions can be reflected, such as not wanting the listener to approach a predetermined location, such as a TP (Viewpoint Point) or a given object.

[0077] Forbidden space is a space of arbitrary shape containing TP.

[0078] More specifically, for example, forbidden spaces are like... Figure 2 The spherical space is shown. In this example, TP is a location in the free viewpoint space (FVS) with a cuboid shape.

[0079] Furthermore, a sphere P11 with a predetermined radius centered at TP is a forbidden space. In other words, the center of the sphere P11, which is set as a forbidden space, is TP.

[0080] Listeners are allowed to move freely within the free viewpoint space FVS while viewing the content, but are not allowed to move (enter) into the sphere P11, which is a forbidden space.

[0081] Note that the forbidden space is not limited to a spherical space (sphere), and can be any space with a shape such as a cube, cuboid, polyhedron, or plane, as long as the space includes the TP. Specifically, the TP itself can be set as the forbidden space. Furthermore, the forbidden space does not have to be centered on the TP. Specifically, the location of the TP is not limited to the center of the forbidden space, and can be any location as long as it is within the forbidden space. Additionally, a forbidden space can be defined as a space that includes both the TP and a specific object, such as an object located within a free viewpoint space that the listener should not approach, or an object located in a fixed position within the free viewpoint space regardless of the viewpoint position.

[0082] In this embodiment, it is assumed that the forbidden space is a spherical space, and specifically, the spherical space set as the forbidden space is also called the forbidden space sphere.

[0083] For example, Figure 3 The configuration information in the format shown is provided from the server side that delivers the content to the playback side that plays the content.

[0084] Specifically, Figure 3 An example of the bitstream format for configuration information is shown. Note that, more specifically, the configuration information includes not only... Figure 3 The information shown, and including other information such as information relating to the CVP described later, and illustrations of such information in Figure 3 The middle part is omitted.

[0085] exist Figure 3 In the example shown, the configuration information includes a flag “TPProhibitPresentFlag” indicating whether there is a definition of a prohibited space, i.e., prohibited space information.

[0086] Here, when the value of the prohibited space information existence flag is 1, it indicates that prohibited space information exists to specify the prohibited space, meaning the configuration information contains prohibited space information. On the other hand, when the value of the prohibited space information existence flag is 0, it indicates that prohibited space information does not exist, and the configuration information does not contain prohibited space information.

[0087] When the value of the prohibited space information existence flag is 1, the configuration information stores the radius information "TPProhibitRadius", which represents the radius of the prohibited space sphere, as prohibited space information.

[0088] For example, the radius information represents a real number representing the radius of a forbidden space sphere when the longest side of a cuboid defined in three-dimensional free viewpoint space is normalized to 1.0.

[0089] Note that the radius information can be a value set by the content production tool, configured on the playback side, or configured to be rewritable. For example, on the playback side, the radius information can be changed based on object metadata such as object priority, object size, or the size of the object's region indicated by extended information. For instance, when object metadata indicating object priority is defined as a value from 0 to 7, the radius of the forbidden space sphere can be dynamically changed based on the priority value. Furthermore, the radius of the forbidden space sphere can be dynamically changed based on resource availability or remaining battery levels on the playback side.

[0090] Furthermore, the prohibited space can be predetermined, and the configuration information can store only the radius information without storing the prohibited space presence flag. Additionally, if the radius value is predetermined, the configuration information can store only the prohibited space presence flag without storing the radius information. Furthermore, the prohibited space can be defined independently on the content playback side.

[0091] On the playback side, the listener's position within the free viewpoint space is adjusted (changed) based on configuration information, ensuring the listener does not enter the prohibited space. Specifically, the final listener position is determined.

[0092] For example, the listener position input on the content playback side, the listener position input by the user (listener), or the listener position within a free viewpoint space set based on the output of sensors attached to the user are also referred to as input listener position, and the information representing the input listener position is called input listener position information. Furthermore, head tracking information or eye tracking information can be input using sensors mounted on head-mounted displays (HMDs) used for augmented reality (AR), virtual reality (VR), etc.

[0093] In addition, the information that represents the final listener position in the free viewpoint space determined by the system on the content playback side based on input listener position information, configuration information, etc., is called listener position information.

[0094] In this technology, for example, when the input listener position is within a restricted space, the listener position is changed. Specifically, a position outside the restricted space, different from the input listener position, is set as the final listener position. At this time, the final listener position (the changed listener position) is set to the position closest to the input listener position at the boundary of the restricted space, etc. Furthermore, a position different from the position on the boundary of the restricted space, but closest to the input listener position outside the restricted space, can be set as the final listener position.

[0095] Now, a specific example of a method for determining the final listener's location is described.

[0096] First, on the playback side, calculate the distance Dist from the listener's position (input listener's position) to TP.

[0097] More specifically, suppose the coordinates of the input listener position U in the free viewpoint space (i.e., the common absolute coordinate system) are (UpX, UpY, UpZ), and the coordinates of the position of TP in the common absolute coordinate system are (TpX, TpY, TpZ). In this case, the direction vectors (LVx, LVy, LVz) with TP as the starting point and the input listener position as the ending point are represented by the following formula (3).

[0098] [Mathematical Expression 3]

[0099] Furthermore, the distance Dist can be determined by using the following formula (4) to determine the magnitude (length) of the direction vectors (LVx, LVy, LVz).

[0100] [Mathematical Expression 4]

[0101] For example, such as Figure 4 As shown, consider the case where there are positions A, B, and C, and position A or position C is set as the input listener's position. Note that in Figure 4 In, corresponding to Figure 2 The parts in which the same reference numerals are used are indicated by the same reference numerals, and their descriptions are omitted as necessary. Furthermore, in Figure 4 For clarity, sphere P11, which is shown as a forbidden space sphere, is shown in an enlarged form.

[0102] exist Figure 4 In the example shown, position A is located inside sphere P11, position B is located on the surface of sphere P11, and position C is located outside sphere P11. Furthermore, assume that the coordinates of positions A, B, and C in the free viewpoint space FVS (i.e., the common absolute coordinate system) are (Ax, Ay, Az), (Bx, By, Bz), and (Cx, Cy, Cz).

[0103] Specifically, for example, the distance Dist when the input listener position is at position C outside the sphere P11 is called distance DistCT, and the distance Dist when the input listener position is at position A inside the sphere P11 is called distance DistAT.

[0104] In this case, it can be understood that since the distance DistCT has a value greater than the radius of the sphere P11 (i.e. the radius represented by the radius information TPProhibitRadius), the position C, which is the input listener position, is located outside the sphere P11, which is the forbidden space.

[0105] Therefore, in this case, the input listener position can be directly set as the final listener position to generate the object's metadata, or a free viewpoint space UI can be generated.

[0106] On the other hand, it can be understood that since the distance DistAT has a value smaller than the radius of sphere P11 (i.e., the radius represented by the radius information TPProhibitRadius), the position A, which is the input listener's position, is located within sphere P11, which is the forbidden space. That is, it can be understood that the listener has entered the forbidden space.

[0107] Therefore, in this case, for example, by means of the method described below, the listener is forcibly moved to a position on the surface of the forbidden space sphere (sphere P11), and the moved position is set as the final listener position. Then, based on the final listener position, the object's metadata is generated, or a free viewpoint space UI is generated.

[0108] Below is an example of a specific method for determining the listener's destination (the changed listener location).

[0109] First, for those set to Figure 4 The position A of the input listener position shown is used to determine the direction vector (LVx, LVy, LVz) connecting TP and position A using formula (3). In this case, UpX=Ax, UpY=Ay, and UpZ=Az hold true.

[0110] Next, determine the intersection point between the direction vector (or more specifically, the line containing the direction vector) and the surface of the forbidden space sphere (sphere P11). In this case, there are two intersection points.

[0111] These intersection points are referred to as XP1 and XP2. The coordinates representing the absolute positions of the intersection points XP1 and XP2 in the free viewpoint space (i.e., the common absolute coordinate system) are referred to as (XPlx, XPly, XPlz) and (XP2x, XP2y, XP2z), respectively.

[0112] In this case, the coordinates (XP1x, XP1y, XP1z) of the intersection point XP1 can be determined using the following formula (5), and the coordinates (XP2x, XP2y, XP2z) of the intersection point XP2 can be determined using the following formula (6). The calculations of formulas (5) and (6) use the direction vectors (LVx, LVy, LVz).

[0113] [Mathematical Expression 5]

[0114] [Mathematical Expression 6]

[0115] Note that in formulas (5) and (6), LvecPow is the same as LvecPow in formula (4), and tppr is the radius of the forbidden space sphere (sphere P11) represented by the radius information TPProhibitRadius.

[0116] When two intersection points XP1 and XP2 are determined, the focus of the intersection point that is closer to the input listener position is selected as the listener's destination position, i.e., the final listener position after movement. In other words, the intersection point with a smaller distance from the input listener position is selected as the final listener position.

[0117] More specifically, when the distance from position A, which is the location of the input listener, to the intersection point XP1 is called DistX1, and the distance from position A to the intersection point XP2 is called DistX2, the following formula (7) can be used to obtain the distances DistX1 and DistX2.

[0118] [Mathematical Expression 7]

[0119] Note that in formula (7), the coordinates of the listener's position (UpX, UpY, UpZ) are (Ax, Ay, Az). Specifically, UpX=Ax, UpY=Ay, and UpZ=Az hold true.

[0120] Based on the distances DistX1 and DistX2 obtained in this way, the following formula (8) is used to calculate the intersection point XP1 and XP2, thereby setting one of the intersection points XP1 and XP2 as the final listener position.

[0121] [Mathematical Expression 8]

[0122] Note that in formula (8), the coordinates of the final listener position are (FinalUPx, FinalUPy, FinalUPz), and the coordinates of the listener position determined by the preceding process (i.e., the previous listener position) are (PrevUPx, PrevUPy, PrevUPz).

[0123] In the calculation of formula (8), the intersection point with the smaller distance value is selected as the final listener position. Specifically, when the distance DistX2 is less than the distance DistX1, the intersection point XP2 is set as the final listener position, and when the distance DistX1 is less than the distance DistX2, the intersection point XP1 is set as the final listener position.

[0124] Furthermore, when the distances DistX1 and DistX2 are the same, the intersection point that is shorter than the listener position in the previous process (i.e., the previous listener position) is set as the final listener position.

[0125] More specifically, as in formula (7), the distance DistPX1 between the intersection point XP1 and the previous listener position and the distance DistPX2 between the intersection point XP2 and the previous listener position are determined.

[0126] Then, when the distance DistPX2 is less than the distance DistPX1, the intersection point XP2 is set as the final listener position, and when the distance DistPX1 is less than the distance DistPX2, the intersection point XP1 is set as the final listener position.

[0127] For example, in Figure 4 In the example shown, when the input listener position is position A, position B is set as the final listener position, where position B is the intersection point closer to position A among the intersection points of the straight line (i.e., the direction vector) passing through TP and position A with the surface of the forbidden space sphere (sphere P11).

[0128] exist Figure 4 In the example shown, the forbidden space sphere (sphere P11) is a spherical space centered on TP, and the position B, which is therefore set as the final listener position (i.e., the changed listener position), is the position on the surface of the forbidden space sphere closest to position A (which is the listener position before the change).

[0129] Note that the listener's position after the movement (i.e., the final listener position after the listener has entered the forbidden space) can be any position outside the forbidden space, and is not limited to a position on the boundary portion of the forbidden space (a position on the boundary), such as a position on the surface of the forbidden space sphere.

[0130] As described above, when the input listener position is within the forbidden space sphere, the listener position is moved to a position on the surface of the forbidden space sphere instead of the input listener position, and the moved position (i.e., the position represented by coordinates (FinalUPx, FinalUPy, FinalUPz)) is set as the final listener position (current listener position). Then, based on the final listener position, the object's metadata is generated, or a free viewpoint space UI is generated.

[0131] <Server Configuration Example>

[0132] Figure 5 This is a diagram illustrating an example configuration of a server employing the present technology according to an embodiment.

[0133] Server 11 consists of information processing devices and the like, and is used, for example, as an encoder to encode and deliver content.

[0134] Server 11 includes a meta encoder 21, an audio encoder 22, and a communication unit 23.

[0135] The meta encoder 21 provides metadata (i.e., metadata of the audio data of the object (hereinafter also referred to as object metadata)) and configuration information of one or more objects that constitute the content.

[0136] Prepare object metadata for each control viewpoint (CVP) for each object. The object metadata includes at least object location information indicating the object's location and object gain as the gain of the object's audio data.

[0137] Object position information represents the placement of an object within the CVP polar coordinate space, with the CVP position as the reference point (origin). The object's placement is described using coordinates (polar coordinates) in the CVP polar coordinate system. Therefore, the object's position, as represented by the object position information, is the object's position as viewed from the CVP.

[0138] In server 11, object metadata is prepared for each CVP, so that even for the same object, the value of object gain and the arrangement position of the object in free viewpoint space (common absolute coordinate system) can be different depending on the CVP.

[0139] Note that object metadata can include priority information indicating the object's priority, sound source type information indicating the object's type (sound source), and extended information indicating the object's degree of extension, etc.

[0140] The configuration information includes at least one of the aforementioned prohibited space information presence flags and radius information.

[0141] In addition, the configuration information may also include object count information indicating the number of objects constituting the content, CVP count information indicating the number of pre-prepared CVPs, and CVP information about the CVPs. The following will describe the configuration information including a flag indicating the absence of spatial information, radius information, object count information, CVP count information, and assumptions regarding CVP information.

[0142] For example, CVP information includes CVP index, CVP location information, and CVP orientation information.

[0143] The CVP index is the ID information used to uniquely identify the CVP. The CVP location information is the location information representing the absolute position of the CVP in the free viewpoint space, and the absolute position of the CVP is described by coordinates in a common absolute coordinate system.

[0144] CVP orientation information represents the orientation of the virtual listener's face at the CVP located in the free viewpoint space (common absolute coordinate system). In other words, CVP orientation information represents the orientation of the CVP, or more specifically, the orientation of the positive y-axis of the CVP polar coordinate system within the free viewpoint space.

[0145] For example, CVP orientation information consists of CVP horizontal angle (Yaw) information and CVP vertical angle (Pitch) information. CVP horizontal angle information is the horizontal angle representing the horizontal position of TP as viewed from the position of CVP in free viewpoint space, and CVP vertical angle information is the vertical angle representing the vertical position of TP as viewed from the position of CVP in free viewpoint space.

[0146] For example, in Figure 1 In the example shown, when the CVP is assumed to be at position O'', the angle (Yaw) between lines LN11 and LN12 is the horizontal angle information of the CVP, and the angle (Pitch) between the y-axis and the surface of line LN11 is the vertical angle information of the CVP.

[0147] Note that CVP orientation information can include not only CVP horizontal angle information and CVP vertical angle information, but also CVP roll information, which uses the y-axis of the CVP polar coordinate system as the rotation axis to represent the rotation angle (roll) of the CVP polar coordinate system relative to the common absolute coordinate system.

[0148] On the content playback side, when there is CVP information for multiple CVPs, the position of TP in the free viewpoint space (common absolute coordinate system) can be identified from the CVP information. In this case, when a straight line is determined for each CVP that passes through the CVP and extends along the direction represented by the CVP toward the information, and the position of the intersection of such lines is determined, the position of the intersection point is the position of TP.

[0149] Note that TP location information can be sent to the content playback side in place of CVP orientation information. TP location information is the coordinates (absolute coordinates) representing the position of the TP in a common absolute coordinate system. In this case, on the playback side, CVP orientation information can be obtained from the TP location information and CVP location information included in the configuration information.

[0150] Meta encoder 21 encodes (reuses) the object metadata provided to each CVP for each object, and provides the generated multiplexed object metadata and the provided configuration information to audio encoder 22.

[0151] Audio data for each object of the content to be played back (i.e., audio data for playing back the sound of each object) is provided to the audio encoder 22. The audio encoder 22 generates encoded audio data by encoding the provided audio data for each object.

[0152] In addition, the audio encoder 22 generates an encoded bitstream by multiplexing the encoded audio data obtained through encoding with the multiplexed object metadata and configuration information provided from the meta encoder 21, and provides it to the communication unit 23.

[0153] The communication unit 23 transmits (outputs) the encoded bit stream provided by the audio encoder 22 to the client, which is a playback device, via a network or the like.

[0154] Note that this describes an example of storing the object metadata of each CVP in an encoded bitstream, but this is not restrictive, as long as the object metadata of each CVP can be obtained on the client side.

[0155] For example, one could envision a meta-dataset that is prepared in advance as a relative arrangement pattern of each object viewed from a virtual listener (CVP), and as a collection of object metadata for each object in each arrangement pattern.

[0156] In this case, on the server 11 side, for example, an appropriate meta-dataset is selected from multiple meta-datasets (layout patterns) of each CVP, and the meta-dataset index of the selected meta-dataset is stored in the CVP information.

[0157] In this way, by using some means to share the metadata set between server 11 and client, the appropriate object metadata for each object for each CVP can be obtained from the metadata index included in the CVP information on the client side.

[0158] <Encoding Processing>

[0159] The operation of server 11 is described below. For details, see below. Figure 6 The flowchart describes the encoding process performed by server 11.

[0160] In step S11, the meta-encoder 21 generates multiplexed object metadata by performing multiplexing (encoding) of the provided object metadata for each CVP of each object. Furthermore, the meta-encoder 21 provides the multiplexed object metadata and the provided configuration information to the audio encoder 22. Note that the encoding of the configuration information can also be performed as needed.

[0161] In step S12, the audio encoder 22 generates encoded audio data by performing encoding on the audio data provided for each object.

[0162] For example, audio encoder 22 encodes the audio data according to encoding methods used in international standards such as Moving Picture Experts Group (MPEG)-I and / or MPEG-H3D audio. Note that any encoding method can be used to encode the audio data.

[0163] In step S13, the audio encoder 22 generates an encoded bitstream by multiplexing the encoded audio data obtained through encoding with the multiplexed object metadata and configuration information provided from the meta encoder 21, and provides it to the communication unit 23.

[0164] In step S14, the communication unit 23 sends the encoded bitstream provided by the audio encoder 22 to the client and completes the encoding process.

[0165] In the above manner, server 11 generates an encoded bitstream that includes configuration information storing a flag indicating the presence of prohibited spatial information and radius information, and sends it to the client.

[0166] In this way, during content playback on the client, by using configuration information to perform processing related to the prohibited space, the display related to the content can be appropriately performed. Specifically, the free viewpoint space UI can be appropriately displayed.

[0167] <Client Configuration Example>

[0168] Figure 7 This is a diagram illustrating a configuration example of a client that receives (acquires) an encoded bitstream sent (output) from server 11 and performs processing related to content playback.

[0169] Figure 7 The client 61 shown is composed of an information processing device such as a personal computer, smartphone, tablet computer, head-mounted display, or gaming device, and is used, for example, as a decoder.

[0170] The client 61 generates output audio data for playing back the content by performing decoding and other operations on the encoded bitstream received from the server 11, and outputs it to the audio output unit 62. In addition, the client 61 generates image data (video data) of the content-related free viewpoint space UI based on configuration information and the like, and provides it to the presentation unit 63.

[0171] The audio output unit 62 is composed of a speaker system including multiple speakers, and outputs (plays back) sound based, for example, on output audio data provided from the client 61. Note that the audio output unit 62 may be, for example, headphones, earphones, hearing aids, or a microphone.

[0172] The presentation unit 63 is composed of a display or the like, and displays a free-viewpoint spatial UI based on image data provided from the client 61. Note that the presentation unit 63 may include a vibration motor or the like that which presents information to the user (listener) through vibration or the like. Furthermore, the audio output unit 62 and the presentation unit 63 may be located at the client 61.

[0173] The client 61 includes a communication unit 71, an audio decoder 72, a listener information acquisition unit 73, a meta decoder 74, a rendering processing unit 75, and a presentation control unit 76.

[0174] The communication unit 71 receives (acquires) the encoded bit stream transmitted from the server 11 and provides it to the audio decoder 72.

[0175] The audio decoder 72 extracts the encoded audio data, multiplexed object metadata, and configuration information by demultiplexing the encoded bitstream provided by the communication unit 71, and then decodes the encoded audio data to obtain the audio data.

[0176] The audio decoder 72 provides the audio data of each object obtained through decoding to the rendering processing unit 75, and provides the reused object metadata and configuration information to the meta-decoder 74.

[0177] The listener information acquisition unit 73 acquires input listener position information representing the input listener position based on the operation of an input unit (not shown in the figure), such as a user (listener), and provides this input listener position information to the meta-decoder 74. This input listener position is the input listener position within the free viewpoint space. Here, in addition to the listener position, the listener information acquisition unit 73 may also acquire head tracking information (head rotation information) and eye tracking information (gaze detection information) obtained by sensors installed on an HMD used for AR, VR, etc.

[0178] Metadecoder 74 performs demultiplexing (decoding) on ​​the multiplexed object metadata provided by audio decoder 72 to obtain object metadata for each control viewpoint (CVP) of each object.

[0179] Furthermore, the meta-decoder 74 determines the final listener position based on the configuration information and object metadata provided by the audio decoder 72 and the input listener position information provided by the listener information acquisition unit 73, and generates listener reference object metadata.

[0180] The meta-decoder 74 serves as a location determination unit, which identifies whether the listener (user)'s location is within the forbidden space and appropriately changes the listener's location based on the determination result, thereby generating listener reference object metadata.

[0181] The listener reference object metadata is the object metadata of the object at the final listener position in the free viewpoint space, and the meta-decoder 74 provides the generated listener reference object metadata to the rendering processing unit 75.

[0182] The listener reference object metadata includes listener reference object position information representing the relative position of the object as viewed from the final listener position, and listener reference object gain for each object at the final listener position. Specifically, the listener reference object metadata is the metadata of the object when the listener is at the position determined by the meta-decoder 74, and is generated for each object.

[0183] Note that the listener reference object metadata may include information about the object's priority, sound source type, and extended information.

[0184] Furthermore, when generating listener reference object metadata, the meta-decoder 74 also generates listener position information representing the final listener position in the free viewpoint space, a TP warning flag indicating whether the specified (input) input listener position is located in the prohibited space, TP position information, etc., and provides (outputs) them to the presentation control unit 76. For example, the listener position represented by the listener position information is described by coordinates (absolute coordinates) in a common absolute coordinate system.

[0185] Note that this description refers to the TP warning flag as a marker indicating whether the listener's position is within the prohibited space (warning information), and the TP warning flag is always output to the presentation control unit 76. However, it is not limited to this, and the corresponding warning information can be output to the presentation control unit 76 only when the listener's position is within the prohibited space. Note that in addition to presenting characters (messages), colors, or images based on the warning information, notifications based on the warning information can also be provided using, for example, sound or vibration.

[0186] The rendering processing unit 75 performs rendering processing based on the audio data of each object provided by the audio decoder 72 and the listener reference object metadata of each object provided by the meta decoder 74, generating output audio data. The rendering processing unit 75 provides (outputs) the generated output audio data to the audio output unit 62 to reproduce the sound of the content.

[0187] The rendering processing unit 75 performs rendering processing in the polar coordinate system defined in the MPEG-H 3D audio, such as using vector-based amplitude translation (VBAP), to generate output audio data. This output audio data is audio data used for playback of free-viewpoint content including the sounds of all objects.

[0188] Note that the rendering processing unit 75 can also perform convolution processing such as head-related transfer function (HRTF), binaural impulse response (BRIR), intraaural impulse response (RIR), interaural time difference (ITD), interaural intensity difference (IID), and processing using high-order high-fidelity stereo (HOA) as rendering processing.

[0189] The presentation control unit 76 is implemented, for example, by a high-level system that implements the meta-decoder 74, and controls the display of various images (videos) on the presentation unit 63. Furthermore, for example, the presentation control unit 76 can control the presentation of vibration-based information at the presentation unit 63 or the presentation of sound-based information at the audio output unit 62.

[0190] For example, the presentation control unit 76 generates image data (video data) for the free viewpoint space UI based on the listener location information, TP warning sign, TP location information, etc. provided by the meta-decoder 74, and provides it to the presentation unit 63.

[0191] <Instructions for Output Audio Data Generation and Processing>

[0192] The following describes the operations of client 61. Specifically, please refer to the following... Figure 8 The flowchart describes the output audio data generation process performed by client 61.

[0193] In step S41, the communication unit 71 receives (acquires) the encoded bit stream sent from the server 11 and provides it to the audio decoder 72.

[0194] In step S42, the audio decoder 72 performs decoding on the encoded bitstream provided from the communication unit 71.

[0195] More specifically, the audio decoder 72 extracts the encoded audio data, multiplexed object metadata, and configuration information by demultiplexing the encoded bitstream, and obtains the audio data by decoding the encoded audio data.

[0196] The audio decoder 72 provides the audio data of each object obtained through decoding to the rendering processing unit 75, and provides the reused object metadata and configuration information to the meta-decoder 74.

[0197] In step S43, in response to an input operation by a user (listener), the listener information acquisition unit 73 acquires input listener position information representing the input listener position input by the listener, and provides it to the meta-decoder 74. For example, the input listener position information is the coordinates (absolute coordinates) representing the absolute input listener position in the free viewpoint space (i.e., the common absolute coordinate system).

[0198] In step S44, the meta-decoder 74 generates listener reference object metadata based on the configuration information provided from the audio decoder 72, the multiplexed object metadata, and the input listener location information provided from the listener information acquisition unit 73.

[0199] For example, the meta-decoder 74 determines the final listener position based on the prohibited space information presence flag, radius information, CVP information, and input listener position information included in the configuration information, and generates listener position information representing that listener position.

[0200] Note that, as detailed later, when the input listener position is outside the forbidden space (note, including the boundary of the forbidden space), the input listener position is directly set as the final listener position. That is, the listener position is not changed.

[0201] On the other hand, when the input listener position is within the forbidden space, the final listener position is set to a position outside the forbidden space that is different from the input listener position. Specifically, the listener position is changed to a position outside the forbidden space that is different from the input listener position.

[0202] In addition, the meta-decoder 74 generates a TP warning flag and TP location information during the process of generating listener location information, and provides the listener location information, TP warning flag and TP location information to the presentation control unit 76.

[0203] For example, when a listener enters a prohibited space, that is, when the input listener position is within a prohibited space (when a position within a prohibited space is designated as the input listener position), the TP warning flag is set to "true". A "true" value for the TP warning flag indicates that the listener has entered a prohibited space, or more specifically, that the listener is attempting to enter a prohibited space.

[0204] On the other hand, for example, when the listener is not in the prohibited space, that is, when the input listener position is outside the prohibited space (when the position outside the prohibited space is designated as the input listener position), the TP warning flag is set to "false". A value of "false" for the TP warning flag indicates that the listener is outside the prohibited space.

[0205] As described above, providing the TP warning flag to the presentation control unit 76 can also be regarded as a notification that the listener's position has moved from the input listener position (i.e., the listener's position has changed).

[0206] The meta-decoder 74 generates listener baseline object metadata for each object based on listener location information, CVP information included in the configuration information, and object metadata for each CVP.

[0207] The listener reference object metadata includes listener reference object location information and listener reference object gain. For example, the location of the object, as indicated by the listener reference object location information, is its location in the listener polar coordinate space and is described by coordinates (polar coordinates) of the listener polar coordinate system.

[0208] Note that although the generation of the listener reference object metadata will be described in detail later, the listener reference object metadata is generated, for example, in step S44 by performing interpolation processing such as vector synthesis.

[0209] At this point, using the object metadata of all or several CVPs around the listener's location, interpolation processing is performed to correspond to the positional relationship between the listener's location and CVPs identified based on CVP information and listener location information, thereby generating listener baseline object metadata.

[0210] The meta-decoder 74 provides the generated listener baseline object metadata to the rendering processing unit 75.

[0211] In step S45, the rendering processing unit 75 performs rendering processing based on the audio data of each object provided by the audio decoder 72 and the listener reference object metadata provided by the meta decoder 74.

[0212] For example, the rendering processing unit 75 performs gain correction on the audio data of each object based on the listener reference object gain of each object.

[0213] Then, the rendering processing unit 75 performs rendering processing such as VBAP based on the audio data of each object after gain correction and the listener reference object position information, thereby generating output audio data.

[0214] The rendering processing unit 75 outputs the generated output sound data to the sound output unit 62, enabling it to play back the sound of the content. In this way, it is possible to play back content in which any position outside the forbidden space within the free viewpoint space is set as the listener (listening position), that is, to play back free viewpoint content (6DoF content) from multiple viewpoints.

[0215] Furthermore, the presentation control unit 76 generates image data for the free viewpoint space UI based on the listener location information, TP warning flag, TP location information, etc., provided by the meta-decoder 74, and provides it to the presentation unit 63 so that it can display the free viewpoint space UI. The free viewpoint space UI can be generated using CVP information, listener reference object metadata, radius information, etc., as needed.

[0216] The presentation control unit 76 calculates the orientation of the listener's face in the free viewpoint space based on the listener's position information and TP position information, and uses the calculation result to generate the free viewpoint space UI. In other words, the presentation control unit 76 generates the free viewpoint space UI (i.e., the image related to the free viewpoint space) based on the listener's position information, the listener's facial orientation in the free viewpoint space, and the TP warning sign, and causes the presentation unit 63 to display the free viewpoint space UI. Note that, as with the above formulas (1) and (2), the listener's facial orientation can be obtained through calculation.

[0217] For example, all or part of the image in the free view space can be displayed as a free view space UI. In this case, TP, forbidden space, objects, etc., can be displayed on the free view space UI.

[0218] In a free-viewpoint spatial UI, a TP (Prohibited Point) can be displayed in a prominent color, such as red, or it can be displayed in a flashing manner to keep listeners away from the TP. Similarly, the areas (spaces) that constitute a forbidden space can also be displayed in a specific color or in a flashing manner.

[0219] Furthermore, for example, when the TP warning flag is "true", a message indicating that entry into the prohibited space is prohibited can be displayed on the free viewpoint space UI. Note that the notification (notification control) indicating that entry into the prohibited space is prohibited provided by the presentation control unit 76 is not limited to the display of a message; it can also be provided using a flashing display of the TP or the prohibited space, a warning sound, vibration, or a combination thereof. When the TP warning flag is "true", the aforementioned changes in the color of the TP, the flashing display, and the warning notification can be automatically set by the system or arbitrarily set by the user.

[0220] In step S46, client 61 determines whether to terminate the process.

[0221] For example, in step S46, when the encoded bitstream of all frames of the content has been received and the output audio data has been generated, or when the listener (user) has indicated the termination of content playback, the termination process is determined.

[0222] In step S46, when it is determined that the process has not yet terminated, the process returns to step S41 and the above process is repeated.

[0223] On the other hand, when the termination process is determined in step S46, the client 61 terminates the operation of each part and completes the output audio data generation process.

[0224] In the above manner, client 61 determines the final listener position based on configuration information and / or input listener position information, and generates a free viewpoint space UI and / or outputs audio data corresponding to the determination result.

[0225] In this way, a free-viewpoint spatial UI can be appropriately displayed. Specifically, content-related displays can be appropriately executed. Furthermore, based not only on the physical relationship between the listener and the object surface, but also on the listener's position, content playback with the music quality expected by the content creator can be achieved, thereby fully conveying the appeal of the content to the listener.

[0226] <Description of Metadata Generation and Processing>

[0227] In addition, in implementing the reference Figure 8 The output audio data generation and processing described are performed simultaneously in Figure 9 The metadata generation process shown is a process corresponding to step S44 in client 61.

[0228] In the following text, see references Figure 9 The flowchart describes the metadata generation process of client 61.

[0229] In step S81, the meta-decoder 74 sets the TP warning flag to "false" and maintains the TP warning flag. Specifically, the meta-decoder 74 temporarily sets the value of the TP warning flag to "false".

[0230] In step S82, the meta-decoder 74 determines whether prohibited space information exists.

[0231] For example, when the configuration information provided from the audio decoder 72 includes the above reference Figure 3 The meta-decoder 74 determines the presence of prohibited spatial information when the prohibited spatial information presence flag "TPProhibitPresentFlag" is described and its value is "1". When the prohibited spatial information presence flag is "1", the configuration information includes the radius information "TPProhibitRadius" which is the prohibited spatial information.

[0232] In step S82, if it is determined that there is no prohibited space information, that is, no prohibited space (prohibited space sphere) is set on the server 11 side, the next step is to proceed to step S83.

[0233] In step S83, the meta-decoder 74 generates listener reference object metadata at the current listener position (i.e., the input listener position) and provides it to the rendering processing unit 75.

[0234] More specifically, the meta-decoder 74 directly sets the input listener position (i.e., the current listener position) represented by the input listener position information provided by the listener information acquisition unit 73 as the final listener position, and generates listener position information representing that listener position.

[0235] In addition, based on the listener location information, the CVP information included in the configuration information, and the object metadata of each CVP, the meta-decoder 74 generates listener reference object metadata for each object located in the free viewpoint space through interpolation and other processes.

[0236] Furthermore, based on the CVP information of each CVP included in the configuration information, especially the CVP location information and CVP orientation information included in the CVP information, the meta-decoder 74 identifies the location of the TP and also generates TP location information representing the location of the TP.

[0237] The meta-decoder 74 provides the listener reference object metadata to the rendering processing unit 75, and provides the listener position information, the TP warning flag with a value of "false", and the TP position information to the rendering control unit 76. Then, the processing proceeds to step S91. Note that when it is determined in step S82 that there is no prohibited space information, prohibited space information (radius information) can be set on the client 61 side, and the processing of steps S84 to S90, which will be described later, can be performed.

[0238] In addition, when it is determined in step S82 that there is prohibited space information, a prohibited space is set on the server 11 side, so the process proceeds to step S84.

[0239] In step S84, based on the input listener location information and configuration information, the meta-decoder 74 calculates the distance UT between the input listener location (i.e., the input listener location) and TP.

[0240] More specifically, the meta-decoder 74 calculates the TP location information based on the CVP information of each CVP included in the configuration information, and performs the calculation in the same way as the above formulas (3) and (4) based on the input listener location information and the TP location information, thereby calculating the distance UT. In this case, the distance Dist obtained using formulas (3) and (4) corresponds to the distance UT.

[0241] In step S85, the meta-decoder 74 compares the distance UT calculated in step S84 with the radius of the prohibited space sphere, which is represented by the radius information and is included in the prohibited space information in the configuration information, to determine whether the distance UT is less than the radius of the prohibited space sphere.

[0242] This determination process can be viewed as identifying (determining) whether the listener's position, as indicated by the input listener's position information, is located within the prohibited space, based on the input listener's position information and the prohibited space information (radius information) and CVP information included in the configuration information.

[0243] In step S85, when it is determined that the distance UT is not less than the radius of the forbidden space sphere, the input listener position is outside the forbidden space sphere, and there is no need to change the listener position (forced movement of the listener). Therefore, the process then proceeds to step S83.

[0244] In this case, the meta-decoder 74 does not change the listener's position, but performs the above-mentioned processing in step S83, and the processing then proceeds to step S91.

[0245] On the other hand, when it is determined in step S85 that the distance UT is less than the radius of the forbidden space sphere, the input listener position is located inside the forbidden space sphere, and the listener position needs to be changed (forced movement of the listener). Therefore, the process then proceeds to step S86.

[0246] In step S86, the meta-decoder 74 sets the TP warning flag to "true" and maintains the TP warning flag. Specifically, the meta-decoder 74 changes the value of the TP warning flag it maintains from "false" to "true".

[0247] In step S87, based on the input listener position information and TP position information, the meta-decoder 74 calculates the direction vector connecting the current listener position (i.e., the input listener position) and TP. For example, the meta-decoder 74 calculates the direction vector by performing the same calculation as in formula (3) above. Note that this direction vector is obtained from the distance UT calculation process performed in step S84, therefore, the direction vector obtained in step S84 can be retained and used as the result of the processing in step S87.

[0248] In step S88, based on the radius information as forbidden space information, the TP position information included in the configuration information, and the direction vector obtained in step S87, the meta-decoder 74 calculates the intersection point of the direction vector and the surface of the forbidden space sphere.

[0249] For example, the meta-decoder 74 calculates the two intersections by performing the same calculations as in Equations (5) and (6) above.

[0250] In step S89, the meta-decoder 74 selects the intersection point that is closer to the input listener's position from the two intersections calculated in step S88.

[0251] For example, the meta-decoder 74 identifies the intersection point closer to the input listener position by performing calculations based on the input listener position information and the positions of the two intersection points in the same way as in formulas (7) and (8) above. At this time, for calculation purposes, the meta-decoder 74 also uses the listener position information of the listener position prior to the input listener position, i.e., the final listener position determined in the final executed step S83 or step S89, as needed.

[0252] Note that, depending on the shape of the forbidden space, there may be three or more intersection points between the direction vector and the forbidden space surface. In this case, the position of the intersection point closest to the input listener position is selected from multiple intersection points and set as the changed listener position (final listener position). Furthermore, when performing step S89, it may be determined in a previous timing (such as the previous frame) in step S85 that the distance UT is less than the radius of the forbidden space sphere, and step S89 has already been performed. In this case, in the current step S89, the meta-decoder 74 can set the listener position immediately preceding the input listener position (i.e., the final listener position determined in the last executed step S89) as the currently changed final listener position. In this case, when the user continuously inputs positions within the forbidden space sphere as input listener positions, the same positions on the surface of the forbidden space sphere are continuously set as final listener positions.

[0253] In step S90, the meta-decoder 74 uses the location of the intersection point selected in step S89 as the location of the listener's destination (i.e., the changed final listener location), generates listener reference object metadata, and provides it to the rendering processing unit 75.

[0254] More specifically, when the selected intersection location is taken as the final listener location, the meta-decoder 74 generates listener location information representing that listener location. Specifically, the listener location is changed from the input listener location to the selected intersection location, and listener location information representing the changed listener location is generated.

[0255] Furthermore, as in step S83, for each object located in the free viewpoint space, the meta-decoder 74 generates listener reference object metadata based on listener location information, CVP information, and object metadata of each CVP through interpolation processing and the like.

[0256] The meta-decoder 74 provides the listener reference object metadata to the rendering processing unit 75, and provides the listener position information, the TP warning flag with a value of "true", and the TP position information to the rendering control unit 76. Then, the processing proceeds to step S91. Note that the TP position information calculated in step S84 can also be used.

[0257] When the processing of step S90 is performed, or when the processing of step S83 is performed, the processing of step S91 is then performed.

[0258] In step S91, the metadata decoder 74 determines whether there is any data to be processed. For example, in step S91, when all object metadata of the content has been processed, it is determined that there is no data to be processed.

[0259] In step S91, when it is determined that there is data to be processed, the processing returns to step S81, and the above processing is repeated.

[0260] On the other hand, when it is determined in step S91 that there is no data to be processed, the metadata generation process is completed.

[0261] Using the above method, client 61 determines the final listener position based on whether the input listener position is within the prohibited space, and generates listener reference object metadata based on the determination result. In this way, situations where the listener's orientation cannot be calculated can be prevented, and the free viewpoint space UI display can always be performed appropriately.

[0262] Interpolation processing

[0263] Now, the description is in Figure 9 A specific example of the interpolation processing performed when generating listener reference object metadata in steps S83 and / or S90 is provided. Specifically, the case of performing polar coordinate vector synthesis is described below.

[0264] For example, suppose the position LP21 in the free viewpoint space is as follows: Figure 10 The listener's position is shown on the left side of the diagram. Additionally, assume that CVP1 to CVP5 are set in free viewpoint space, and that CVP1 to CVP5 are used to determine the listener's reference object position information. Note that, for simplicity, Figure 10 An example of placing a CVP on a two-dimensional plane is shown.

[0265] In this example, TP is located at the center, and CVP1 to CVP5 are located around TP. Additionally, the positive y-axis of the CVP polar coordinate system for each CVP is set from the CVP toward TP.

[0266] Furthermore, the positions of the same object viewed from CVP1 to CVP5 (hereinafter also referred to as the object of interest) are positions OBP1 to OBP5. That is, the polar coordinates in the CVP polar coordinate system representing the positions of OBP1 to OBP5 are object position information about CVP1 to CVP5.

[0267] In this case, assume that for each CVP, an axis rotation is performed so that the y-axis of the CVP polar coordinate system becomes the vertical direction, i.e., the upward direction in the figure.

[0268] Furthermore, when the objects of interest are rearranged such that the origin O'' of the polar coordinate system of each rotated CVP becomes the same origin of the single CVP polar coordinate system, the relationship between the position surfaces of the objects of interest OBP1 to OBP5 at the corresponding CVPs viewed from the origin of the CVP polar coordinate system will be as shown on the right side of the attached figure. Specifically, the right side of the attached figure shows the object positions when each CVP is set as the origin and the midplane is set as the positive y-axis direction.

[0269] In the CVP polar coordinate system of each CVP, the positional relationship shown on the right side of the attached figure can be easily determined because there is a constraint that the direction of the midplane (y-axis) is towards the direction of TP.

[0270] Furthermore, on the right side of the figure, the vectors that start from the position of CVP (i.e., the origin) and end at the positions of the objects of interest OBP1 to OBP5 are represented as vectors V41 to V45.

[0271] In this case, such as Figure 11 As shown, by assigning weights to each vector (CVP) and combining vectors V41 into vector V45, it is possible to determine vector V51 representing the position of the object of attention as seen from the listener's position LP21. Note that in Figure 11 In this example, for the sake of simplicity, the weight of each CVP is set to 1.

[0272] Furthermore, when the weight of each CVP during vector synthesis is referred to as the contribution rate, the contribution rate can be determined, for example, based on the ratio of the distances from the listener's position to the CVP in the free viewpoint space (common absolute coordinate system).

[0273] More specifically, for example, suppose as Figure 12 As shown, the listener position represented by the listener position information is position F, and the positions of the three CVPs used for interpolation are positions A to C. It is also assumed that the absolute coordinates of position F in the common absolute coordinate system are (xf, yf, zf), and the absolute coordinates of positions A, B, and C in the common absolute coordinate system are (xa, ya, za), (xb, yb, zb), and (xc, yc, zc), respectively. Note that the absolute coordinates representing the positions of the CVPs in the common absolute coordinate system can be determined based on the CVP position information included in the configuration information.

[0274] The meta-decoder 74 determines the ratio (distance ratio) between the distance AF from position F to position A, the distance BF from position F to position B, and the distance CF from position F to position C, and sets the reciprocal of this distance ratio as the ratio (dependency ratio) of the contribution rate of CVP at each position.

[0275] Specifically, when AF:BF:CF=a:b:c and the CVP located between positions A and C has dependencies on the listener's position (listener reference object position information) of dp(AF), dp(BF), and dp(CF), the meta-decoder 74 calculates the following formula (9).

[0276] [Mathematical Expression 9]

[0277] Note that a, b, and c in formula (9) are represented as in formula (10) below.

[0278] [Mathematical Expression 10]

[0279] Furthermore, by using the following formula (11), the meta-decoder 74 normalizes the dependencies dp(AF), dp(BF), and dp(CF) expressed in formula (9) and determines the normalized dependencies ndp(AF), ndp(BF), and ndp(CF) as the final contribution rates. Note that a, b, and c in formula (11) are also determined using formula (10).

[0280] [Mathematical Expression 11]

[0281] Regarding the contribution rates ndp(AF) to ndp(CF) obtained in this way, the shorter the distance from the listener's position to the CVP, the closer the contribution rate of that CVP is to 1. Note that the contribution rate of each CVP is not limited to the example above and can be determined by any other method.

[0282] Based on the listener location information and CVP location information, the meta-decoder 74 calculates the contribution rate of each CVP by determining the ratio of the distance from the listener location to the CVP.

[0283] Based on the above, the following describes a specific example of calculating listener reference object metadata (or more specifically, listener reference object location information and listener reference object gain) through interpolation processing.

[0284] First, the meta-decoder 74 selects the CVPs for interpolation processing based on the listener location information and CVP location information included in the configuration information. Note that the CVPs used for interpolation processing can be only a portion of the CVPs located around the listener location, or all CVPs can be used for interpolation processing.

[0285] The meta-decoder 74 calculates a 3D object position vector for each selected CVP, starting from the CVP and ending at the object's position as viewed from the CVP, based on object position information.

[0286] For example, when the object 3D position vector of the j-th object viewed from the i-th CVPi is called (Obj_vector_x[i][j], Obj_vector_y[i][j], Obj_vector_z[i][j]), the object 3D position vector can be obtained by using the following formula (12).

[0287] Note that in this case, the polar coordinates represented by the object position information of the j-th object viewed from the i-th CVPi are called (Azi[i][j], Ele[i][j], rad[i][j]).

[0288] [Mathematical Expression 12]

[0289] Next, the meta-decoder 74 determines the contribution rate ndp(i) of each CVPi based on the CVP location information for each CVPi included in the listener location information and configuration information, using the same calculations as in Equations (9) to (11) above. This contribution rate ndp(i) is used as a weighting coefficient for interpolation processing. The contribution rate ndp(i) is a weighting coefficient determined based on the ratio of the distance from the listener location to the CVPi, or more specifically, the inverse ratio of the distance.

[0290] Furthermore, the meta-decoder 74 performs the following calculation based on the object's three-dimensional position vector obtained by using formula (12), the contribution rate ndp(i) of each CVPi, and the object gain Obj_gain[i][j] of the j-th object viewed from the CVPi. In this way, the listener reference object position information (Intp_x(j), Intp_y(j), Intp_z(j)) and the listener reference object gain Intp_gain(j) of the j-th object are obtained.

[0291] [Mathematical Expression 13]

[0292] In formula (13), the weighted vector sum is determined. Specifically, the sum of the object three-dimensional position vector of each CVPi multiplied by the contribution rate ndp(i) is determined as the listener's reference object position information. In addition, the sum of the object gain of each CVPi multiplied by the contribution rate ndp(i) is obtained as the listener's reference object gain.

[0293] Note that the listener reference object position information determined using formula (13) is the absolute coordinate in the absolute coordinate system, where the listener position is the origin and the direction from the listener position toward TP is the positive y-axis direction, i.e. the direction of the mid-plane.

[0294] However, when rendering in polar coordinates is performed in the rendering processing unit 75, the position information of the listener reference object in polar coordinates is required.

[0295] In view of this, by calculating the following formula (14), the meta-decoder 74 converts the listener reference object position information (Intp_x(j), Intp_y(j), Intp_z(j)) obtained using formula (13) in absolute coordinates into listener reference object position information (Intp_azi(j), Intp_ele(j), Intp_rad(j)) in polar coordinates.

[0296] [Mathematical Expression 14]

[0297] The meta-decoder 74 sets the listener reference object location information (Intp_azi(j), Intp_ele(j), Intp_rad(j)) obtained in this way as the final listener reference object location information.

[0298] The polar coordinates of the listener reference object position information obtained using formula (14) are polar coordinates of the polar coordinate system, where the listener position is the origin and the direction from the listener position toward TP is the positive y-axis direction (i.e., the direction of the mid-plane).

[0299] Through the above processing, listener reference object metadata, including listener reference object location information and listener reference object gain, can be obtained.

[0300] In this technique, by setting a single TP to be common to all CVPs, the desired listener reference object location information and listener reference object gain can be obtained through simple calculations.

[0301] Note that when the listener reference object metadata includes priority information, extended information, etc., priority information, extended information, etc., can also be generated through interpolation or other methods. Furthermore, for example, among the priority information and / or extended information included in multiple object metadata used to generate the listener reference object metadata, one of the priority information and / or extended information with the largest or smallest value, or one of the priority information and / or extended information from the CVP closest to the listener's location, can be used as the priority information and / or extended information regarding the listener reference object metadata. Additionally, the median or average of the priority information and / or extended information can be used as the priority information and / or extended information regarding the listener reference object metadata.

[0302] <Second Implementation Method>

[0303] <Description of Metadata Generation and Processing>

[0304] As mentioned above, the forbidden space only needs to be a space that includes at least the target point TP, and the target point (TP) itself can be a forbidden space.

[0305] When the TP itself is set to a forbidden space, the listener position is changed (forced to move) when the listener position coincides with the TP (i.e., when the input listener position becomes the TP).

[0306] In this scenario, the previous listener position can be set as the final listener position. Furthermore, when the previous listener position is set as the final listener position, the listener baseline object metadata obtained for that previous listener position is used directly.

[0307] If both server 11 and client 61 know that TP is set to forbidden space, the configuration information does not need to include the forbidden space information presence flag and / or radius information.

[0308] When TP itself is set to a forbidden space, during the execution of reference... Figure 8 The description describes the output audio data generation and processing, while executing... Figure 13 The metadata generation process shown is a process corresponding to step S44 in client 61.

[0309] See below. Figure 13 The flowchart describes the metadata generation process performed by client 61.

[0310] In step S141, the meta-decoder 74 sets the variable used to store past listener positions (i.e., the variable PrevPos representing the previous listener position) to its initial value.

[0311] For example, the meta-decoder 74 sets the common absolute coordinate system coordinates representing the position of a given location in free viewpoint space, excluding TP, as the initial value of the variable PrevPos. The initial value of the variable PrevPos can be a pre-set given value or a randomly determined value, as long as it is a value other than the value of the TP position information.

[0312] In step S142, the meta-decoder 74 sets the TP warning flag to "false" and maintains the TP warning flag. Specifically, the meta-decoder 74 temporarily sets the value of the TP warning flag to "false".

[0313] In step S143, the meta-decoder 74 determines whether the input listener position is the same as TP.

[0314] Specifically, the meta-decoder 74 identifies the position of the TP based on the CVP information of each CVP included in the configuration information provided by the audio decoder 72, especially the CVP position information and CVP orientation information included in the CVP information, and generates TP position information representing the position of the TP.

[0315] The meta-decoder 74 determines whether the input listener position is the same as the TP by comparing the TP position information represented by the TP position with the input listener position information provided by the listener information acquisition unit 73.

[0316] In step S143, when it is determined that the input listener position is the same as TP, it is necessary to change the listener position (forced movement of the listener), so the process then proceeds to step S144.

[0317] In step S144, the meta-decoder 74 sets the listener position to PrevPos. Specifically, the meta-decoder 74 sets the previous listener position (not the input listener position) represented by the variable PrevPos to the final listener position (the changed listener position) and generates listener position information representing that listener position.

[0318] Therefore, the listener who has attempted to move to the TP position is forced to move back to the previous position. In other words, movement to the input listener position is not accepted through the listener's input operations, and the listener remains in the previous position without moving.

[0319] In step S145, the meta-decoder 74 sets the TP warning flag to "true". Specifically, the meta-decoder 74 changes the value of the TP warning flag from "false" to "true".

[0320] When the process of step S145 is executed, the process then proceeds to step S146.

[0321] Furthermore, when it is determined in step S143 that the input listener position is not the same as the TP position, there is no need to change the listener position. Therefore, the processing in steps S144 and S145 is not performed, and the processing continues to step S146.

[0322] In this case, the meta-decoder 74 sets the input listener position to the final listener position and generates listener position information representing the input listener position.

[0323] When step S145 has been executed, or when it is determined in step S143 that the input listener position is not the same as the TP position, step S146 is executed.

[0324] In step S146, the meta-decoder 74 generates listener baseline object metadata based on the listener location information, configuration information, and multiplexed object metadata (object metadata) provided from the audio decoder 72.

[0325] For example, in step S146, by performing as in Figure 9 The same process as step S83 generates listener reference object metadata, and provides this listener reference object metadata to the rendering processing unit 75. Furthermore, the meta-decoder 74 provides the TP warning flag, listener position information, and TP position information it retains to the rendering control unit 76.

[0326] In this case, for example, when it is determined in step S143 that the input listener position is not the same as the TP position, the input listener position is set as the final listener position, and listener reference object metadata is generated. Additionally, a TP warning flag with a value of "false" is provided to the presentation control unit 76.

[0327] On the other hand, when performing the processing of step S145, the position different from the input listener position is set as the final listener position, listener reference object metadata is generated, and a TP warning flag with a value of "true" is provided to the presentation control unit 76.

[0328] Note that during the processing of step S145, since the previous listener position was directly set as the final listener position (i.e., the current listener position) in step S144, the listener did not move from the previous position. Therefore, more specifically, the listener reference object metadata already generated for the previous listener position is provided again to the rendering processing unit 75.

[0329] In step S147, the meta-decoder 74 updates the variable PrevPos that it holds.

[0330] More specifically, the meta-decoder 74 updates the variable PrevPos, making the value of the variable PrevPos represent the final listener position, that is, the value of the listener position information.

[0331] Therefore, for example, when it is determined in step S143 that the input listener position is not the same as TP, the value of the variable PrevPos is set to the value representing the input listener position. In this case, when the next process is executed, the input listener position entered in the current process is regarded as the previous listener position.

[0332] On the other hand, for example, when it is determined in step S143 that the input listener position is the same as TP, the value of the variable PrevPos is essentially not updated. Specifically, it is assumed that the listener has not moved from the previous listener position.

[0333] In step S148, the meta-decoder 74 determines whether there is data to be processed. The determination in step S148 is as follows: Figure 9 The steps are the same as in step S91.

[0334] When it is determined in step S148 that there is data to be processed, the process returns to step S142 and the above process is repeated.

[0335] On the other hand, when it is determined in step S148 that there is no data to be processed, the metadata generation process is completed.

[0336] In this way, client 61 determines the final listener position based on whether the input listener position coincides with TP, which serves as a forbidden space, and generates listener reference object metadata based on the determination result. This method prevents situations where the listener's orientation cannot be calculated and ensures that the free viewpoint space UI display is always performed appropriately.

[0337] Note that the metadata generation process described in the first and second embodiments above can be performed by the metadata encoder 21 on the server 11 side, rather than by the client 61 side. For example, when the server 11 obtains the input listener location information from the client 61 and generates listener reference object metadata, the metadata generation process can be performed by the server 11 as described above. Figure 9 and Figure 13 Metadata generation and processing are then performed. The listener reference object metadata, TP warning flags, and final listener location information are then stored in the encoded bitstream.

[0338] As described above, according to this technology, during the movement of the listener within a three-dimensional free viewpoint space where the direction of the TP is set to the line of sight (i.e., the orientation of the listener's face), even when the input listener position is the same as the TP, the inability to calculate the orientation of the listener's face can be prevented. That is, the situation where the horizontal and / or vertical angles indicating the orientation of the listener's face become infinitely large can be avoided. In this way, a free viewpoint space UI display can always be performed appropriately.

[0339] <Computer Configuration Example>

[0340] The aforementioned series of processes can be performed by hardware or software. In the case where the series of processes are performed by software, the program constituting the software is installed in the computer. Here, the computer includes computers embedded in dedicated hardware, as well as general-purpose personal computers capable of performing various functions by, for example, installing various programs.

[0341] Figure 14 This is a block diagram illustrating an example configuration of the hardware of a computer that uses a program to perform the above series of processes.

[0342] In a computer, the central processing unit (CPU) 501, read-only memory (ROM) 502, and random access memory (RAM) 503 are interconnected via a bus 504.

[0343] The input / output interface 505 is further connected to the bus 504. The input unit 506, output unit 507, recording unit 508, communication unit 509 and driver 510 are connected to the input / output interface 505.

[0344] The input unit 506 comprises a keyboard, mouse, microphone, imaging element, etc. The output unit 507 comprises a display, speakers, etc. The recording unit 508 comprises a hard disk, non-volatile memory, etc. The communication unit 509 comprises a network interface, etc. The driver 510 drives a removable recording medium 511 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0345] In a computer with the above configuration, for example, CPU 501 loads the program recorded in the recording unit 508 into RAM 503 and executes it via input / output interface 505 and bus 504, thereby performing the series of processes described above.

[0346] The program executed by the computer (CPU 501) can be provided via a removable recording medium 511, such as an encapsulated medium. The program can also be provided via wired or wireless transmission media, such as a local area network, the Internet, or digital satellite broadcasting.

[0347] In a computer, by installing the removable recording medium 511 onto the drive 510, a program can be installed into the recording unit 508 via the input / output interface 505. Alternatively, the program can be received by the communication unit 509 and installed into the recording unit 508 via a wired or wireless transmission medium. Furthermore, the program can be pre-installed in the ROM 502 and / or the recording unit 508.

[0348] Note that a program executed by a computer may be a program that performs processing in a time sequence according to the order described in this specification, or it may be a program that performs processing in parallel or at necessary times (such as when invoked).

[0349] Furthermore, the implementation of this technology is not limited to the above-described implementation, and various modifications can be made without departing from the spirit of this technology.

[0350] For example, this technology can be configured using cloud computing, where a single function is distributed and processed collaboratively by multiple devices over a network.

[0351] Furthermore, the steps described in the above reference flowchart can be performed not only by a single device but also by being distributed across multiple devices.

[0352] Furthermore, in cases where a single step includes multiple processes, the multiple processes included in that single step can be executed not only by a single device but also by being assigned to multiple devices.

[0353] In addition, this technology may have the following configurations.

[0354] (1) An information processing device, comprising: The location determination unit is configured to: based on location information representing the user's location in space, and prohibited space information for specifying a prohibited space including a predetermined target point and prohibiting the user from entering, identify whether the user's location, as indicated by the location information, is within a prohibited space, and When a user's location is within the prohibited space, the user's location will be changed to a location outside the prohibited space that is different from the location indicated by the location information.

[0355] (2) According to the information processing device of (1), the position determination unit sets the position on the surface of the prohibited space or outside the prohibited space that is closest to the position of the user represented by the position information as the changed position of the user.

[0356] (3) According to the information processing device of (1), the position determination unit sets the position of the intersection of the surface of the prohibited space and the vector connecting the user's position and the target point represented by the position information as the user's changed position.

[0357] (4) According to the information processing device of (3), the location determination unit sets the location of the intersection point among multiple intersection points that is closest to the user's location represented by the location information as the user's changed location.

[0358] (5) An information processing device according to any one of (1) to (4), wherein the forbidden space is a spherical space.

[0359] (6) An information processing apparatus according to any one of (1) to (5), wherein the prohibited space is a space that includes a given object located within the space.

[0360] (7) The information processing device according to (1), wherein the forbidden space is the target point.

[0361] (8) The information processing device according to (1), wherein when the user’s position is within the prohibited space, the position determination unit sets the user’s previous position as the user’s changed position.

[0362] (9) An information processing apparatus according to any one of (1) to (8), wherein the location determination unit outputs a warning message indicating whether the user's location is within a prohibited space.

[0363] (10) The information processing apparatus according to (9) further includes a presentation control unit configured to display a space-related image based on location information indicating the user’s final location, the user’s orientation in the space, and warning information.

[0364] (11) The information processing apparatus according to (10), wherein at least one of the prohibited space, the target point and the object is displayed in the image.

[0365] (12) The information processing apparatus according to (10) or (11), wherein a notification indicating that entry into the prohibited space is prohibited is displayed in the image.

[0366] (13) An information processing apparatus according to any one of (1) to (12), wherein when the user’s location is within a prohibited space, the location determination unit generates metadata of an object located in the space based on location information representing the user’s changed location.

[0367] (14) The information processing device according to (13), wherein, Multiple locations in the space are set as control viewpoints, and The location determination unit generates metadata at the user's changed location based on the metadata of each of the multiple control viewpoints and the location information representing the user's changed location.

[0368] (15) The information processing apparatus according to (14), wherein the location determination unit generates metadata at the user's changed location by interpolation processing.

[0369] (16) An information processing apparatus according to any one of (13) to (15), wherein the metadata includes at least one of object location information in space, object gain, object-related priority information, sound source type information indicating the type of object, and extension information indicating the extent of object extension.

[0370] (17) The information processing apparatus according to any one of (13) to (16) further includes a rendering processing unit configured to perform rendering processing based on the metadata of the object at the user's changed location and the audio data of the object, and to generate output audio data.

[0371] (18) The information processing apparatus according to (17), wherein the rendering process is a process using at least one of HRTF, BRIR, RIR, ITD, IID, HOA and VBAP.

[0372] (19) An information processing apparatus according to any one of (1) to (18), wherein when the user's position is outside the prohibited space, the position determination unit does not change the user's position.

[0373] (20) The information processing apparatus according to (9) further includes a presentation control unit configured to control a notification indicating that entry into a prohibited space is prohibited based on a warning message.

[0374] (21) An information processing method, comprising: Based on location information representing the user's position in space, and prohibited space information used to specify a prohibited space including a predetermined target point and prohibiting the user from entering, it is determined whether the user's position, as indicated by the location information, is within the prohibited space. When a user's location is within the prohibited space, the user's location will be changed to a location outside the prohibited space that is different from the location indicated by the location information.

[0375] (22) A program configured to cause a computer to perform processing, the processing including: Based on location information representing the user's position in space, and prohibited space information used to designate a prohibited space including a predetermined target point and prohibiting the user from entering, it is determined whether the user's position, as indicated by the location information, is within the prohibited space. When a user's location is within the prohibited space, the user's location will be changed to a location outside the prohibited space that is different from the location indicated by the location information.

[0376] Reference number list

[0377] 11 Server; 21 Meta Encoder; 22 Audio Encoder; 23 Communication Department; 61 Client; 71 Communication Department; 72 Audio Decoder; 73 Listener Information Acquisition Department; 74 Meta Decoder; 75 Rendering Processing Department; 76 Presentation Control Department.

Claims

1. An information processing apparatus, comprising: The positioning unit is configured as follows: Based on location information representing the user's position in space, and prohibited space information used to specify a prohibited space including a predetermined target point and prohibiting the user from entering, it is determined whether the user's position, represented by the location information, is located within the prohibited space. When the user's location is within the prohibited space, the user's location is changed to a location outside the prohibited space that is different from the location indicated by the location information.

2. The information processing apparatus according to claim 1, wherein, The location determination unit sets the location on the surface of the prohibited space or outside the prohibited space that is closest to the user's location represented by the location information as the user's changed location.

3. The information processing apparatus according to claim 1, wherein, The location determination unit sets the position of the intersection point of the surface of the prohibited space and the vector connecting the user's position, represented by the location information, and the target point as the user's changed position.

4. The information processing apparatus according to claim 3, wherein, The location determination unit sets the location of the intersection point among the plurality of intersection points that is closest to the user's location represented by the location information as the user's changed location.

5. The information processing apparatus according to claim 1, wherein, The forbidden space is a spherical space.

6. The information processing apparatus according to claim 1, wherein, The forbidden space is a space that includes a given object located within the space.

7. The information processing apparatus according to claim 1, wherein, The prohibited space is the target point.

8. The information processing apparatus according to claim 1, wherein, When the user's location is within the prohibited space, the location determination unit sets the user's previous location as the user's changed location.

9. The information processing apparatus according to claim 1, wherein, The location determination unit outputs a warning message indicating whether the user's location is within the prohibited space.

10. The information processing apparatus of claim 9, further comprising a presentation control unit configured to display an image related to the space based on location information representing the user's final location, the user's orientation in the space, and the warning information.

11. The information processing apparatus according to claim 10, wherein, The image shows at least one of the forbidden space, the target point, and the object.

12. The information processing apparatus according to claim 10, wherein, The image shows a notice indicating that entry into the restricted space is prohibited.

13. The information processing apparatus according to claim 1, wherein, When the user's location is within the prohibited space, the location determination unit generates metadata of objects located within the space based on location information representing the user's changed location.

14. The information processing apparatus according to claim 13, wherein, Multiple locations in the space are set as control viewpoints, and The location determination unit generates metadata for the user's changed location based on the metadata of each of the plurality of control viewpoints and location information representing the user's changed location.

15. The information processing apparatus according to claim 14, wherein, The location determination unit generates the metadata for the user's changed location through interpolation processing.

16. The information processing apparatus according to claim 13, wherein, The metadata includes at least one of the following: the location information of the object in the space, the gain of the object, the priority information related to the object, the sound source type information indicating the type of the object, and the extension information indicating the extent of the object's extension.

17. The information processing apparatus of claim 13 further includes a rendering processing unit configured to perform rendering processing based on the metadata of the object at the user's changed location and the audio data of the object, and to generate output audio data.

18. The information processing apparatus according to claim 17, wherein, The rendering process is performed using at least one of HRTF, BRIR, RIR, ITD, IID, HOA, and VBAP.

19. An information processing method, comprising: The information processing device identifies whether the user's location, as indicated by the location information, is within the prohibited space based on location information representing the user's location in space and prohibited space information used to designate a prohibited space including a predetermined target point and prohibiting the user from entering. as well as When the user's location is within the prohibited space, the information processing device changes the user's location to a location outside the prohibited space that is different from the location indicated by the location information.

20. A program configured to cause a computer to perform processing, said processing comprising: Based on location information representing the user's location in space, and prohibited space information for specifying a prohibited space including a predetermined target point and prohibiting the user from entering, it is identified whether the user's location, represented by the location information, is located within the prohibited space; as well as When the user's location is within the prohibited space, the user's location is changed to a location outside the prohibited space that is different from the location indicated by the location information.

Images