Information processing device and method

Abstract

The present disclosure pertains to an information processing device and method making it possible to curb any increase in the load of playback processing while also curbing any decrease in the quality of a user experience. The definition of a static 3D model showing the outer appearance of a 3D model is set, and the definition of skeleton information indicating a moving structure of the 3D model is set independently from the static 3D model. Content that includes the static 3D model and skeleton information corresponding to the set definitions, as well as skin weight information indicating a relationship of correspondence between the outer appearance and the moving structure, is played back. In addition, control information for controlling the supply of the content is supplied, the control information representing, independently from one another, a variation in the definition of the static 3D model, a variation in the definition of the skeleton information, and a variation in the skin weight information. The present disclosure can be applied, for example, to an information processing device, an electronic apparatus, an information processing method, a program, or the like.

Description

Information processing device and method

[0001] The present disclosure relates to an information processing device and method, and more particularly to an information processing device and method that are capable of suppressing an increase in the load of playback processing while suppressing a decrease in the quality of user experience.

[0002] Conventionally, skinned models have been used as 3D models that represent the three-dimensional shape of objects. Because the playback process of such 3D models places a heavy load on the user, a method for reducing the load has been sought. For example, there has been a method for reducing the playback process load by reducing the resolution of the appearance and movement structure of a 3D model depending on the distance from the viewpoint (see, for example, Patent Document 1). However, this method only allows control based on the distance from the viewpoint, making it difficult to reduce the playback process load when, for example, the 3D model is located close to the viewpoint. Furthermore, reducing the playback process load requires reducing the quality of the appearance, which can potentially degrade the quality of the user experience. As such, this method is not always optimal.

[0003] In response to this, there is a method for reducing the load of the playback process by reducing the resolution (resolution of the movement) of the motion data that moves the Skinned model (see, for example, Non-Patent Document 1).

[0004] Japanese Patent Application Laid-Open No. 2012-212246

[0005] Shingo Hasegawa, Koji Nishio, Kenichi Kobori, "A Method for LOD of Motion Capture Data", FIT2010 (9th Forum on Information Science and Technology), https: / / www.ieice.org / publications / conference-FIT-DVDs / FIT2010 / pdf / I / I_052.pdf

[0006] However, with this method, there is a risk that the reduction in the load of the playback process may be limited, for example, because the amount of information in the Skinned model remains unchanged. In other words, this method is not always optimal. For these reasons, there has been a demand for other methods that can reduce the increase in the load of the playback process while minimizing the reduction in the quality of the user experience.

[0007] The present disclosure has been made in light of such circumstances, and makes it possible to suppress an increase in the load of playback processing while suppressing a decrease in the quality of the user experience.

[0008] An information processing device according to one aspect of the present technology includes: a control information supply unit that supplies control information for controlling the supply of content including a still 3D model that indicates the appearance of a 3D model, skeleton information that indicates a motion structure of the 3D model, and skin weight information that indicates a correspondence between the appearance and the motion structure; and a content supply unit that supplies the content including the still 3D model with a level of resolution requested based on the control information, the skeleton information with a level of resolution requested based on the control information, and the skin weight information requested based on the control information, wherein the control information represents variations in the level of resolution of the still 3D model, variations in the level of resolution of the skeleton information, and variations in the skin weight information independently of one another.

[0009] An information processing method according to one aspect of the present technology is an information processing method that supplies control information for controlling the supply of content including a still 3D model that indicates the appearance of a 3D model, skeleton information that indicates the motion structure of the 3D model, and skin weight information that indicates the correspondence between the appearance and the motion structure, and supplies the content including the still 3D model with a level of detail requested based on the control information, the skeleton information with a level of detail requested based on the control information, and the skin weight information requested based on the control information, wherein the control information represents variations in the level of detail of the still 3D model, variations in the level of detail of the skeleton information, and variations in the skin weight information independently of one another.

[0010] An information processing device according to another aspect of the present technology includes: a still 3D model resolution setting unit that sets the resolution of a still 3D model that indicates the appearance of the 3D model; a skeleton information resolution setting unit that sets the resolution of skeleton information that indicates the movement structure of the 3D model independently of the still 3D model; and a content playback unit that plays back content that corresponds to the set resolution and includes the still 3D model, the skeleton information, and skin weight information that indicates the correspondence between the appearance and the movement structure.

[0011] Another aspect of the present technology is an information processing method that sets the resolution of a still 3D model that indicates the appearance of the 3D model, sets the resolution of skeletal information that indicates the movement structure of the 3D model independently of the still 3D model, and plays content that includes the still 3D model, the skeletal information, and skin weight information that indicates the correspondence between the appearance and the movement structure, corresponding to the set resolution.

[0012] In an information processing device and method according to one aspect of the present technology, control information is supplied that controls the supply of content including a still 3D model that indicates the appearance of the 3D model, skeleton information that indicates the motion structure of the 3D model, and skin weight information that indicates the correspondence between the appearance and the motion structure, and content is supplied that includes a still 3D model of a level of resolution requested based on the control information, skeleton information of a level of resolution requested based on the control information, and skin weight information requested based on the control information. In the control information, variations in the resolution of the still 3D model, variations in the resolution of the skeleton information, and variations in the skin weight information are expressed independently of one another.

[0013] In an information processing device and method according to another aspect of the present technology, the resolution of a still 3D model that indicates the appearance of the 3D model is set, and the resolution of skeletal information that indicates the movement structure of the 3D model is set independently of the still 3D model, and content is played back that includes the still 3D model, skeletal information, and skin weight information that indicates the correspondence between the appearance and movement structure, corresponding to the set resolution.

[0014] 1 is a diagram for explaining a Skinned model. FIG. 1 is a diagram for explaining control of the level of definition of appearance. FIG. 2 is a diagram for explaining an overview of control of the level of definition of movement. FIG. 3 is a diagram for explaining an example of motion data. FIG. 4 is a diagram for explaining control of the level of definition of movement. FIG. 5 is a diagram for explaining an example of a method for controlling the level of definition of a 3D model. FIG. 6 is a diagram for explaining an example of an overview of control of the level of definition of a 3D model. FIG. 7 is a diagram for explaining an example of skin weight information. FIG. 8 is a diagram for explaining an example of skin weight information. FIG. 9 is a diagram for explaining an example of skin weight information. FIG. 10 is a diagram for explaining an example of movement. FIG. 11 is a diagram for explaining playback processing. FIG. 12 is a diagram for explaining the load in the case of highest definition. FIG. 13 is a diagram for explaining an example of reducing the level of definition of movement. FIG. 14 is a diagram for explaining an example of the load when reducing the level of definition of movement. FIG. 15 is a diagram for explaining an example of reducing the level of definition of movement structure. FIG. 16 is a diagram for explaining an example of the load when reducing the level of definition of movement structure. FIG. 17 is a diagram for explaining an example of glTF2.0. FIG. 18 is a diagram for explaining an example of glTF2.0. FIG. 19 is a diagram for explaining an example of an MP4 file. FIG. 19 is a diagram for explaining an example of MPD. FIG. 19 is a diagram for explaining an example of Universal Scene Description. FIG. 19 is a diagram for explaining an example of Universal Scene Description. FIG. 1 is a block diagram showing an example of the main configuration of a distribution device; FIG. 2 is a flowchart explaining an example of the flow of a distribution process; FIG. 3 is a flowchart explaining an example of the flow of a generation process; FIG. 4 is a block diagram showing an example of the main configuration of a playback device; FIG. 5 is a flowchart explaining an example of the flow of a playback process; FIG. 6 is a flowchart explaining an example of the flow of an acquisition process; FIG. 7 is a flowchart explaining an example of the flow of an acquisition process; FIG. 8 is a block diagram showing an example of the main configuration of a computer.

[0015] The following describes modes for carrying out the present disclosure (hereinafter referred to as embodiments). The description will be given in the following order: 1. Literature supporting technical content and technical terminology 2. Playback of Skinned Models 3. Control of 3D Model Resolution 4. First Embodiment (Distribution Device) 5. Second Embodiment (Playback Device) 6. Supplementary Notes

[0016] <1. Literature, etc. supporting technical content and technical terminology> The scope of disclosure of the present technology includes not only the content described in the embodiments, but also the content described in the following patent documents and non-patent documents that were publicly known at the time of filing, as well as the content of other documents referenced in the following patent documents and non-patent documents.

[0017] Patent document 1: (described above) Non-patent document 1: (described above)

[0018] In other words, the contents of the above-mentioned documents and the contents of other documents referenced in the above-mentioned documents are also used as the basis for determining support requirements.

[0019] 2. Reproduction of Skinned Models Skinned Models Skinned models have been used as 3D models that represent the three-dimensional shape of objects. For example, as shown in Fig. 1, a Skinned model 11 includes skeleton information 21, a static 3D model 22, and skin weight information 23.

[0020] The skeleton information 21 is information that indicates the skeletal structure of a 3D model, called bones. In the skeleton information 21 in FIG. 1, the straight lines between black dots indicate bones. The joints between bones (black dots in the skeleton information 21 in FIG. 1) are structured to be movable, and these joints represent the joints of the 3D model. In other words, the skeleton information 21 can also be considered information that indicates the posture of the 3D model based on the arrangement (movement) of the bones. In other words, because the bone structure determines how the 3D model can move, the skeleton information 21 can also be considered information that indicates the movement structure of the 3D model.

[0021] The static 3D model 22 is information that represents the three-dimensional structure of an object, such as a mesh or a point cloud. In other words, the static 3D model 22 can also be considered information that indicates the appearance of the 3D model. The static 3D model 22 is formed according to the shape (bone arrangement) of the skeleton information 21, so that it can represent the appearance of the posture indicated by the skeleton information 21.

[0022] The skin weight information 23 is information that associates the skeleton information 21 (bones) with the static 3D model (vertices, etc.). In other words, the skin weight information 23 can also be considered information that indicates the correspondence between the skeleton information 21 (bones) and the static 3D model 22 (vertices, etc.). According to the correspondence indicated in this skin weight information 23, the shape of the static 3D model 22 can be made to correspond to the shape of the skeleton information 21. Note that, as described above, the skeleton information indicates the motion structure of the 3D model, and the static 3D model 22 indicates the appearance of the 3D model, so the skin weight information 23 can also be considered information that indicates the correspondence between the appearance and the motion structure.

[0023] Such a skinned model 11 can be moved by motion data 12. The motion data 12 is information that indicates how the skinned model 11 moves. More specifically, the motion data 12 is composed of information that indicates how each bone in the skeleton information 21 moves, etc. Therefore, the motion data 12 can also be said to be information that indicates the movement of the 3D model.

[0024] Currently, development is underway on video streaming technology (6DoF video streaming technology) that provides a viewing experience from any viewpoint. To enable viewing from any position, everything that appears in the video content is composed of 3D objects such as meshes and point clouds, and the content is distributed along with a scene description file that manages scene information such as where each 3D object is placed in the virtual space. This allows users to move freely within the virtual space and view the content from any position they like. Recently, a technology that captures users' own movements as motion data and reproduces those movements through an avatar in the virtual space has attracted attention, enabling not only one-way viewing but also two-way remote communication such as conversation, gestures, and collaboration with other users.

[0025] For such experiential content, the quality of the user experience in virtual space is important. Reality is one of the values ​​that constitutes that quality. In other words, the closer an experience is to the real-world experience, the less uncomfortable the user feels and the more naturally they can accept it. In other words, the quality of the user experience improves. Therefore, a more realistic experience (a more accurate reproduction of the real-world experience) is required in virtual space. For example, not only one-way viewing but also two-way remote communication such as conversation, gestures, and collaboration with other users is realized in virtual space. To improve the realism of such user experiences, improvements are required, for example, in the visual quality of avatars (i.e., the detail of their appearance) and the faithful reproduction of human movements (i.e., the detail of their movements, etc.). Ideally, what is needed is a true metaverse that enables a realistic virtual space reproduction that is indistinguishable from real space and natural interactions with people in remote locations as if they were in the same space.

[0026] By applying bone animation using the above-described Skinned model 11 to such an avatar, it is possible to express more realistic appearances and movements, for example by projecting the user's facial expressions, gestures, and lip movements in real time. However, bone animation, due to its structure, contains a large amount of information and places a heavy load on the playback process. For this reason, for example, because the number of bones significantly affects the processing load of animation, it is sometimes recommended to limit the number of bones used for one object to around 30 in order to achieve optimal performance.

[0027] However, making avatars appear as believable as real-world avatars requires a massive number of bones. For example, some avatars use as many as 713 bones for the face alone to achieve a high level of realism. This large amount of data must be transmitted to all users in the space without any delay and processed in real time (instantly). Even the slightest delay can reduce the realism and cause users to feel uncomfortable. To process everything in real time without compromising realism, some say 1,000 times the computing resources currently available would be required, and resources are insufficient. Therefore, optimal processing load adjustments that reduce processing resources without compromising the realism perceived by users are crucial. In other words, a method to reduce the load on the playback process was needed.

[0028] For example, Patent Document 1 proposes a method for reducing the load of the playback process by reducing the resolution of the appearance and motion structure of a 3D model depending on the distance from the viewpoint. In other words, in 3D CG, a single 3D object has multiple model data (static 3D models and skeleton information) with different levels of resolution, and the model data to be displayed is switched depending on the distance from the user's viewpoint to the 3D object. For example, as shown in Figure 2, static 3D models 31 and 32 are used. When the distance from the viewpoint is close, the more detailed static 3D model 31 is displayed, and when the distance from the viewpoint is farther, the coarser static 3D model 32 is displayed. The static 3D model 32 has fewer vertices and faces and therefore less information than the static 3D model 31. Therefore, by displaying the static 3D model 32, the load of the playback process can be reduced. By displaying a rough static 3D model such as the static 3D model 32 for an avatar located far from the viewpoint, it is possible to make it difficult for a user to notice a reduction in the resolution of the appearance (the geometric resolution of the model, such as the number of polygons in a mesh or the number of vertices in a point cloud). In other words, the impact of the reduction in resolution of the appearance on the quality of the user experience is suppressed.

[0029] In the method described in Patent Literature 1, in addition to controlling the resolution of the static 3D model (i.e., appearance), the resolution of the skeleton information (i.e., movement structure) is also controlled. For example, when the distance from the viewpoint to the 3D model is short, higher-resolution skeleton information (i.e., skeleton information with a larger number of bones) is selected. Conversely, when the distance from the viewpoint to the 3D model is long, lower-resolution skeleton information (i.e., skeleton information with a smaller number of bones) is selected. Generally, the more bones there are, the more complex movements can be expressed, but the greater the load on the playback process. Conversely, the fewer bones there are, the simpler movements can be expressed, but the lower the load on the playback process. By selecting lower-resolution skeleton information for avatars located farther from the viewpoint, the user can be less likely to notice the reduction in the resolution of the movement structure (number of bones). In other words, the impact of the reduction in the resolution of the movement structure on the quality of the user experience is suppressed.

[0030] However, this method only allowed control based on the distance from the viewpoint. For example, high-resolution static 3D models and high-resolution skeleton information were applied to 3D models located close to the viewpoint. Therefore, it was difficult to reduce the playback processing load when 3D models were located close to the viewpoint. For example, an increase in the number of 3D models located close to the viewpoint could increase the playback processing load. Furthermore, because the resolution of the static 3D models and skeleton information was controlled based on a common parameter (distance), reducing the playback processing load required reducing the appearance quality. This could result in a degradation of the user experience. For example, if the number of 3D models located close to the viewpoint increased as described above, reducing the playback processing load required applying low-resolution static 3D models to the close 3D models. This could result in a degradation of the appearance quality of the 3D models located close to the viewpoint. In other words, the quality of the user experience could be significantly reduced. As such, this method was not always optimal.

[0031] Furthermore, Non-Patent Document 1 proposes a method for reducing the load of playback processing by reducing the resolution (resolution of movement) of the motion data that drives a skinned model. In this method, as shown in Figure 3, for example, high-resolution motion data (High motion data 12-1) and low-resolution motion data (Low motion data 12-2) are prepared, and the motion data to be used is switched depending on the situation.

[0032] FIG. 4 shows examples of descriptions of high motion data 12-1 and low motion data 12-2. Description example 51 shows an example of description of high motion data 12-1. Description example 52 shows an example of description of low motion data 12-2. Motion data is a collection of Transform information (translation, rotation, and scale information) that represents the movement of each bone that makes up the skeleton of the skinned model to be animated, based on the skeletal structure of the model. Description example 51 shows Transform information for three bones (Bone01, Bone02, and Bone03), while description example 52 shows Transform information for two bones (Bone01 and Bone02). In other words, description example 52 omits the description of Transform information for Bone03 compared to description example 51, resulting in a smaller amount of information than description example 51. Transform information not shown in the motion data (bone movement (posture change)) is processed as if the bone did not move, allowing for independent adjustment of the animation processing load.

[0033] In this specification, bones connected to the end of a series of bones are referred to as child bones, grandchild bones, etc. (collectively referred to as descendant bones), and bones connected to the opposite end are referred to as parent bones, grandparent bones, etc. (collectively referred to as ancestor bones). In motion data, the movement of a bone is represented as a movement based on its parent bone (relative movement). In other words, the actual movement of a bone is the sum of the movements of that bone and all of its ancestor bones described in the motion data. Therefore, for example, in the case of Low motion data 12-2 (description example 52), Bone03 is fixed (does not move) relative to Bone02, but moves from an overall perspective (its global position moves).

[0034] For example, in the initial state, the position of Bone01 is triangle 61 shown on the left of Fig. 5, the position of Bone02 is triangle 62 shown on the left of Fig. 5, and the position of Bone03 is triangle 63 shown on the left of Fig. 5. In other words, it is assumed that Bone01 to Bone03 are connected in series in the same direction (straight line).

[0035] In the case of High motion data 12-1 (description example 51), the movements of Bone01 to Bone03 are described, so that, for example, as shown in the center of Figure 5, Bone01 to Bone03 can be moved in directions independent of each other. In the example shown in the center of Figure 5, Bone01 moves from the position of triangle 61 to the position of triangle 61A. Bone02 moves from the position of triangle 62 to the position of triangle 62A. Bone03 moves from the position of triangle 63 to the position of triangle 63A. As shown by the arrows, after the movements, Bone02 (triangle 62A) and Bone03 (triangle 63A) are facing in different directions.

[0036] In contrast, in the case of Low motion data 12-2 (description example 52), the movements of only Bone01 and Bone02 are described, so the positional relationship (relative posture) between Bone02 and Bone03 remains fixed, as shown on the right side of Figure 5, for example. In the example shown on the right side of Figure 5, Bone01 moves from the position of triangle 61 to the position of triangle 61B. Bone02 moves from the position of triangle 62 to the position of triangle 62B. Bone03 moves from the position of triangle 63 to the position of triangle 63B. As shown by the arrows, after the movements, the orientations of Bone02 (triangle 62B) and Bone03 (triangle 63B) are the same.

[0037] In other words, in the case of low motion data 12-2 (description example 52), the degree of freedom of movement of Bone 03 is reduced compared to the case of high motion data 12-1 (description example 51). In other words, in the case of low motion data 12-2 (description example 52), the resolution of movement is reduced compared to the case of high motion data 12-1 (description example 51). Therefore, the amount of processing required for movement-related playback processing can be reduced.

[0038] For example, in the case of an avatar, the reduction in the resolution of the motion data (i.e., the resolution of the movements) described above increases the number of user movements that the avatar cannot display. For example, even if a user bends their finger, the movement may not be represented in the motion data, and the movement may not be reflected in the avatar's movements. However, a user viewing the avatar may not notice the reduction in the resolution of the movements because they cannot tell whether the avatar's user has bent their finger. In other words, the increase in the load of the playback process can be suppressed while suppressing a reduction in the quality of the user experience. In particular, if the avatar is not important to the viewing user (e.g., if it is not a conversation partner), the discomfort caused by the reduction in the resolution of the movements (reduced number of moving parts and limited movements) is even less likely to occur.

[0039] This method only controls the resolution of the motion data (i.e., the resolution of the movement), but does not change the resolution of the appearance. Therefore, even if this method is applied to an avatar located close to the viewpoint, it is possible to suppress an increase in the load of the playback process while suppressing a decrease in the quality of the user experience.

[0040] However, this method only reduces the amount of information in the motion data, and the amount of information in the Skinned model remains unchanged, so there is a risk that the reduction in the load on the playback process may be limited. In other words, this method is not always optimal.

[0041] In view of the above, there has been a demand for another method for suppressing an increase in the load of playback processing while suppressing a decrease in the quality of the user experience.

[0042] 3. Control of the Level of Resolution of a 3D Model Method 1 Therefore, as shown in the top row of the table in FIG. 6, the level of resolution of the static 3D model and the level of resolution of the skeleton information are controlled independently of each other (Method 1).

[0043] For example, the first information processing device may include a still 3D model resolution setting unit that sets the resolution of a still 3D model that indicates the appearance of the 3D model, a skeleton information resolution setting unit that sets the resolution of skeleton information that indicates the movement structure of the 3D model independently of the still 3D model, and a content playback unit that plays content that corresponds to the set resolution and includes the still 3D model, skeleton information, and skin weight information that indicates the correspondence between the appearance and movement structure.

[0044] For example, in a first information processing method executed by a first information processing device, the resolution of a still 3D model that indicates the appearance of the 3D model may be set, and the resolution of skeletal information that indicates the movement structure of the 3D model may be set independently of the still 3D model, and content may be played that includes the still 3D model, skeletal information, and skin weight information that indicates the correspondence between the appearance and movement structure, corresponding to the set resolution.

[0045] Here, a static 3D model may be any information that represents the appearance of a 3D model. For example, a static 3D model may be composed of 3D data that represents the three-dimensional structure of an object. For example, a static 3D model may be point cloud data that represents the shape of an object using a collection (distribution) of points that indicate the presence of an object in a small area. Alternatively, a static 3D model may be mesh data that represents the surface shape of an object using vertices, the connections (edges) between the vertices, and the faces formed by them. Furthermore, a static 3D model may include attribute information such as texture. Of course, a static 3D model may also be other 3D data.

[0046] A still 3D model may have a variable level of detail (i.e., the level of detail of its appearance). For example, a still 3D model may have variations in the level of detail of its appearance, and one of these may be applied. In other words, still 3D models may be hierarchically organized according to the level of detail of their appearance. For example, static 3D models with multiple levels of detail may be prepared as candidates, such as a high-resolution still 3D model and a low-resolution still 3D model, and a still 3D model with a desired level of detail may be selected and applied from these candidates. In this specification, this level is also referred to as LoD (Level of Detail). Note that the number of variations (LoD number) of this still 3D model may be any number greater than or equal to two. Of course, a still 3D model with a freely set level of detail may be generated and provided without providing variations (multiple candidates).

[0047] In a static 3D model, the level of detail of the appearance may be expressed in any manner. For example, if the static 3D model is point cloud data, the level of detail of the appearance may be expressed by the number of vertices. For example, the more points there are, the higher the detail of the static 3D model, and the fewer points there are, the lower the detail of the static 3D model. Furthermore, if the static 3D model is mesh data, the level of detail of the appearance may be expressed by the number of vertices or connections (or the number of faces). For example, the more faces there are, the higher the detail of the static 3D model, and the fewer faces there are, the lower the detail of the static 3D model. Furthermore, the level of detail of the appearance may be expressed by the resolution of attribute information such as texture. Of course, the level of detail of the appearance may be expressed in ways other than these examples.

[0048] Furthermore, the skeleton information may be any information that indicates the movement structure of the 3D model. For example, the skeleton information may indicate the movement structure of the 3D model using a group of bones, or may indicate the movement structure of the 3D model using information other than bones.

[0049] The level of detail of the skeleton information (i.e., the level of detail of the motion structure) is variable. For example, the skeleton information may have variations in the level of detail of the motion structure, and one of these may be applied. In other words, the skeleton information may be hierarchically organized according to the level of detail of the motion structure. For example, multiple levels of detail of skeleton information may be prepared as candidates, such as high-resolution skeleton information and low-resolution skeleton information, and a desired level of detail may be selected and applied from these candidates. In this specification, this level is also referred to as LoD. Note that the number of variations of this skeleton information (LoD number) may be any number equal to or greater than two. Of course, without preparing variations (multiple candidates), skeleton information with a freely set level of detail may be generated and provided.

[0050] In skeleton information, the resolution of the movement structure may be expressed in any way. For example, if the skeleton information is information that indicates the movement structure of a 3D model using a group of bones, the resolution of the movement structure may be expressed by the number of bones (the number of joints (connections between bones)). For example, the greater the number of bones, the higher the resolution of the skeleton information may be, and the fewer the number of bones, the lower the resolution of the skeleton information may be. Of course, the resolution of the movement structure may be expressed in ways other than this example.

[0051] "Controlling independently of each other" in Method 1 means controlling one of the two independently of the other. For example, it means controlling the resolution of the still 3D model independently of controlling the resolution of the skeleton information. It also means controlling the resolution of the skeleton information independently of controlling the resolution of the still 3D model. Therefore, for example, as shown in Figure 7, it is possible to control the resolution of the skeleton information without changing the resolution of the still 3D model.

[0052] 7, in the Skinned model 111, the resolution of the still 3D model remains constant (remains the High still 3D model 122), while the resolution of the skeleton information is switched (controlled) by applying high-resolution High skeleton information 121-1 or low-resolution Low skeleton information 121-2. By such control, for example, it is possible to reduce the resolution of the skeleton information without reducing the resolution of the still 3D model.

[0053] By reducing the resolution of the skeleton information (reducing the resolution of the motion structure), the increase in the load of the playback process can be suppressed. Reducing the resolution of the still 3D model (reducing the resolution of the appearance) can also suppress the increase in the load of the playback process, but in that case, the quality of the appearance may be reduced, which may degrade the quality of the user experience. By reducing the resolution of the skeleton information without reducing the resolution of the still 3D model, as in the example of Figure 7, the increase in the load of the playback process can be suppressed without reducing the resolution of the appearance. In other words, the increase in the load of the playback process can be suppressed while suppressing a reduction in the quality of the user experience.

[0054] Of course, as in the example of FIG. 8 , the resolution of the still 3D model can also be controlled. As described above, the control of the resolution of the still 3D model can be performed independently of the control of the resolution of the skeleton information. That is, in the example of FIG. 8 , switching between the high-resolution High still 3D model 122-1 and the low-resolution Low still 3D model 122-2 can be performed independently of switching between the High skeleton information 121-1 and the Low skeleton information 121-2. Therefore, for example, the High skeleton information 121-1 and the High still 3D model 122-1 may be applied. Alternatively, the High skeleton information 121-1 and the Low still 3D model 122-2 may be applied. Alternatively, the Low skeleton information 121-2 and the High still 3D model 122-1 may be applied. Alternatively, the Low skeleton information 121-2 and the Low still 3D model 122-2 may be applied. In this way, more diverse control can be performed. This allows for more diverse control of the playback processing load, making it possible to control the playback processing load in a way that is more suited to the situation, and suppressing an increase in the playback processing load while suppressing a decrease in the quality of the user experience.

[0055] For example, by reducing the resolution of a static 3D model for a 3D model where the reduction in appearance quality is not noticeable, such as a 3D model located far from the viewpoint, it is possible to further reduce the increase in the playback processing load while suppressing the reduction in the quality of the user experience. Furthermore, for a 3D model located at a certain distance from the viewpoint but with striking and noticeable movement, it is possible to maintain the quality of the movement while reducing the quality of the appearance by reducing the resolution of the static 3D model without reducing the resolution of the skeleton information. In other words, it is possible to control the quality of less noticeable information to be reduced while maintaining the quality of more noticeable information. Therefore, it is possible to further reduce the increase in the playback processing load while further suppressing the reduction in the quality of the user experience in a wider variety of situations.

[0056] <Examples of variations in the resolution of the movement structure> Next, examples of variations in the resolution of the movement structure, such as the high skeleton information 121-1 and the low skeleton information 121-2, will be described. In order to reproduce realistic movement, the skeletal structures of recent skinned models (for example, the bone structures of avatars) have evolved into complex, highly detailed structures with deep hierarchical structures, as shown below.

[0057] Level 1: A skeleton structure made up of bones that express the movements of human bones and muscles, such as the hands, ankles, fingers, mouths, and eyes. Level 2: A skeleton structure in which, in addition to the bones in level 1, bones are added to the end to reproduce the movement of swaying objects, such as clothing and hair, that occurs when the hands and ankles move. Level 3: A skeleton structure in which, in addition to level 2, bones are added to the end to reproduce minute changes in movement, such as wrinkles in the face and clothing.

[0058] This is an evolutionary step (Level 1 → Level 2 → Level 3) based on a Level 1 skeletal structure that reproduces basic human movements, with the addition of bones to reproduce more detailed movements in pursuit of even greater realism. Therefore, the higher the level of detail, the more bones are added to reproduce more subtle movements, which can be considered less noticeable depending on the viewing situation. Furthermore, the number of bones added to express these subtle movements is inversely proportional to the fineness of the movements being expressed. Therefore, simply switching from a Level 3 skeletal structure to a Level 2 skeletal structure reduces the number of bones significantly, resulting in a significant reduction in processing load, while minimizing the impact on the realism perceived by the user in certain viewing situations.

[0059] Therefore, such levels may be used to realize a hierarchical structure of the motion structure, for example, a high-definition layer (H-LoD) may have a level 3 skeletal structure, a medium-definition layer (M-LoD) may have a level 2 skeletal structure, and a low-definition layer (L-LoD) may have a level 1 skeletal structure.

[0060] The number of levels of detail in the motion structure may be any number, for example, three levels as in this example, two levels as in the example of Fig. 7, or four or more levels.

[0061] Each variation of the skeletal structure data with different levels of detail may be generated in advance, or variations of the skeletal structure at lower levels of detail may be generated in real time (instantly) by deleting the end bones from the H-LOD skeleton before streaming of the motion data begins.

[0062] <Control of Skin Weight Information> As shown in the examples of Figures 7 and 8, skin weight information is also controlled in accordance with control of the resolution of the static 3D model and the resolution of the skeleton information. This skin weight information may be any information that indicates the correspondence between the appearance and the motion structure. For example, if the static 3D model is point cloud data and the skeleton information is information configured from a group of bones, the skin weight information may be information that indicates the correspondence between the points and the bones. Also, if the static 3D model is mesh data and the skeleton information is information configured from a group of bones, the skin weight information may be information that indicates the correspondence between the faces (or vertices or connections) of the mesh and the bones.

[0063] In other words, the content of the skin weight information depends on the still 3D model and skeleton information. In other words, the content of the skin weight information depends on the resolution of the still 3D model and skeleton information. In other words, as described above, when the still 3D model and skeleton information are layered according to their respective levels of resolution, the content of the skin weight information also varies, and one of these is applied. In other words, the skin weight information may also be layered. However, this layered structure depends on both the resolution of the appearance and the resolution of the motion structure.

[0064] As in the example of Figure 7, when the still 3D model is fixed at High still 3D model 122 and the skeleton information has variations of High skeleton information 121-1 and Low skeleton information 121-2, the skin weight information has variations of skin weight information 123-1 whose content corresponds to High skeleton information 121-1 and High still 3D model 122, and skin weight information 123-2 whose content corresponds to Low skeleton information 121-2 and High still 3D model 122.

[0065] As in the example of Figure 8, when the still 3D model has variations of high still 3D model 122-1 and low still 3D model 122-2 and the skeleton information has variations of high skeleton information 121-1 and low skeleton information 121-2, the skin weight information has variations of skin weight information 123-11 corresponding to the high skeleton information 121-1 and the high still 3D model 122-1, skin weight information 123-12 corresponding to the low skeleton information 121-2 and the high still 3D model 122-1, skin weight information 123-13 corresponding to the high skeleton information 121-1 and the low still 3D model 122-2, and skin weight information 123-14 corresponding to the low skeleton information 121-2 and the low still 3D model 122-2.

[0066] Then, from these variations, one is selected that corresponds to the selected level of detail for the still 3D model and the skeleton information. Note that the number of variations (LoD numbers) of this skin weight information depends on the number of variations (LoD numbers) for the still 3D model and the skeleton information. Of course, it is also possible to generate and provide skin weight information that corresponds to the selected level of detail for the still 3D model and the skeleton information without preparing candidates (variations).

[0067] 9, it is assumed that bone A, bone B, and bone C are described in high skeleton information 121-1, and vertices i (i=0 to i=16) corresponding to these bones are described in high still 3D model 122. Note that the following description will be given assuming that the still 3D model is mesh data.

[0068] A description example 131 shown in FIG. 10 is an example of the description of the skin weight information 123-1 in this case. As shown in FIG. 10, the skin weight information describes, for each bone, the index and weight value of the vertex corresponding to that bone. The weight value indicates the degree of influence of the vertex on the movement of the bone, ranging from 0 to 1.0. In other words, the index and weight value indicate the extent to which each vertex is influenced by the movement of the bone. For example, if the position of a certain vertex P depends on the movement of bones A and B, the index and weight value of vertex P are associated with bones A and B, respectively. The sum of the weight value associated with bone A and the weight value associated with bone B is 1.0. If the vertex is more strongly influenced by the movement of bone A, the weight value on the bone A side is increased and the weight value on the bone B side is decreased. Conversely, if the vertex is more strongly influenced by the movement of bone B, the weight value on the bone A side is decreased and the weight value on the bone B side is increased.

[0069] In the case of skin weight information 123-1, as shown in example 131, the correspondence between vertex i and each of bones A to C and their weight values ​​are shown as a list of vertices. For example, vertex (i=0,1,15,16) is associated with bone A only, and its weight value is 1.0. vertex (i=2,14) is associated with bones A and B, and its weight values ​​are 0.9 and 0.1, respectively. vertex (i=3,13) is associated with bones A and B, and its weight values ​​are 0.1 and 0.9, respectively. vertex (i=4,12) is associated with bone B only, and its weight value is 1.0. vertex (i=5,11) is associated with bones B and C, and its weight values ​​are 0.9 and 0.1, respectively. vertex (i=6,10) is associated with bones B and C, and its weight values ​​are 0.1 and 0.9, respectively. Also, the vertex (i=7, 8, 9) is associated only with bone C, and its weight value is 1.0.

[0070] In contrast to this, as shown in the example of FIG. 11, in the Low skeleton information 121-2, bone C is omitted and bones A and B (bones that replace bones B and C in the example of FIG. 15) are described.

[0071] Example description 132 in FIG. 12 shows an example of the skin weight information 123-2 in this case. In the case of the skin weight information 123-2, as shown in example description 132, the correspondence between vertex i and bone A and the weight value thereof are shown as a list of vertices. In other words, each vertex that was assigned to bone A through bone C in example description 131 (FIG. 10) has been reassigned to either bone A or bone B. For example, vertex (i=0,1,15,16) is assigned only to bone A, and its weight value is 1.0. Vertex (i=2,14) is assigned to bone A and bone B, and its weight values ​​are 0.9 and 0.1, respectively. Vertex (i=3,13) is assigned to bone A and bone B, and its weight values ​​are 0.1 and 0.9, respectively. Also, vertices (i=4, 5, 6, 7, 8, 9, 10, 11, 12) are associated only with bone B, and their weight value is 1.0. In other words, the correspondence between bones and vertices has been updated. At that time, the weight value is also updated as necessary.

[0072] From these variations, as described above, one is selected according to the control of the resolution of the still 3D model and the resolution of the skeleton information. For example, in FIG. 7, when high skeleton information 121-1 is applied, skin weight information 123-1 is applied. When low skeleton information 121-2 is applied, skin weight information 123-2 is applied. Also, in FIG. 8, when high skeleton information 121-1 and a high still 3D model 122-1 are applied, skin weight information 123-11 is applied. When low skeleton information 121-2 and a high still 3D model 122-1 are applied, skin weight information 123-12 is applied. When high skeleton information 121-1 and a low still 3D model 122-2 are applied, skin weight information 123-13 is applied. When low skeleton information 121-2 and a low still 3D model 122-2 are applied, skin weight information 123-14 is applied.

[0073] In this way, the skin weight information can accurately indicate the correspondence between the appearance and the motion structure, regardless of the resolution applied to the static 3D model or the skeleton information. If the correspondence between the appearance and the motion structure indicated by the skin weight information is inaccurate, the quality of the 3D model may be reduced, such as by distorting the three-dimensional shape or causing unnatural movement when the 3D model is reconstructed. This may also reduce the subjective quality of the displayed image (rendered image), resulting in a reduced user experience. In particular, when the resolution of the static 3D model and the resolution of the skeleton information are controlled independently, the correspondence between the appearance and the motion structure becomes more diversified, making the content of the skin weight information more likely to be inaccurate. This may further reduce the quality of the 3D model and the quality of the user experience. Therefore, as described above, by suppressing the reduction in the quality of the skin weight information and more accurately indicating the correspondence between the appearance and the motion structure, it is possible to suppress the reduction in the quality of the 3D model, suppress the reduction in the subjective quality of the displayed image (rendered image), and suppress the reduction in the quality of the user experience. In other words, by controlling the content of the skin weight information in addition to controlling the resolution of the static 3D model and skeleton information, it is possible to suppress an increase in the load of the playback process while suppressing a decrease in the quality of the user experience.

[0074] <Method 1-1> When the above-described Method 1 is applied, the resolution of the still 3D model and the resolution of the skeleton information may be controlled based on different parameters (Method 1-1), as shown in the second row from the top of the table in Fig. 6. In other words, the resolution of the still 3D model and the resolution of the skeleton information may be controlled independently of each other, and may be controlled based on different parameters.

[0075] For example, in the first information processing device described above, the still 3D model resolution setting unit may set the resolution of the still 3D model based on a first parameter, and the skeleton information resolution setting unit may set the resolution of the skeleton information based on a second parameter different from the first parameter.

[0076] By controlling based on mutually independent parameters in this way, the resolution of the static 3D model and the resolution of the skeleton information can be controlled independently of each other.

[0077] For example, the second parameter may be a parameter of a different type from the first parameter. That is, the level of definition of the static 3D model and the level of definition of the skeleton information may be controlled based on parameters of different types.

[0078] For example, the first parameter may be a parameter relating to the distance from the viewpoint position to the 3D model, and the second parameter may be a parameter relating to the content that is different from the distance. That is, the resolution of the static 3D model may be controlled based on a parameter relating to the distance from the viewpoint position (the position of the 3D model corresponding to the viewpoint user) to the 3D model to be processed. Also, the resolution of the skeleton information may be controlled based on a parameter relating to the content including the 3D model.

[0079] The distance-related parameter may be any information related to the distance from the viewpoint to the 3D model. For example, the parameter may directly or indirectly indicate the distance (absolute value) from the viewpoint to the 3D model. The parameter may also indicate a direction. The parameter may indicate the distance from the viewpoint to a specific portion of the 3D model. The parameter may indicate a change in the distance from the viewpoint to the 3D model. The parameter may also indicate a statistical value (e.g., an average value, a median, etc.) related to the distance from the viewpoint to the 3D model. Of course, the distance-related parameter may indicate information other than these examples. Multiple parameters may be applied as the distance-related parameter.

[0080] The content-related parameter may be any information related to the content. For example, the parameter may indicate a type of scene or the importance of a scene. The parameter may indicate a type of 3D model or the importance of a 3D model. The parameter may indicate a position of a 3D model or the orientation of a 3D model. The parameter may indicate a shape of a 3D model, a size of a 3D model, or a color or pattern of a 3D model. The parameter may indicate what a 3D model is doing (how it moves). The parameter may indicate the number of 3D models being played simultaneously or the degree of overlap of 3D models in a displayed image. The parameter may indicate a current relationship with a viewpoint user or a past relationship with a viewpoint user. The parameter may be a statistical value (e.g., an average value, a median value, etc.) of some information including the examples described above. Of course, the content-related parameter may indicate information other than these examples. Furthermore, a plurality of parameters may be applied as parameters relating to this content.

[0081] In this way, by controlling the resolution of the skeleton information based on content-related parameters, it is possible to control the resolution of the skeleton information according to the content situation, regardless of the distance from the viewpoint position. For example, the resolution of the skeleton information of a 3D model can be reduced in scenes where a reduction in the resolution of the motion structure is less noticeable. Generally, when the resolution of the motion structure is reduced, the resolution of the motion of the 3D model also decreases. Therefore, reducing the resolution of the motion structure may cause the motion of the 3D model to become unnatural (reduced realism), which may in some cases lead to a reduction in the quality of the user experience. However, as described above, by switching the resolution of the skeleton information according to the content situation, it is possible to minimize the unnaturalness of such object motion. For example, the resolution of the skeleton information of a 3D model in situations where a reduction in the resolution of the motion structure is less noticeable, such as a 3D model with little movement, a 3D model located in a position difficult for the viewpoint user to see, or a 3D model with little relationship to the viewpoint user, can be reduced. This control allows the resolution of skeleton information to be reduced even for 3D models located close to the viewpoint. In this case, the skeleton information is controlled independently of the static 3D model, so the increase in the load of the playback process can be suppressed while suppressing a decrease in the quality of the user experience.

[0082] For example, if the skinned model is an avatar, the resolution of the motion structure (LoD of skeleton information) may be controlled based on whether the avatar is currently interacting with the viewpoint user. For example, if the avatar is currently interacting with the viewpoint user, it is expected that the viewpoint user will pay close attention to the avatar. Therefore, if the resolution of the motion structure of such an avatar is reduced, the reduction in the realism of the avatar will be noticeable, and the viewpoint user will likely feel uncomfortable. In other words, it is expected that the quality of the user experience will likely be reduced. Therefore, the resolution of the motion structure is increased for avatars that are currently interacting with the viewpoint user, and reduced for avatars that are not. By controlling in this manner, it is possible to suppress an increase in the load of the playback process while suppressing a reduction in the quality of the user experience.

[0083] Furthermore, the resolution of the motion structure (LoD of skeleton information) may be controlled based on whether the avatar has previously interacted with the viewpoint user (e.g., within a predetermined time period). For example, if the avatar is an avatar that has previously interacted with the viewpoint user, it is likely that the viewpoint user will still be paying close attention to the avatar. Therefore, if the resolution of the motion structure of such an avatar is reduced, the reduction in the realism of the avatar will be noticeable, and the viewpoint user will likely feel uncomfortable. In other words, it is likely that the quality of the user experience will be reduced. Therefore, the resolution of the motion structure is increased for avatars that have previously interacted with the viewpoint user, and reduced for avatars that have not. By controlling in this manner, it is possible to suppress an increase in the load of the playback process while suppressing a reduction in the quality of the user experience.

[0084] Alternatively, the first parameter may be a parameter relating to the distance from the viewpoint position to the 3D model, and the second parameter may be a parameter relating to the load of the content playback process. That is, the resolution of the still 3D model may be controlled based on a parameter relating to the distance from the viewpoint position (the position of the 3D model corresponding to the viewpoint user) to the 3D model to be processed. Also, the resolution of the skeleton information may be controlled based on a parameter relating to the load of the playback process.

[0085] The parameter related to the load of the regeneration process may be any information related to the load of the regeneration process. For example, the parameter may directly or indirectly indicate the load of the regeneration process. Furthermore, the parameter may indicate the current load of the regeneration process, the past load of the regeneration process, or the future load of the regeneration process (i.e., a predicted value). Furthermore, the parameter may be a statistical value related to the load of the regeneration process (e.g., an average value, a median value, etc.). Of course, the parameter related to the load of the regeneration process may indicate information other than these examples. Furthermore, multiple parameters may be applied as the parameter related to the load of the regeneration process.

[0086] In this way, by controlling the resolution of the skeleton information based on parameters related to the load of the playback process, it is possible to control the resolution of the skeleton information according to the load status of the playback process, regardless of the distance from the viewpoint position. For example, when the load of the playback process increases and a delay is likely to occur, the resolution of the skeleton information of the 3D model can be reduced as an emergency measure. If a delay occurs in the playback process, for example, the displayed image (moving image) may stop or frames may be lost, which may significantly reduce the quality of the user experience. By reducing the resolution of the movement structure of the 3D model, the increase in the load of the playback process can be suppressed, thereby preventing such situations from occurring. Compared to a delay in the playback process, reducing the resolution of the movement structure of the 3D model generally reduces the quality of the user experience less. In other words, by controlling the resolution of the skeleton information based on parameters related to the load of the playback process, it is possible to suppress an increase in the load of the playback process while suppressing a reduction in the quality of the user experience. By controlling the resolution of the skeleton information based on parameters related to the load of the playback process, it is possible to reduce the resolution of the skeleton information even for 3D models located close to the viewpoint, thereby suppressing an increase in the load of the playback process. Furthermore, because the skeleton information is controlled independently of the static 3D model, suppressing a reduction in the resolution of the static 3D model can suppress a reduction in the quality of the user experience.

[0087] The resolution of the skeleton information may also be adaptively controlled depending on the load of the playback process. For example, the higher the current load of the playback process of content including a Skinned model (the smaller the resource load margin), the lower the resolution of the motion structure. Alternatively, the lower the current load of the playback process (the larger the resource load margin), the higher the resolution of the motion structure. Alternatively, both may be performed. For example, a range of load magnitude (resource margin) may be predetermined for each hierarchical level of resolution of the motion structure, and the hierarchical level corresponding to the current load magnitude may be set as the resolution of the motion structure. Alternatively, the hierarchical level of resolution of the motion structure to be applied may be derived from the load magnitude (resource margin) by a predetermined calculation.

[0088] Furthermore, the second parameter may be a parameter having a different value from the first parameter. That is, the resolution of the still 3D model and the resolution of the skeleton information may be controlled based on parameters having different values. That is, the first parameter and the second parameter may be the same type of parameter, as long as their values ​​are different. For example, the resolution of the still 3D model and the resolution of the skeleton information may be controlled by parameters relating to the distance from the viewpoint to different parts of the 3D model. Alternatively, one may be controlled by the absolute value of the distance from the viewpoint to the 3D model, and the other may be controlled by a signed distance from the viewpoint to the 3D model. Furthermore, the quantization units of the distance may be different. Alternatively, one may be controlled by a parameter that directly indicates the distance from the viewpoint to the 3D model, and the other may be controlled by a parameter that indirectly indicates the distance from the viewpoint to the 3D model. In this way, even when controlled based on parameters having different values, the resolution of the still 3D model and the resolution of the skeleton information can be controlled independently of each other. This parameter may be any information, and may of course be other than the distance described above.

[0089] Furthermore, the second parameter may be a parameter at a different timing than the first parameter. That is, the resolution of the still 3D model and the resolution of the skeleton information may be controlled based on parameters at different timings. That is, the first parameter and the second parameter may be the same type of parameter as long as they are acquired at different times. For example, both the resolution of the still 3D model and the resolution of the skeleton information may be controlled at different times by a parameter related to the distance from the viewpoint to the 3D model. "Different times" refers to the timing at which the parameters are acquired (e.g., the timing at which the distance is measured). Of course, the timing at which control is performed may also be different. For example, the distance from the viewpoint to the 3D model may be measured at times A, B, C, and D, and the resolution of the still 3D model may be controlled at times A and C based on the distance measured at those times, and the resolution of the skeleton information may be controlled at times B and D based on the distance measured at those times. Alternatively, for example, the resolution of the static 3D model may be controlled based on the current distance, and the resolution of the skeleton information may be controlled based on the past distance. In this way, even when control is based on parameters with different timings, the resolution of the static 3D model and the resolution of the skeleton information can be controlled independently. Note that this parameter may be any information, and may of course be other than the distance described above.

[0090] <Load of Playback Processing> Next, we will explain the impact on the load of playback processing due to control of the resolution of the still 3D model (resolution of appearance) and the resolution of the skeleton information (resolution of movement structure). First, we will explain the load of playback processing. In animation processing, two processes, "skeleton posture transformation processing" and "skinning processing", are performed in this order.

[0091] The skeleton posture transformation process calculates a 4x4 posture transformation matrix that represents the current posture of each bone. In other words, this process (calculation) is performed as many times as there are bones. The skinning process moves each vertex of the mesh by performing a matrix operation on the posture transformation matrix of the bone to which the vertex is linked. In other words, the process (calculation) is performed as many times as there are vertices.

[0092] Therefore, the load of the skinning process depends on the number of vertices in the mesh, but not on the number of bones in the skeleton. In contrast, the load of the skeleton posture transformation process depends on the resolution of the motion structure (number of bones).

[0093] For example, as shown in FIG. 13 , let us say that the initial posture of Bone_1 is posture 141. Let us say that the initial posture of Bone_2 is posture 142. Let us say that the initial posture of Bone_(i-1) is posture 143. Let us say that the initial posture of Bone_i is posture 144. In other words, let us say that Bone_1 to Bone_i are facing the same direction and connected in series in their initial postures. Also, let us say that mesh vertices P1 to P4 are associated with Bone_i based on skin weight information. Let us say that the initial position of vertex P1 is P1_(before, local), the initial position of vertex P2 is P2_(before, local), the initial position of vertex P3 is P3_(before, local), and the initial position of vertex P4 is P4_(before, local).

[0094] Assume that the postures of Bone_1 to Bone_i are changed by animation processing as shown in the example of Fig. 13. That is, the posture of Bone_1 is changed from posture 141 to posture 141A. The posture of Bone_2 is changed from posture 142 to posture 142A. The posture of Bone_(i-1) is changed from posture 143 to posture 143A. The posture of Bone_i is changed from posture 144 to posture 144A.

[0095] In this case, the motion matrix (posture transformation matrix) showing the movement (posture change) of Bone_1 is defined as M_1. The motion matrix showing the movement of Bone_2 is defined as M_2. The motion matrix showing the movement of Bone_(i-1) is defined as M_(i-1). The motion matrix showing the movement of Bone_i is defined as M_i. These motion matrices M (M_1 to M_i) show the change in posture of the corresponding bone based on the posture of its parent bone. Therefore, since Bone_1 is a root bone with no parent bone, M_1 shows the difference between posture 141 and posture 141A, while M_2 shows the difference between posture 141A and posture 142A. Similarly, M_(i-1) shows the difference between posture 143A and posture 144A of Bone_(i-2) after movement. M_i shows the difference between posture 144A and posture 143A.

[0096] <Load of skeleton posture deformation processing> As shown in a box 151 in Fig. 14, the skeleton posture deformation processing for Bone_1 to Bone_i in Fig. 13 can be expressed by the following formula (1). Note that within the box 151, characters with "_" may be shown as subscripts.

[0097] M_(pose_i) = M_(boneOffset_i) × M_i × M_(i-1) × ... × M_2 ×M_1 ... (1)

[0098] In equation (1), M_(pose_i) is a motion matrix that indicates the movement of Bone_i due to this animation process, i.e., the difference between posture 144 and posture 144A. M_(boneOffset_i) is the bone offset matrix of Bone_i. In other words, M_(boneOffset_i) indicates the inverse matrix of the initial posture matrix of Bone_i. As shown in FIG. 13, the posture of Bone_i also changes depending on the movement of its ancestor bones. Therefore, as shown in equation (1), the derivation of M_(pose_i) must reflect not only the motion matrix of the bone (Bone_i) but also the motion matrices of all ancestor bones (root bone to parent bones).

[0099] Such skeleton posture deformation processing (the calculation of equation (1)) is executed for each of Bone_1 to Bone_i. These calculations constitute a load for the skeleton posture deformation processing.

[0100] <Load of Skinning Processing> As shown in a box 152 in Fig. 14, the skinning processing for vertices P1 to P4 in Fig. 13 can be expressed by the following equations (2) to (5). Note that within the box 152, characters with "_" may be shown as subscripts.

[0101] P1_(after,local) = P1_(before,local) × M_(pose_i) ・・・(2) P2_(after,local) = P2_(before,local) × M_(pose_i) ・・・(3) P3_(after,local) = P3_(before,local) × M_(pose_i) ・・・(4) P4_(after,local) = P4_(before,local) × M_(pose_i) ...(5)

[0102] In equations (2) to (5), P1_(after, local) indicates the position of vertex P1 after a pose change (after movement by animation processing). P2_(after, local) indicates the position of vertex P2 after a pose change (after movement by animation processing). P3_(after, local) indicates the position of vertex P3 after a pose change (after movement by animation processing). P4_(after, local) indicates the position of vertex P4 after a pose change (after movement by animation processing). Note that in equations (2) to (5), M_(pose_i) is applied as the motion matrix, and the motion matrix of the bone corresponding to the vertex being processed is applied to this motion matrix. As in equations (2) to (5), calculations are performed for each vertex to derive the position of the vertex after movement. These calculations add load to the skinning process.

[0103] <Load Comparison> Assume that the animation process shown in Fig. 13 is performed with the highest resolution for the motion structure (and motion). In this case, the calculation (Equation (1)) for the skeleton posture transformation process described above is performed for all bones (Bone_1 to Bone_i). In other words, the calculations shown in the box 153 in Fig. 15 are performed.

[0104] In contrast, let us assume that the animation process shown in FIG. 13 is performed by reducing the resolution of the movement, as in the method described in Non-Patent Document 1. For example, let us assume that the movement (M_i) of Bone_i is omitted from the motion data. In this case, as shown in FIG. 16, Bone_i changes from posture 144 to posture 144B through this animation process. In other words, the orientation of Bone_i after the movement will be the same as that of Bone_(i-1).

[0105] Therefore, in the calculation of the skeleton posture deformation process, the motion matrix M_i can be omitted in the calculation to derive M_(pose_i), as shown in the box 154 in Fig. 17. Therefore, the amount of calculation can be reduced compared to the example shown in the box 153 in Fig. 15 (when the motion resolution is the highest resolution). In other words, the load of the skeleton posture deformation process can be reduced.

[0106] In contrast, suppose the above-described technology is applied to reduce the resolution of the motion structure and perform the animation process of Fig. 13. For example, suppose Bone_i is omitted from the skeleton information (Bone_(i_1) and Bone_i are treated as one bone). That is, as shown in Fig. 18, the initial posture of Bone_(i-1) is assumed to be posture 161. Then, suppose Bone_(i-1) changes from posture 161 to posture 161A through this animation process.

[0107] In this case, in the calculation of the skeleton posture transformation process, the calculation to derive M_(pose_i) can be omitted entirely, as shown in the box 162 in Fig. 19. Therefore, the amount of calculation can be further reduced compared to the example shown in the box 154 in Fig. 17 (when only the resolution of the movement is reduced). In other words, the load of the skeleton posture transformation process can be further reduced.

[0108] Even if bone movement is omitted or the number of bones is reduced, each vertex of the mesh is linked to one of the bones using skin weight information. Therefore, the load of the skinning process does not depend on the bone movement or number of bones. In other words, the load of the skinning process is approximately the same in each of the above examples.

[0109] In other words, by applying this technology, it is possible to suppress an increase in the load of the reproduction process compared to when the method described in Non-Patent Document 1 is applied.

[0110] <Method 1-2> The content to be played may include motion data in addition to the Skinned model. For example, in the first information processing device described above, the content playback unit may play content that further includes motion data indicating the movement of the 3D model.

[0111] In this case, the resolution of the motion data (resolution of the movement of the 3D model) may be variable and controlled. In other words, when the above-mentioned method 1 is applied, the resolution of the motion data may be further controlled (method 1-2), as shown in the third row from the top of the table in Figure 6. In other words, the resolution of the motion data may be controlled together with the above-mentioned static 3D model, skeleton information, and skin weight information.

[0112] For example, the motion data may have variations in the resolution of the movement, and one of these may be applied. For example, in FIGS. 7 and 8, high motion data 112-1 and low motion data 112-2 may be prepared, and either one of them may be applied. In other words, the motion data may be hierarchically organized according to the resolution of the movement. For example, multiple motion data with different resolutions, such as high-resolution motion data and low-resolution motion data, may be prepared as candidates, and the desired motion data with the desired resolution may be selected and applied from these candidates. In this specification, this hierarchy is also referred to as LoD. Note that the number of variations of the motion data (LoD number) may be any number equal to or greater than two. Of course, it is also possible to generate and provide motion data with a freely set resolution without preparing variations (multiple candidates).

[0113] As shown in Fig. 4, the motion data indicates the movement of a 3D model based on skeleton information (bone structure). That is, as shown in Fig. 7 and Fig. 8, the resolution of the motion data (i.e., the resolution of the movement of the 3D model) depends on the resolution of the skeleton information (resolution of the movement structure). For example, the resolution of the motion data may correspond to the resolution of the skeleton information.

[0114] 7 and 8, for example, high motion data 112-1 indicates the movement of the bone structure of high skeleton information 121-1. In other words, high motion data 112-1 corresponds to high skeleton information 121-1. Also, low motion data 112-2 indicates the movement of the bone structure of low skeleton information 121-2. In other words, low motion data 112-2 corresponds to low skeleton information 121-2. Therefore, for example, the high motion data 112-1 may be applied when high skeleton information 121-1 is applied, and the low motion data 112-2 may be applied when low skeleton information 121-2 is applied. In other words, the high skeleton information 121-1 may be applied when high motion data 112-1 is applied, and the low skeleton information 121-2 may be applied when low motion data 112-2 is applied.

[0115] In this way, by controlling the resolution of the movement in accordance with the control of the resolution of the movement structure, it is possible to suppress an increase in the load of the playback process while suppressing a reduction in the quality of the user experience, regardless of whether High skeleton information 121-1 or Low skeleton information 121-2 is applied.

[0116] <Content Distribution> Content including the above-described Skinned model and motion data may be distributed. That is, this content may be provided from one device to another. For example, a distribution server may distribute this content, and a playback device may acquire and play the content.

[0117] For example, the first information processing device may further include a content acquisition unit that acquires a still 3D model, skeleton information, motion data, and skin weight information as content, and the content playback unit may play back the acquired content.

[0118] The present technology may also be applied when content is distributed in this manner. That is, the resolution of the still 3D model and the resolution of the skeleton information may be controlled independently of each other. For example, a playback device that plays back content may set the resolution of the still 3D model and the resolution of the skeleton information independently of each other and request a still 3D model, skeleton information, and skin weight information according to the set resolution. A distribution device that distributes content may acquire the request and provide the requested content (the requested still 3D model, skeleton information, and skin weight information) to the playback device. The playback device may then acquire and play the provided content. By doing so, as described above, it is possible to suppress an increase in the load of the playback process while suppressing a decrease in the quality of the user experience.

[0119] Note that such distribution with control of resolution may be performed using control information. For example, this control information may be provided to a playback device. This control information indicates the specifications of the content that can be distributed, the location and identification information of the content, etc. Based on this information, the playback device can set the resolution of the still 3D model or skeleton information, or request still 3D model or skeleton information with the set resolution.

[0120] For example, the first information processing device may further include a control information acquisition unit that acquires control information for controlling the supply of content. The still 3D model resolution setting unit may then set the resolution of the still 3D model based on the control information. Furthermore, the skeleton information resolution setting unit may set the resolution of the skeleton information based on the control information. In this case, the resolution of the skeleton information is set independently of the resolution of the still 3D model. While details of the control information will be described later, by utilizing this control information, the playback device (first information processing device) can more easily set the resolution of the still 3D model and the skeleton information. Furthermore, the playback device (first information processing device) can easily select skin weight information and motion data based on the control information.

[0121] As described above, this control information can also be used to acquire (request) content. For example, the first information processing device may further include a content acquisition unit that acquires, as content, still 3D models and skeleton information with resolutions set independently of each other, as well as skin weight information corresponding to the still 3D models and skeleton information, based on the control information. The content playback unit may then play the acquired content. Note that this content may include motion data. In other words, the motion data may also be acquired based on the control information. By using the control information in this way, the playback device (first information processing device) can more easily acquire content (such as still 3D models, skeleton information, skin weight information, and motion data).

[0122] <Control Information> The content of the control information is arbitrary. For example, a data format with the following three points may be used. Point 1: Because there is a one-to-one correspondence between skeleton information and motion data, the data structure should also be integrated. Also, selecting the LoD for one of them may determine the LoD for the other. Point 2: The data structure should allow each of the three types of data - skeleton information and motion data, static 3D model, and skin weight information - to have individual variations. Point 3: Based on the LOD of the static 3D model and the LOD of the skeleton information (or the LOD of the motion data), each of the three types of data - skeleton information and motion data, static 3D model, and skin weight information - should have linking data that determines which variation should be obtained.

[0123] <Method 1-3> This control information may be in any format. For example, it may be written in glTF 2.0. That is, when the above-described method 1 is applied, content distribution may be controlled using glTF 2.0 (Method 1-3), as shown in the fourth row from the top of the table in FIG. 6 . The glTF 2.0 file may be composed of a scene description file written in JSON, binary files that constitute 3D objects and animation data referenced in the glTF file, and still image data such as png and jpg that are used as textures for 3D objects.

[0124] In glTF2.0's existing functionality for handling skinned models, motion data is uniquely determined once the skeleton is determined, satisfying point 1 above. However, the skeleton information, static 3D model, and skin weight information are all integrated, making it impossible to express individual variations. Therefore, point 2 above is not satisfied.

[0125] Therefore, a glTF2.0 function called MSFT_lod, which expresses LOD variations of mesh data (static 3D models), may be used. Syntax 201 shown in Figure 20, syntax 202 shown in Figure 21, and syntax 203 shown in Figure 22 are examples of syntaxes expanded using this function to satisfy this point. For convenience of explanation, the syntaxes are explained separately in Figures 21 and 22, with syntax 202 following syntax 201 and syntax 203 following syntax 202.

[0126] As shown in syntax 201, the LoD representation of meshes (static 3D models) uses the conventional MSFT_lod extension. In contrast, the LoD representation of skeletal information uses a new extension, "Skeleton_lod." In this example, "Skeleton_lod" is used in the same way as "MSFT_lod." The node containing this "Skeleton_lod" indicates skeletal information of LoD0 (H-LOD). Lower LoDs are represented by nodes indicated by "Skeletal_ids" (array of "nodes" index).

[0127] Also, as shown in syntax 203, skin weight information is expressed in the "SkinWeights" node extended using a new extension, together with the "nodes" index information ("mesh_lod_node", "skin_lod_node") that indicates the LoD of the mesh (static 3D model) to which it corresponds and the LoD of the Skin (skeleton information).

[0128] By describing the control information in this way, the above points 1 to 3 can be satisfied, and an increase in the load of the playback process can be suppressed while suppressing a decrease in the quality of the user experience.

[0129] <Method 1-4> This control information may also be an MPD (Media Presentation Description) used when distributing content data using MPEG-DASH (Moving Picture Experts Group Dynamic Adaptive Streaming over HTTP (Hypertext Transfer Protocol)). In other words, when the above-mentioned method 1 is applied, the resolution of the motion structure and the like may be controlled using an MPD (Method 1-4), as shown in the fifth row from the top of the table in Fig. 6. The MPD is control information for distributing content data.

[0130] In this case, the content data, including the static 3D model, skeleton information, skin weight information, and motion data, are packaged as DASH segment files. The static 3D model, skeleton information, and skin weight information are static data that do not change over time. In contrast, motion data is dynamic data that can change over time. In this specification, characteristics that do not change over time are also referred to as "static." Furthermore, in this specification, characteristics that can change over time are also referred to as "dynamic."

[0131] Therefore, these data are packaged, for example, as shown in FIG. 23. FIG. 23 is a diagram showing an example of the structure of an MP4 file. As shown in FIG. 23, skeleton information and motion data are stored in a motion data track (MotionData Track) 211. The motion data track (MotionData Track) 211 is a timed metadata track ("Timed Metadata Track") that stores dynamic data. Motion data, which is dynamic data, is stored in the media data box (MediaDataBox('mdat')) of this motion data track 211. Furthermore, skeleton information (static data) that corresponds one-to-one to this motion data is stored as a sample entry (MotionSampleEntry) in the movie box (MovieBox('moov')) of this motion data track (MotionData Track) 211.

[0132] A skin weight item (SkinWeightItem) 212 is a metabox (MetaBox) that stores meta items. Skin weight information, which is static data, is stored in the media data box of this skin weight item 212. A 3D model data item (3D Model Data Item) 213 is a metabox (MetaBox) that stores meta items. A static 3D model, which is static data, is stored in the media data box of this 3D model data item 213.

[0133] As mentioned in point 2 above, the skeleton information and motion data, the static 3D model, and the skin weight information are stored in different files so that these three types of data can have variations independent of each other. That is, as mentioned above, the skin weight information is stored in a skin weight item 212 that is different from the motion data track 211. The static 3D model is stored in a 3D model data item 213 that is different from the motion data track 211. Note that each variation of these three types of data is also stored in a different file.

[0134] Fig. 24 shows an example of an MPD description in which skeleton information, motion data, still 3D models, and skin weight information are stored in different files for each variation. In the description example 221 shown in Fig. 24, the files storing the variations of skeleton information, motion data, still 3D models, and skin weight information are referenced by an adaptation set (AdaptationSet). In this description example, each adaptation set is linked by the linking data described in point 3 above. In this description example 221, two variations (H-LoD and L-LoD) are prepared for both the skeleton information and the still 3D models.

[0135] Preselection links adaptation sets that reference skeleton information and motion data, adaptation sets that reference static 3D models, and adaptation sets that reference skin weight information. Also, as mentioned in point 3 above, preselection uses essential properties to indicate new descriptors that specify data corresponding to two LoD values: the LoD of the static 3D model and the LoD of the skeleton information (= LoD of the motion data).

[0136] By describing the control information in this way, the above points 1 to 3 can be satisfied, and an increase in the load of the playback process can be suppressed while suppressing a decrease in the quality of the user experience.

[0137] <Method 1-5> This control information may also be a Universal Scene Description (USD). That is, when the above-described Method 1 is applied, the resolution of the motion structure may be controlled using USD (Method 1-4), as shown in the bottom row of the table in Fig. 6. USD is a scene description for describing a scene in a three-dimensional space.

[0138] USD has a mechanism for handling skinned models called SkelRoot prims and a mechanism for handling LoD. With previous USD functions for handling skinned models, motion data was uniquely determined once the skeleton was determined. Therefore, point 1 above was met. However, the mesh (static 3D model) and skin weight information were integrated, making it impossible to express individual variations. Therefore, point 2 above was not met. It is possible to configure skeleton information so that it can be independently switched by using a unique variantSet prim for the skeleton information. However, because the skin weight information and the static 3D model were explicitly linked one-to-one, there was a risk that the intended motion could not be reproduced.

[0139] Therefore, control information may be written as shown in the examples in Figures 25 to 27. Example 231 in Figure 25, example 232 in Figure 26, and example 233 in Figure 27 are examples of USD descriptions. For convenience of explanation, the descriptions are divided into Figures 25 to 27, with example 232 following example 231 and example 233 following example 232. In these examples, the skin weight information embedded in the "jointIndices" and "jointWeights" nodes, which are attributes within the "Mesh" prim, is extracted, and a new "SkinWeight" prim is defined to handle the skin weight information separately. This prim has two string-type attributes, "mesh_LOD" and "skel_LOD," which represent the variant names of the "Mesh" prim and "Skeleton" prim referenced by its own skin weight information, and this information is used to link the correct combination. By using variantSet for this "SkinWeight" prim, it is possible to express individual variations.

[0140] By describing the control information in this way, the above points 1 to 3 can be satisfied, and an increase in the load of the playback process can be suppressed while suppressing a decrease in the quality of the user experience.

[0141] <Provision of Control Information by Distribution Device> This control information may be provided by, for example, a distribution device that distributes content. For example, a second information processing device may include: a control information supply unit that supplies control information for controlling the supply of content including a still 3D model that indicates the appearance of a 3D model, skeleton information that indicates a motion structure of the 3D model, and skin weight information that indicates a correspondence between the appearance and the motion structure; and a content supply unit that supplies content including a still 3D model of a level of resolution requested based on the control information, skeleton information of a level of resolution requested based on the control information, and skin weight information requested based on the control information.

[0142] In addition, in a second information processing method executed by a second information processing device, control information may be supplied that controls the supply of content including a still 3D model that indicates the appearance of the 3D model, skeletal information that indicates the motion structure of the 3D model, and skin weight information that indicates the correspondence between the appearance and the motion structure, and content may be supplied that includes a still 3D model of a level of resolution requested based on the control information, skeletal information of a level of resolution requested based on the control information, and skin weight information requested based on the control information.

[0143] Note that this control information may be information that expresses variations in the level of detail of the still 3D model, variations in the level of detail of the skeleton information, and variations in the skin weight information independently of one another.

[0144] By providing such control information, the playback device (first information processing device) can set the resolution of the still 3D model and skeleton information, select skin weight information, and acquire content including them based on the control information, as described above. In other words, the distribution device (second information processing device) can suppress an increase in the playback processing load while suppressing a decrease in the quality of the user experience.

[0145] Note that this control information may be prepared in advance or may be generated. For example, the second information processing device may further include a control information generating unit that generates control information based on the content. Then, the control information supplying unit may supply the generated control information. In this manner, the distribution device (second information processing device) can provide control information corresponding to the content to be distributed even if the control information to be provided is not prepared.

[0146] The content to be distributed may be prepared in advance or may be generated. For example, the still 3D model may be prepared in advance or may be generated. For example, the second information processing device may further include a still 3D model generation unit that generates variations in the resolution of the still 3D model. The control information generation unit may then generate control information that represents variations in the resolution of the generated still 3D model. In this way, the distribution device (second information processing device) can generate and provide a still 3D model even if a still 3D model is not prepared.

[0147] Any method may be used to generate this still 3D model. For example, a still 3D model may be generated by capturing a real object. Also, variations in resolution may be generated from a single still 3D model. For example, in the second information processing device, a still 3D model generation unit may generate a still 3D model with lower resolution based on a still 3D model with the highest resolution. For example, the still 3D model generation unit may generate a still 3D model with lower resolution by thinning out information from the still 3D model with the highest resolution. For example, if the still 3D model is point cloud data, some points in the point cloud may be thinned out. Also, if the still 3D model is mesh data, the number of faces may be reduced by thinning out some vertices or connections. Using these methods, the distribution device (second information processing device) can easily generate variations in resolution of a still 3D model.

[0148] The skeleton information may be prepared in advance or may be generated. For example, the second information processing device may further include a skeleton information generation unit that generates variations in the level of detail of the skeleton information. The control information generation unit may then generate control information that represents variations in the level of detail of the generated skeleton information. In this manner, the distribution device (second information processing device) can generate and provide skeleton information even when skeleton information is not prepared.

[0149] Any method may be used to generate this skeleton information. For example, skeleton information may be generated by capturing a real object. Also, variations in the level of resolution may be generated from a single skeleton information. For example, in the second information processing device, the skeleton information generation unit may generate skeleton information with lower resolution based on skeleton information with the highest resolution. For example, the skeleton information generation unit may generate skeleton information with lower resolution by thinning out bones from the skeleton information with the highest resolution. In this case, the skeleton information generation unit may generate skeleton information for a layer one level lower by, for example, deleting the bones at the end. In other words, skeleton information for each layer may be generated by deleting bones starting from the end. Using this method, the distribution device (second information processing device) can easily generate variations in the level of resolution of skeleton information.

[0150] Furthermore, the skin weight information may be prepared in advance or may be generated. For example, the second information processing device may further include a skin weight information generation unit that generates variations in skin weight information based on variations in the resolution of the still 3D model and variations in the resolution of the skeleton information. The control information generation unit may then generate control information that represents the variations in the generated skin weight information. In this manner, the distribution device (second information processing device) can generate and provide skin weight information even if skin weight information is not prepared.

[0151] Any method may be used to generate this skin weight information. For example, variations may be generated from a single piece of skin weight information. For example, in the second information processing device, the skin weight information generation unit may generate skin weight information corresponding to a case where at least one of the still 3D model and the skeleton information has lower resolution, based on skin weight information corresponding to a case where the still 3D model and the skeleton information have the highest resolution. Furthermore, variations in the skin weight information may be generated based on variations in the resolution of the still 3D model and the skeleton information. For example, variations in the skin weight information may be generated in accordance with a method of reducing the information of each LoD of the still 3D model and the skeleton information. Using this method, the distribution device (second information processing device) can easily generate variations in the resolution of the skin weight information.

[0152] Motion data may also be distributed as content. That is, the distributed content may include motion data. In this case, the distribution of the motion data may be controlled by the control information described above (Method 1-2). For example, the control information may further represent variations in the resolution of the motion data indicating the movement of the 3D model. Then, in the second information processing device, a content supply unit may supply content including motion data with a resolution requested based on the control information. By doing so, the playback device (first information processing device) can set the resolution of the motion data based on the control information as described above and acquire content including that motion data. That is, the distribution device (second information processing device) can suppress an increase in the playback processing load while suppressing a decrease in the quality of the user experience.

[0153] The motion data may be prepared in advance or may be generated. For example, the second information processing device may further include a motion data generation unit that generates variations in the resolution of the motion data. The control information generation unit may then generate control information that represents the variations in the resolution of the generated motion data. In this way, the distribution device (second information processing device) can generate and provide motion data even when the motion data is not prepared.

[0154] Any method may be used to generate this motion data. For example, a real object may be captured to generate motion data. Also, variations in resolution may be generated from a single piece of motion data. For example, in the second information processing device, the motion data generation unit may generate lower resolution motion data based on the highest resolution motion data. Also, variations in resolution of the motion data may be generated according to variations in skeleton information. With such a method, the distribution device (second information processing device) can easily generate variations in resolution of the motion data.

[0155] The control information generated by the control information generator may be in any format. For example, the control information may be written in glTF2.0 (Method 1-3). Alternatively, the control information may be MPD (Method 1-4). Alternatively, the control information may be Universal Scene Description (Method 1-5).

[0156] <Regarding the Method> In other words, each of the above-described methods may be applied not only to the first information processing device but also to the second information processing device. Each of the above-described methods may be applied in combination with other methods as long as no contradiction occurs. For example, any two or more of methods 1-1 to 1-5 shown in the table of FIG. 6 may be applied in combination. Of course, the method shown in FIG. 6 may be applied in combination with other methods.

[0157] In the description of this specification, a higher-level method may include a lower-level method. In other words, applying method 1 may include applying one or more of methods 1-1 to 1-5. Therefore, for example, when describing "applying method 1," this may include applying one or more of methods 1-1 to 1-5.

[0158] <4. First Embodiment> <Distribution Device> The present technology described above may be applied to any device. FIG. 28 is a block diagram showing an example of the configuration of a distribution device, which is one aspect of an information processing device to which the present technology is applied. The distribution device 300 shown in FIG. 28 is a device that encodes a 3D model (skinned model) of an object and its motion data and distributes the encoded data as content data to a playback device. The distribution device 300 also supplies control information to the playback device that controls the provision of skeleton information indicating the motion structure of the distributed 3D model in accordance with the resolution of the motion structure. Of course, the distribution device 300 may perform other processes.

[0159] Note that Fig. 28 shows the main processing units, data flows, and the like, and is not necessarily all that is shown in Fig. 28. In other words, distribution device 300 may have processing units that are not shown as blocks in Fig. 28, or processes or data flows that are not shown as arrows or the like in Fig. 28.

[0160] 28 , distribution device 300 has a control unit 301 and a distribution processing unit 302. Control unit 301 controls each processing unit of distribution processing unit 302. Distribution processing unit 302 is controlled by control unit 301 to perform processing related to the distribution of content data, etc. Of course, control unit 301 and distribution processing unit 302 may also perform other processing.

[0161] The distribution processing unit 302 has a data input unit 311, a content processing unit 312, a control information generation unit 313, a storage unit 314, and a supply unit 315. The content processing unit 312 has a pre-processing unit 321, an encoding unit 322, and a file generation unit 323. The pre-processing unit 321 has a still 3D model generation unit 331, a skeleton information generation unit 332, a skin weight information generation unit 333, and a motion data generation unit 334. The supply unit 315 has a control information supply unit 341 and a content supply unit 342.

[0162] The data input unit 311 performs processing related to acquisition of data to be supplied to the distribution device 300. For example, the data input unit 311 may acquire information such as skeleton information, a still 3D model, skin weight information, and motion data supplied from outside the distribution device 300. In this case, the data input unit 311 may acquire only the highest-resolution information (information of the highest LoD) for each piece of information. Alternatively, the data input unit 311 may acquire not only the highest-resolution information (information of the highest LoD) for each piece of information but also information of lower LoDs. Alternatively, the data input unit 311 may acquire other information supplied from outside the distribution device 300. The data input unit 311 may supply some or all of the acquired information to the preprocessing unit 321. Alternatively, the data input unit 311 may supply some or all of the acquired information to the file generation unit 323. Alternatively, the data input unit 311 may supply some or all of the acquired information to the control information generation unit 313. Of course, the data input unit 311 may perform other processing.

[0163] The content processing unit 312 performs processing related to content data (such as a skinned model or motion data). For example, the content processing unit 312 may acquire content data supplied from the data input unit 311. The content processing unit 312 may also preprocess and encode the content data and store it in a file (also referred to as a content file). The content processing unit 312 may also supply the content file to the control information generation unit 313. The content processing unit 312 may also supply the content file to the storage unit 314. Of course, the content processing unit 312 may also perform other processing.

[0164] The control information generator 313 performs processing related to the generation of control information related to the provision of content data. For example, the control information generator 313 may acquire information (e.g., content data, etc.) supplied from the data input unit 311. The control information generator 313 may also acquire a content file supplied from the content processing unit 312 (file generator 323). The control information generator 313 may also generate control information based on the acquired information. The control information generator 313 may also store the control information in a file (also referred to as a control file). The control information generator 313 may also supply the generated control file (including the control information) to the storage unit 314. Of course, the control information generator 313 may also perform other processing.

[0165] The storage unit 314 has any storage medium and performs processing related to the storage of information. For example, the storage unit 314 may acquire a content file supplied from the content processing unit 312 and store the content file using its own storage medium. The storage unit 314 may also acquire a control file (including control information) supplied from the control information generation unit 313 and store the control file using its own storage medium. The storage unit 314 may also read out the content file or control file stored in the storage medium and supply it to the supply unit 315. Of course, the storage unit 314 may also perform other processing, such as storing other information.

[0166] The supply unit 315 includes a communication unit and communicates with other devices using the communication unit to perform processing related to the supply (distribution) of control files, content files, and the like. For example, the supply unit 315 may acquire a control file supplied from the storage unit 314 and supply the control file to a playback device that plays the content file. The supply unit 315 may also acquire a request for a content file supplied from a playback device based on control information stored in the control file. The supply unit 315 may also read the requested content file from the storage unit 314 and supply it to the playback device that issued the request. For example, the supply unit 315 may read a content file containing LoD content data specified in the request (e.g., skeleton information, still 3D model, skin weight information, motion data, etc.) from the storage unit 314 and supply it to the playback device that issued the request. Of course, the supply unit 315 may also perform other processing, such as transmitting and receiving other information.

[0167] The preprocessing unit 321 of the content processing unit 312 executes processing related to preprocessing of content data (processing performed before encoding). For example, the preprocessing unit 321 may acquire content data (skeleton information, still 3D model, skin weight information, motion data, etc.) supplied from the data input unit 311. The preprocessing unit 321 may also perform preprocessing on the content data. The preprocessing unit 321 may also supply the preprocessed content data to the encoding unit 322. Of course, the preprocessing unit 321 may also perform other processing.

[0168] The encoding unit 322 executes processing related to encoding of content data. For example, the encoding unit 322 may acquire content data supplied from the preprocessing unit 321. The encoding unit 322 may also encode the content data using a predetermined encoding method to generate the encoded data. The encoding unit 322 may also supply the generated encoded data of the content data to the file generation unit 323. Of course, the encoding unit 322 may also perform other processing.

[0169] The file generation unit 323 performs processing related to the generation of a content file. For example, the file generation unit 323 may acquire coded data of the content data supplied from the encoding unit 322. The file generation unit 323 may also acquire information supplied from the data input unit 311. The file generation unit 323 may also store the acquired coded data, etc. in a content file. The file generation unit 323 may also supply the generated content file to the control information generation unit 313. The file generation unit 323 may also supply the content file to the storage unit 314. Of course, the file generation unit 323 may also perform other processing.

[0170] The still 3D model generation unit 331 performs processing related to the generation of a still 3D model. For example, the still 3D model generation unit 331 may generate a still 3D model. Alternatively, the still 3D model generation unit 331 may acquire a still 3D model supplied from the data input unit 311 and generate variations in the level of detail using the still 3D model. Alternatively, the still 3D model generation unit 331 may supply the generated still 3D model (variations in the level of detail of the still 3D model) to the encoding unit 322 as content data. Of course, the still 3D model generation unit 331 may also perform other processing.

[0171] The skeleton information generation unit 332 performs processing related to the generation of skeleton information. For example, the skeleton information generation unit 332 may generate skeleton information. Alternatively, the skeleton information generation unit 332 may acquire skeleton information supplied from the data input unit 311 and use the skeleton information to generate variations in the level of detail. Alternatively, the skeleton information generation unit 332 may supply the generated skeleton information (variations in the level of detail of the skeleton information) to the encoding unit 322 as content data. Of course, the skeleton information generation unit 332 may also perform other processing.

[0172] The skin weight information generation unit 333 performs processing related to the generation of skin weight information. For example, the skin weight information generation unit 333 may generate skin weight information. Alternatively, the skin weight information generation unit 333 may acquire skin weight information supplied from the data input unit 311 and use the skin weight information to generate variations. Alternatively, the skin weight information generation unit 333 may supply the generated skin weight information (variations of skin weight information) to the encoding unit 322 as content data. Of course, the skin weight information generation unit 333 may also perform other processing.

[0173] The motion data generation unit 334 performs processing related to the generation of motion data. For example, the motion data generation unit 334 may generate motion data. Alternatively, the motion data generation unit 334 may acquire motion data supplied from the data input unit 311 and use the motion data to generate variations in resolution. Alternatively, the motion data generation unit 334 may supply the generated motion data (variations in resolution of the motion data) to the encoding unit 322 as content data. Of course, the motion data generation unit 334 may also perform other processing.

[0174] The control information supply unit 341 performs processing related to the supply of control information. For example, the control information supply unit 341 may acquire a control file supplied from the storage unit 314 and supply the control file to a playback device that plays back the content file. Of course, the control information supply unit 341 may also perform other processing, such as transmitting or receiving other information.

[0175] The content supply unit 342 performs processing related to the supply of content (skeleton information, still 3D models, skin weight information, motion data, etc.). For example, the content supply unit 342 may acquire a request for a content file supplied from a playback device based on control information. The content supply unit 342 may also read the requested content file from the storage unit 314 and supply it to the playback device that issued the request. For example, the content supply unit 342 may read from the storage unit 314 a content file that stores LoD content data specified in the request (skeleton information, still 3D models, skin weight information, motion data, etc.) and supply it to the playback device that issued the request. Of course, the content supply unit 342 may also perform other processing, such as transmitting and receiving other information.

[0176] The present technology described above in <3. Control of resolution related to 3D model> may be applied to the distribution device 300 configured as above. For example, the above-described method 1 may be applied to this distribution device 300.

[0177] For example, the control information supply unit 341 may supply control information that controls the supply of content including a still 3D model that indicates the appearance of the 3D model, skeleton information that indicates the motion structure of the 3D model, and skin weight information that indicates the correspondence between the appearance and the motion structure. Alternatively, the content supply unit 342 may supply content including a still 3D model with a level of detail requested based on the control information, skeleton information with a level of detail requested based on the control information, and skin weight information requested based on the control information. Note that this control information may express variations in the level of detail of the still 3D model, variations in the level of detail of the skeleton information, and variations in the skin weight information independently of one another.

[0178] Alternatively, the control information generator 313 may generate control information based on the content, and the control information supplier 341 may supply the generated control information.

[0179] Furthermore, the still 3D model generation unit 331 may generate variations in the level of resolution of the still 3D model. Then, the control information generation unit 313 may generate control information that represents the variations in the level of resolution of the generated still 3D model. Note that the still 3D model generation unit 331 may also generate a still 3D model with lower resolution based on a still 3D model with the highest resolution.

[0180] Alternatively, the skeleton information generating unit 332 may generate variations in the level of detail of the skeleton information. Then, the control information generating unit 313 may generate control information that represents variations in the level of detail of the generated skeleton information. Note that the skeleton information generating unit 332 may generate skeleton information with lower resolution based on skeleton information with the highest resolution.

[0181] The skin weight information generation unit 333 may also generate variations in skin weight information based on variations in the level of resolution of the still 3D model and variations in the level of resolution of the skeleton information. The control information generation unit 313 may then generate control information that represents the variations in the generated skin weight information. Note that the skin weight information generation unit 333 may also generate skin weight information with content corresponding to a case where at least one of the still 3D model and the skeleton information has lower resolution, based on skin weight information with content corresponding to a case where the still 3D model and the skeleton information have the highest resolution.

[0182] The control information may also represent variations in the level of detail of the motion data that represents the movement of the 3D model, and the content provider 342 may then provide content that includes motion data of the requested level of detail based on the control information.

[0183] Furthermore, the motion data generation unit 334 may generate variations in the resolution of the motion data. Then, the control information generation unit 313 may generate control information that represents the variations in the resolution of the generated motion data. Note that the motion data generation unit 334 may also generate motion data with lower resolution based on the highest resolution motion data.

[0184] The control information may be written in glTF2.0, MPD, or Universal Scene Description.

[0185] With this configuration, the distribution device 300 can suppress an increase in the load of the playback process performed by the playback device while suppressing a decrease in the quality of the user experience.

[0186] <Flow of Distribution Process> An example of the flow of distribution process executed by the distribution device 300 configured as above by applying the above-described method 1 will be described with reference to the flowchart of FIG.

[0187] When the distribution process starts, the data input unit 311 of the distribution device 300 acquires data of the skinned model (for example, skeleton information, static 3D model, skin weight information, etc.) in step S301.

[0188] In step S302, the data input unit 311 and the preprocessing unit 321 execute generation processing to generate variations of a skinned model (static 3D model, skeleton information, skin weight information) and motion data.

[0189] In step S303, the encoding unit 322 encodes content data such as the Skinned model and motion data acquired in step S301, as well as variations of the Skinned model and motion data generated by the generation process in step S302, and generates the encoded data.

[0190] In step S304, the file generation unit 323 generates a content file by storing the encoded data generated in step S303 in a file. The storage unit 314 stores the content file.

[0191] In step S305, the control information generator 313 generates control information and stores the control information in a control file, which is then stored in the storage unit 314.

[0192] In step S306, the control information supply unit 341 reads the control file stored in the storage unit 314 and supplies it to another device, such as a playback device, that plays back the content data.

[0193] In step S307, the content supply unit 342 acquires a content file distribution request supplied from another device such as a playback device that plays back the content data, reads out from the storage unit 314 a content file that stores the LoD content data specified in the request, and supplies the content file to the other device such as a playback device that plays back the content data.

[0194] When the process of step S307 ends, the distribution process ends.

[0195] <Flow of Generation Process> Next, an example of the flow of the generation process executed in step S302 of Fig. 29 will be described with reference to the flowchart of Fig. 30. Note that in this description, the Skinned model is an avatar that represents the posture and movements of a user in a virtual space. Furthermore, in that virtual space, each user's avatar reproduces the captured movements of that user.

[0196] When the generation process is started, in step S331, the still 3D model generation unit 331 sets the still 3D model acquired in step S301 of the distribution process described above as the highest LoD, and uses the still 3D model of the highest LoD to generate still 3D models of lower LoDs.

[0197] In step S332, the skeleton information generation unit 332 sets the skeleton information acquired in step S301 of the distribution process described above as the highest LoD, and uses the skeleton information of the highest LoD to generate skeleton information of lower LoDs.

[0198] In step S333, the skin weight information generation unit 333 generates skin weight information for all combinations of variations in the LoD (i.e., the level of detail of the appearance) of the still 3D model and variations in the LoD (i.e., the level of detail of the motion structure) of the skeleton information. For example, the skin weight information generation unit 333 may generate variations in the skin weight information using the skin weight information acquired in step S301 of the distribution process described above.

[0199] In step S334, the control unit 301 determines whether these processes have been performed for all Skinned models (avatars). If it is determined that an unprocessed Skinned model exists, the process returns to step S331, and the subsequent processes are repeated. In other words, the processes of steps S331 to S334 are performed for all Skinned models.

[0200] If it is determined in step S334 that there are no unprocessed Skinned models and the above-described processing has been completed for all Skinned models, the processing proceeds to step S335.

[0201] In step S335, the data input unit 311 acquires motion data of the 3D model to be processed.

[0202] In step S336, the motion data generation unit 334 sets the acquired motion data as the highest LoD, and uses the motion data of the highest LoD to generate motion data of lower LoDs.

[0203] In step S337, the control unit 301 determines whether the motion data of all 3D models (skinned models) for the current frame time has been processed. If it is determined that an unprocessed 3D model exists for the current frame time, the process returns to step S335, and the subsequent processes are repeated. In other words, the processes of steps S335 to S337 are executed for all 3D models.

[0204] Then, in step S337, if it is determined that there are no unprocessed 3D models at the current frame time and the above-described processing has been completed for all 3D models, the processing proceeds to step S338.

[0205] In step S338, the control unit 301 updates the frame to be processed, that is, the control unit 301 updates the time of the frame to be processed (that is, the current frame time).

[0206] In step S339, the control unit 301 determines whether the current frame time is the end time. If it is determined that the current frame time being processed has not reached the end time, the process returns to step S335, and the subsequent processes are repeated. In other words, the processes of steps S335 to S339 are executed until the end time is reached.

[0207] If it is determined in step S339 that the current frame time has reached the end time, the generation process ends and the process returns to FIG.

[0208] In such a distribution process and generation, the present technology described above in <3. Control of the resolution of 3D models> may be applied. For example, the above-described method 1 may be applied to this distribution device 300.

[0209] For example, in step S306 (FIG. 29), the control information supply unit 341 may supply control information for controlling the supply of content including a still 3D model indicating the appearance of the 3D model, skeleton information indicating the motion structure of the 3D model, and skin weight information indicating the correspondence between the appearance and the motion structure. Furthermore, in step S307 (FIG. 29), the content supply unit 342 may supply content including a still 3D model with a level of detail requested based on the control information, skeleton information with a level of detail requested based on the control information, and skin weight information requested based on the control information. Note that this control information may express variations in the level of detail of the still 3D model, variations in the level of detail of the skeleton information, and variations in the skin weight information independently of one another.

[0210] By performing each process as described above, the distribution device 300 can suppress an increase in the load of the playback process performed by the playback device while suppressing a decrease in the quality of the user experience.

[0211] 5. Second Embodiment Playback Device Fig. 31 is a block diagram showing an example of the configuration of a playback device, which is one aspect of an information processing device to which the present technology is applied. The playback device 400 shown in Fig. 31 is a playback device that acquires content data based on control information and performs playback processing of the acquired content data. For example, the playback device 400 plays content data (such as a skinned model and motion data) stored in a content file generated by the distribution device 300. Of course, the playback device 400 may also perform other processing.

[0212] Note that Fig. 31 shows the main processing units, data flows, etc., and does not necessarily show everything. In other words, playback device 400 may have processing units that are not shown as blocks in Fig. 31, or may have processes or data flows that are not shown as arrows, etc. in Fig. 31.

[0213] As shown in Fig. 31, the playback device 400 has a control unit 401 and a playback processing unit 402. The control unit 401 performs processing related to control of each processing unit of the playback processing unit 402. The playback processing unit 402 performs processing related to playback of content data. Of course, the control unit 401 and the playback processing unit 402 may also perform other processing.

[0214] The playback processing unit 402 has a display control unit 411, a file acquisition unit 412, a content playback unit 413, and a display unit 414. The display control unit 411 also has a still 3D model resolution setting unit 421 and a skeleton information resolution setting unit 422. The file acquisition unit 412 also has a control information acquisition unit 431 and a content acquisition unit 432. The content playback unit 413 also has a file processing unit 441, a decoding unit 442, and a display information generation unit 443.

[0215] The display control unit 411 executes processing related to image display control. For example, the display control unit 411 may control the file acquisition unit 412 to acquire information provided from another device. For example, the display control unit 411 may cause the control information acquisition unit 431 to acquire, provide, and acquire control information. The display control unit 411 may also set the resolution of the still 3D model and skeleton information based on the control information. The display control unit 411 may also select a variation of skin weight information or set the resolution of motion data. The display control unit 411 may also cause the content acquisition unit 432 to acquire the still 3D model, skeleton information, skin weight information, motion data, etc. of the selected variation. The display control unit 411 may also control the display information generation unit 443 to generate display information. Of course, the display control unit 411 may also execute other processing.

[0216] The file acquisition unit 412 performs processing related to information acquisition under the control of the display control unit 411. For example, the file acquisition unit 412 may acquire a control file supplied from another device (e.g., the distribution device 300). The file acquisition unit 412 may then supply the acquired control file to the display control unit 411. The file acquisition unit 412 may then supply the distribution device 300 with a request for distribution of a content file storing content data of a desired level of definition (LoD) under the control of the display control unit 411 based on the control file. The file acquisition unit 412 may also acquire a content file distributed by the distribution device 300 based on the request. The file acquisition unit 412 may also supply the acquired content file to the file processing unit 441. Of course, the file acquisition unit 412 may also perform other processing.

[0217] The content playback unit 413 executes processing related to the content. For example, the content playback unit 413 may acquire a content file supplied from the file acquisition unit 412. The content playback unit 413 may also extract content data from the acquired content file, decrypt the data, and generate display information. The content playback unit 413 may also supply the generated display information to the display unit 414 to display it. Of course, the content playback unit 413 may also execute other processing.

[0218] The display unit 414 has a display device and performs processing related to image display. For example, the display unit 414 may acquire display information supplied from the display information generation unit 443 and display the information using the display device. Of course, the display unit 414 may also perform other processing, such as displaying other information.

[0219] The still 3D model resolution setting unit 421 performs processing related to setting the resolution of the still 3D model. For example, the still 3D model resolution setting unit 421 may set the resolution of the still 3D model based on control information. The still 3D model resolution setting unit 421 may also cause the content acquisition unit 432 to acquire a still 3D model with the set resolution. Of course, the still 3D model resolution setting unit 421 may also perform other processing.

[0220] The skeleton information resolution setting unit 422 performs processing related to setting the resolution of the skeleton information. For example, the skeleton information resolution setting unit 422 may set the resolution of the skeleton information based on control information. Furthermore, the skeleton information resolution setting unit 422 may cause the content acquisition unit 432 to acquire skeleton information of the set resolution. Of course, the skeleton information resolution setting unit 422 may also perform other processing.

[0221] The control information acquisition unit 431 performs processing related to the acquisition of control information. For example, the control information acquisition unit 431 may acquire a control file supplied from an external source (e.g., the distribution device 300) and supply the control file to the display control unit 411. Of course, the control information acquisition unit 431 may also perform other processing.

[0222] The content acquisition unit 432 performs processing related to content acquisition. For example, the content acquisition unit 432 may request and acquire a content file from an external device (e.g., the distribution device 300) under the control of the display control unit 411. The content acquisition unit 432 may also supply the acquired content file to the file processing unit 441. Of course, the content acquisition unit 432 may also perform other processing.

[0223] The file processing unit 441 performs processing related to a content file. For example, the file processing unit 441 may acquire a content file supplied from the file acquisition unit 412. The file processing unit 441 may also extract encoded data, metadata, and the like of the content data stored in the acquired content file. The file processing unit 441 may also supply the extracted encoded data of the content data to the decoding unit 442. The file processing unit 441 may also supply the extracted metadata, and the like, to the display information generation unit 443. Of course, the file processing unit 441 may also perform other processing.

[0224] The decoding unit 442 performs processing related to decoding of encoded data of content data. For example, the decoding unit 442 may acquire encoded data of content data supplied from the file processing unit 441. The decoding unit 442 may also decode the encoded data using a predetermined decoding method to generate (restore) content data. Note that this decoding method may be any method as long as it corresponds to the encoding method applied during encoding. The decoding unit 442 may also supply the generated content data to the display information generation unit 443. Of course, the decoding unit 442 may also perform other processing.

[0225] The display information generation unit 443 executes processing related to the generation of display information to be displayed on the display unit 414. For example, the display information generation unit 443 may acquire content data supplied from the decoding unit 442. The display information generation unit 443 may also acquire metadata, etc. supplied from the file processing unit 441. The display information generation unit 443 may also acquire information supplied from the display control unit 411. The display information generation unit 443 may generate display information using the acquired information (content data, etc.). For example, the display information generation unit 443 may generate display information by playing back content data (3D models) and reconstructing or rendering avatars (objects) in a virtual space. The display information generation unit 443 may also supply the generated display information to the display unit 414 for display. Of course, the display information generation unit 443 may execute other processing. The display information generation unit 443 may execute some or all of these processing steps under the control of the display control unit 411.

[0226] The present technology described above in <3. Control of the level of detail of a 3D model> may be applied to the playback device 400 configured as above. For example, the above-described method 1 may be applied to this playback device 400.

[0227] For example, the still 3D model resolution setting unit 421 may set the resolution of a still 3D model that indicates the appearance of the 3D model. Furthermore, the skeleton information resolution setting unit 422 may set the resolution of skeleton information that indicates the motion structure of the 3D model independently of the still 3D model. Furthermore, the content playback unit 413 may play content that includes a still 3D model, skeleton information, and skin weight information that indicates the correspondence between the appearance and motion structure, corresponding to the set resolution.

[0228] Alternatively, the still 3D model resolution setting unit 421 may set the resolution of the still 3D model based on a first parameter, and the skeleton information resolution setting unit 422 may set the resolution of the skeleton information based on a second parameter different from the first parameter. For example, the second parameter may be a parameter of a different type from the first parameter. For example, the first parameter may be a parameter related to the distance from the viewpoint position to the 3D model, and the second parameter may be a parameter related to the content that is different from the distance. Alternatively, the first parameter may be a parameter related to the distance from the viewpoint position to the 3D model, and the second parameter may be a parameter related to the load of the content playback process. Alternatively, the second parameter may be a parameter with a different value from the first parameter. Alternatively, the second parameter may be a parameter with a different timing from the first parameter.

[0229] Alternatively, the control information acquisition unit 431 may acquire control information for controlling the supply of content. Then, the still 3D model resolution setting unit 421 may set the resolution of the still 3D model based on the acquired control information. Also, the skeleton information resolution setting unit 422 may set the resolution of the skeleton information independently of the still 3D model based on the acquired control information. Also, the content acquisition unit 432 may acquire, as content, a still 3D model and skeleton information with resolutions set independently of each other, as well as skin weight information corresponding to the still 3D model and skeleton information, based on the control information. Then, the content playback unit 413 may play back the acquired content.

[0230] Note that this control information may be written in ITF2.0, or may be MPD, or may be Universal Scene Description.

[0231] The content playback unit 413 may also play content that further includes motion data that indicates the movement of the 3D model. The resolution of the motion data may correspond to the resolution of the skeleton information. The content acquisition unit 432 may also acquire the still 3D model, skeleton information, motion data, and skin weight information as content. The content playback unit 413 may then play the acquired content.

[0232] With this configuration, the playback device 400 can independently control the resolution of the still 3D model and the skeleton information. Therefore, the playback device 400 can suppress an increase in the playback processing load while suppressing a decrease in the quality of the user experience.

[0233] <Flow of Playback Processing> An example of the flow of playback processing executed by playback device 400 configured as described above using Method 1 will be described with reference to the flowchart in Fig. 32. Note that in this description, the object represented by the Skinned model is an avatar that represents the posture and movement of a user in a virtual space.

[0234] When playback processing starts, the control information acquisition unit 431 of the playback device 400 acquires a control file in step S401. In this control file, variations in the level of detail of the still 3D model, variations in the level of detail of the skeleton information, and variations in the skin weight information are expressed independently of one another.

[0235] In step S402, the still 3D model definition setting unit 421 sets the LoD (level of definition of appearance) of the still 3D model that is requested to be supplied from the distribution device 300.

[0236] In step S403, the skeleton information resolution setting unit 422 sets the LoD (resolution of the movement structure) of the skeleton information that is requested to be supplied to the distribution device 300. The skeleton information resolution setting unit 422 performs this setting independently of the resolution setting of the still 3D model.

[0237] In step S404, the content acquisition unit 432 executes an acquisition process, and based on the control of the display control unit 411, acquires a Skinned model and motion data according to the LoD set as described above.

[0238] In step S405, the content playback unit 413 generates a full 3D model for the current frame time using the Skinned model and motion data acquired by the processing in step S404.

[0239] In step S406, the display information generating unit 443 generates display information for the current frame time generated in the process of step S405. The display unit 414 displays the display information.

[0240] In step S407, the control unit 401 updates the frame time.

[0241] In step S408, the control unit 401 determines whether the current frame time is the end time. If it is determined that the current frame time is not the end time, the process returns to step S402, and the subsequent processes are repeated. Also, if it is determined in step S408 that the current frame time has reached the end time, the playback process ends.

[0242] <Acquisition Process Flow 1> Next, an example of the acquisition process flow executed in step S404 in Fig. 32 will be described with reference to the flowchart in Fig. 33. In this flowchart, an example will be described in which control information is written in glTF2.0.

[0243] When the acquisition process starts, in step S431, the content acquisition unit 432 searches MSFT_lod for a mesh node that indicates the relevant LoD information (LoD set in step S402) from the LoD of the still 3D model, and acquires the relevant mesh data.

[0244] In step S432, the content acquisition unit 432 searches Skeleton_lod for a skin node indicating the LoD information (LoD set in step S403) in question from the LoD of the skeleton information, and acquires the relevant skeleton information.

[0245] In step S433, the content acquisition unit 432 acquires the motion data of each bone contained in the skeleton information from the animations that refer to the index of the node representing the bone information.

[0246] In step S434, the content acquisition unit 432 searches for and acquires the corresponding skin weight information from SkinWeights based on the node index indicating the LoD of the still 3D model and the node index indicating the LoD of the skeleton information.

[0247] When the process of step S434 ends, the acquisition process ends and the process returns to FIG.

[0248] <Acquisition Process Flow 2> Next, an example of the acquisition process flow when control information is described in an MPD will be described with reference to the flowchart in FIG.

[0249] When the acquisition process is started, in step S451, the content acquisition unit 432 selects a corresponding preselection from the LoD of the still 3D model and the corresponding LoD.

[0250] In step S452, the content acquisition unit 432 acquires all necessary data (skeleton information of the desired LoD, static 3D model, skin weight information, motion data, etc.) by acquiring the adaptation set referenced by the preselection.

[0251] When the process of step S452 ends, the acquisition process ends and the process returns to FIG.

[0252] <Acquisition Process Flow 3> Next, an example of the acquisition process flow when control information is described in USD will be described with reference to the flowchart in FIG.

[0253] When the acquisition process is started, in step S471, the content acquisition unit 432 searches for a Variant of a mesh prim that indicates the LoD information in question from the variantSet of the Mesh, and acquires the mesh data in question.

[0254] In step S472, the content acquisition unit 432 searches for a Skeleton prim that indicates the relevant LoD information from the variantSet of Skeleton, and acquires the relevant skeleton information.

[0255] In step S473, the content acquisition unit 432 acquires the motion data of each bone contained in the relevant skeleton information from the SkelAnimation prim defined in the Variant.

[0256] In step S474, the content acquisition unit 432 acquires corresponding skin weight information from the Variant of the SkinWeight prim based on the Variant name of the Mesh prim indicating the LoD of the still 3D model and the Variant name of the Skeleton prim indicating the LoD of the skeleton information.

[0257] When the process of step S474 ends, the acquisition process ends and the process returns to FIG.

[0258] In such a playback process and acquisition process, the present technology described above in <3. Control of the resolution of the 3D model> may be applied. For example, the above-described method 1 may be applied to this playback device 400.

[0259] For example, in step S402 (FIG. 32), the still 3D model resolution setting unit 421 may set the resolution of a still 3D model that indicates the appearance of the 3D model. Also, in step S403 (FIG. 32), the skeleton information resolution setting unit 422 may set the resolution of skeleton information that indicates the motion structure of the 3D model independently of the still 3D model. Also, in steps S405 and S406 (FIG. 32), the content playback unit 413 may play content that includes a still 3D model, skeleton information, and skin weight information that indicates the correspondence between the appearance and motion structure, corresponding to the set resolution.

[0260] By performing each process as described above, the playback device 400 can independently control the resolution of the still 3D model and the skeleton information. Therefore, the playback device 400 can suppress an increase in the playback processing load while suppressing a decrease in the quality of the user experience.

[0261] 6. Supplementary Notes Computer The above-described series of processes can be executed by hardware or software. When the series of processes are executed by software, the programs that make up the software are installed on a computer. Here, the term computer includes computers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0262] FIG. 36 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0263] In a computer 900 shown in FIG. 36, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0264] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.

[0265] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, and an input terminal. The output unit 912 includes, for example, a display, a speaker, and an output terminal. The storage unit 913 includes, for example, a hard disk, a RAM disk, and a non-volatile memory. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0266] In a computer configured as described above, the CPU 901 performs the above-described series of processes by, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data necessary for the CPU 901 to execute various processes as appropriate.

[0267] The program executed by the computer can be applied by recording it on, for example, a removable medium 921 such as a package medium. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by inserting the removable medium 921 into the drive 915.

[0268] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0269] Alternatively, this program can be installed in advance in the ROM 902 or the storage unit 913 .

[0270] <Application of the Present Technology> The present technology can be applied to any configuration. For example, the present technology can be applied to various electronic devices.

[0271] Furthermore, for example, the present technology can also be implemented as part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set in which other functions are added to a unit (e.g., a video set).

[0272] Furthermore, for example, the present technology can also be applied to a network system configured with multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology may be implemented in a cloud service that provides image (video)-related services to any terminal, such as a computer, an AV (Audio Visual) device, a portable information processing terminal, or an IoT (Internet of Things) device.

[0273] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0274] <Fields and uses to which this technology can be applied> Systems, devices, processing units, etc. to which this technology is applied can be used in any field, for example, transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, nature monitoring, etc. In addition, the uses thereof are also arbitrary.

[0275] <Others> In this specification, a "flag" refers to information for identifying multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the value that this "flag" can take may be, for example, two values, 1 / 0, or three or more values. That is, the number of bits constituting this "flag" is arbitrary, and may be one bit or multiple bits. Furthermore, identification information (including flags) can be included not only in a bitstream, but also in a bitstream that includes differential information of the identification information relative to certain reference information. Therefore, in this specification, "flag" and "identification information" encompass not only the information itself, but also differential information relative to the reference information.

[0276] Furthermore, various types of information (e.g., metadata) related to the coded data (bitstream) may be transmitted or recorded in any form as long as they are associated with the coded data. Here, the term "associate" means, for example, that one piece of data can be used (linked) when processing the other piece of data. That is, the associated pieces of data may be combined into one piece of data or may be separate pieces of data. For example, information associated with coded data (image) may be transmitted over a transmission path separate from that of the coded data (image). Furthermore, for example, information associated with coded data (image) may be recorded on a recording medium separate from that of the coded data (image) (or on a different recording area of ​​the same recording medium). Note that this "association" may refer not to the entire data, but to only part of the data. For example, an image and information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a portion of a frame.

[0277] In this specification, terms such as "composite," "multiplex," "add," "integrate," "include," "store," "embed," "insert," and the like refer to combining multiple items into one, such as combining encoded data and metadata into one piece of data, and refer to one method of "associating" as described above.

[0278] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0279] For example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0280] Furthermore, for example, the above-described program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and is able to obtain the necessary information.

[0281] Also, for example, each step of a single flowchart may be executed by a single device, or may be shared and executed by multiple devices. Furthermore, when a single step includes multiple processes, the multiple processes may be executed by a single device, or may be shared and executed by multiple devices. In other words, multiple processes included in a single step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as a single step.

[0282] For example, the steps of a program executed by a computer may be executed in chronological order in the order described herein, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the steps may be executed in an order different from the order described above. Furthermore, the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0283] Furthermore, for example, the multiple technical elements included in the present technology can be implemented independently and individually, as long as no contradiction occurs. Of course, any multiple technical elements among them can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the present technology described above can be implemented in combination with other technologies not described above.

[0284] Note that the present technology can also be configured as follows: (1) An information processing device comprising: a control information supply unit that supplies control information for controlling supply of content including a still 3D model that indicates the appearance of a 3D model, skeleton information that indicates a motion structure of the 3D model, and skin weight information that indicates a correspondence between the appearance and the motion structure; and a content supply unit that supplies the content including the still 3D model with a level of detail requested based on the control information, the skeleton information with a level of detail requested based on the control information, and the skin weight information requested based on the control information, wherein the control information represents variations in the level of detail of the still 3D model, variations in the level of detail of the skeleton information, and variations in the skin weight information independently of one another. (2) The information processing device according to (1), further comprising: a control information generation unit that generates the control information based on the content, wherein the control information supply unit is configured to supply the generated control information. (3) The information processing device according to (2), further comprising a still 3D model generation unit that generates variations in the level of resolution of the still 3D model, wherein the control information generation unit is configured to generate the control information that represents variations in the level of resolution of the generated still 3D model. (4) The information processing device according to (3), wherein the still 3D model generation unit generates the still 3D model with lower resolution based on the still 3D model with highest resolution. (5) The information processing device according to any of (2) to (4), further comprising a skeleton information generation unit that generates variations in the level of resolution of the skeleton information, wherein the control information generation unit is configured to generate the control information that represents variations in the level of resolution of the generated skeleton information. (6) The information processing device according to (5), wherein the skeleton information generation unit generates the skeleton information with lower resolution based on the skeleton information with highest resolution.(7) The information processing device according to any one of (2) to (6), further comprising a skin weight information generation unit that generates variations of the skin weight information based on variations in resolution of the still 3D model and variations in resolution of the skeleton information, wherein the control information generation unit is configured to generate the control information representing the variations of the generated skin weight information. (8) The information processing device according to (7), wherein the skin weight information generation unit generates the skin weight information of a content corresponding to a case where at least one of the still 3D model and the skeleton information has lower resolution, based on the skin weight information of a content corresponding to a case where the still 3D model and the skeleton information have the highest resolution. (9) The information processing device according to any one of (2) to (8), wherein the control information further represents variations in resolution of motion data indicating movement of the 3D model, and the content supply unit is configured to supply the content including the motion data of a resolution requested based on the control information. (10) The information processing device according to (9), further comprising a motion data generation unit that generates variations in resolution of the motion data, wherein the control information generation unit is configured to generate the control information that represents variations in resolution of the generated motion data. (11) The information processing device according to (10), wherein the motion data generation unit generates the motion data with lower resolution based on the motion data with the highest resolution. (12) The information processing device according to any of (1) to (11), wherein the control information is described in glTF2.0. (13) The information processing device according to any of (1) to (11), wherein the control information is MPD (Media Presentation Description). (14) The information processing device according to any of (1) to (11), wherein the control information is Universal Scene Description.(15) An information processing method comprising: supplying control information for controlling the supply of content including a still 3D model that indicates the appearance of a 3D model, skeleton information that indicates the motion structure of the 3D model, and skin weight information that indicates the correspondence between the appearance and the motion structure; supplying the content including the still 3D model with a resolution requested based on the control information, the skeleton information with a resolution requested based on the control information, and the skin weight information requested based on the control information; and the control information expressing variations in resolution of the still 3D model, variations in resolution of the skeleton information, and variations in the skin weight information independently of each other.

[0285] (21) An information processing device comprising: a still 3D model resolution setting unit that sets a resolution of a still 3D model that indicates the appearance of the 3D model; a skeleton information resolution setting unit that sets a resolution of skeleton information that indicates a motion structure of the 3D model independently of the still 3D model; and a content playback unit that plays back content that corresponds to the set resolution and includes the still 3D model, the skeleton information, and skin weight information that indicates a correspondence between the appearance and the motion structure. (22) The information processing device described in (21), wherein the still 3D model resolution setting unit sets the resolution of the still 3D model based on a first parameter, and the skeleton information resolution setting unit sets the resolution of the skeleton information based on a second parameter that is different from the first parameter. (23) The information processing device described in (22), wherein the second parameter is a parameter of a different type from the first parameter. (24) The information processing device according to (23), wherein the first parameter is a parameter related to a distance from a viewpoint position to the 3D model, and the second parameter is a parameter related to the content that is different from the distance. (25) The information processing device according to (23), wherein the first parameter is a parameter related to a distance from a viewpoint position to the 3D model, and the second parameter is a parameter related to a load of a playback process of the content. (26) The information processing device according to (22), wherein the second parameter is a parameter having a different value from the first parameter. (27) The information processing device according to (22), wherein the second parameter is a parameter having a different timing from the first parameter. (28) The information processing device according to any one of (21) to (27), further comprising a control information acquisition unit that acquires control information that controls the supply of the content, wherein the still 3D model resolution setting unit is configured to set the resolution of the still 3D model based on the control information, and the skeleton information resolution setting unit is configured to set the resolution of the skeleton information independently of the still 3D model based on the control information.(29) The information processing device according to (28), further comprising a content acquisition unit that acquires, as the content, the still 3D model and the skeleton information, with resolutions set independently of each other, and the skin weight information corresponding to the still 3D model and the skeleton information, based on the control information, wherein the content playback unit is configured to play the acquired content. (30) The information processing device according to (28) or (29), wherein the control information is described in glTF2.0. (31) The information processing device according to (28) or (29), wherein the control information is MPD (Media Presentation Description). (32) The information processing device according to (28) or (29), wherein the control information is Universal Scene Description. (33) The information processing device according to any of (21) to (32), wherein the content playback unit plays the content further including motion data indicating movement of the 3D model. (34) The information processing device according to (33), wherein the resolution of the motion data corresponds to the resolution of the skeleton information. (35) The information processing device according to (34), further comprising a content acquisition unit that acquires the still 3D model, the skeleton information, the motion data, and the skin weight information as the content, wherein the content playback unit is configured to play back the acquired content. (36) An information processing method that sets a resolution of a still 3D model that indicates the appearance of a 3D model, sets a resolution of skeleton information that indicates a motion structure of the 3D model independently of the still 3D model, and plays back content that includes the still 3D model, the skeleton information, and skin weight information that indicates a correspondence between the appearance and the motion structure, corresponding to the set resolution.

[0286] 300 Distribution device, 301 Control unit, 302 Distribution processing unit, 311 Data input unit, 312 Content processing unit, 313 Control information generation unit, 314 Memory unit, 315 Supply unit, 321 Pre-processing unit, 322 Encoding unit, 323 File generation unit, 331 Still 3D model generation unit, 332 Skeleton information generation unit, 333 Skin weight information generation unit, 334 Motion data generation unit, 341 Control information supply unit, 342 Content supply unit, 400 Playback device, 401 Control unit, 402 Playback processing unit, 411 Display control unit, 412 File acquisition unit, 413 Content playback unit, 414 Display unit, 421 Still 3D model resolution setting unit, 422 Skeleton information resolution setting unit, 431 Control information acquisition unit 432 Content acquisition unit, 441 File processing unit, 442 Decryption unit, 443 Display information generation unit, 900 Computer

Claims

1. An information processing device comprising: a control information supply unit that supplies control information to control the supply of content including a still 3D model that indicates the appearance of a 3D model, skeleton information that indicates a motion structure of the 3D model, and skin weight information that indicates a correspondence between the appearance and the motion structure; and a content supply unit that supplies the content including the still 3D model of a level of resolution requested based on the control information, the skeleton information of a level of resolution requested based on the control information, and the skin weight information requested based on the control information, wherein the control information represents variations in resolution of the still 3D model, variations in resolution of the skeleton information, and variations in the skin weight information independently of each other.

2. The information processing device according to claim 1, further comprising a control information generating unit that generates the control information based on the content, wherein the control information supplying unit is configured to supply the generated control information.

3. The information processing device according to claim 2, further comprising a still 3D model generation unit that generates variations in the level of detail of the still 3D model, wherein the control information generation unit is configured to generate the control information representing the variations in the level of detail of the generated still 3D model.

4. The information processing device according to claim 2, further comprising a skeleton information generating unit that generates variations in the level of detail of the skeleton information, wherein the control information generating unit is configured to generate the control information that represents the variations in the level of detail of the generated skeleton information.

5. The information processing device of claim 2, further comprising a skin weight information generation unit that generates variations in the skin weight information based on variations in the resolution of the static 3D model and variations in the resolution of the skeleton information, wherein the control information generation unit is configured to generate the control information that represents the variations in the generated skin weight information.

6. The information processing device of claim 2, wherein the control information further represents variations in resolution of motion data indicating the movement of the 3D model, and the content supply unit is configured to supply the content including the motion data of a resolution requested based on the control information.

7. The information processing device according to claim 1, wherein the control information is described in glTF2.

0.

8. The information processing device according to claim 1, wherein the control information is a Media Presentation Description (MPD).

9. The information processing device according to claim 1, wherein the control information is a Universal Scene Description.

10. An information processing method comprising: supplying control information for controlling the supply of content including a still 3D model indicating the appearance of a 3D model, skeletal information indicating a motion structure of the 3D model, and skin weight information indicating a correspondence between the appearance and the motion structure; supplying the content including the still 3D model of a level of resolution requested based on the control information, the skeletal information of a level of resolution requested based on the control information, and the skin weight information requested based on the control information; and the control information expressing variations in resolution of the still 3D model, variations in resolution of the skeletal information, and variations in the skin weight information independently of one another.

11. An information processing device comprising: a still 3D model resolution setting unit that sets a resolution of a still 3D model that indicates the appearance of a 3D model; a skeleton information resolution setting unit that sets a resolution of skeleton information that indicates the movement structure of the 3D model independently of the still 3D model; and a content playback unit that plays content that corresponds to the set resolution and includes the still 3D model, the skeleton information, and skin weight information that indicates the correspondence between the appearance and the movement structure.

12. The information processing device according to claim 11, wherein the still 3D model resolution setting unit sets the resolution of the still 3D model based on a first parameter, and the skeleton information resolution setting unit sets the resolution of the skeleton information based on a second parameter different from the first parameter.

13. The information processing device according to claim 12, wherein the second parameter is a parameter of a different type from the first parameter.

14. The information processing device according to claim 13, wherein the first parameter is a parameter relating to a distance from a viewpoint position to the 3D model, and the second parameter is a parameter relating to the content that is different from the distance.

15. The information processing device according to claim 13, wherein the first parameter is a parameter relating to a distance from a viewpoint position to the 3D model, and the second parameter is a parameter relating to a load of a playback process of the content.

16. The information processing device according to claim 11, further comprising a control information acquisition unit that acquires control information for controlling the supply of the content, wherein the still 3D model resolution setting unit is configured to set the resolution of the still 3D model based on the control information, and the skeleton information resolution setting unit is configured to set the resolution of the skeleton information independently of the still 3D model based on the control information.

17. The information processing device of claim 16, further comprising a content acquisition unit that acquires, as the content, the still 3D model and the skeleton information, each having a resolution set independently of the other, and the skin weight information corresponding to the still 3D model and the skeleton information, based on the control information, and the content playback unit is configured to play back the acquired content.

18. The information processing device according to claim 11, wherein the content playback unit plays back the content further including motion data indicating a movement of the 3D model.

19. The information processing device according to claim 18, wherein the resolution of the motion data corresponds to the resolution of the skeleton information.

20. An information processing method comprising: setting a resolution of a still 3D model that indicates the appearance of a 3D model; setting a resolution of skeletal information that indicates the motion structure of the 3D model independently of the still 3D model; and playing content that includes the still 3D model, the skeletal information, and skin weight information that indicates the correspondence between the appearance and the motion structure, corresponding to the set resolution.

Images