Information processing system and method for producing a 3D animation

Abstract

An information processing method including receiving two different pieces of skeletal information, each defining a structure of a 3D model or motion data for the 3D animation, the skeletal information including parts corresponding to joints or extremities and their positional or connection relations, estimating, using a machine-learning-based estimator trained on skeletal knowledge, a correspondence between the parts of the two different pieces of skeletal information, and outputting a correspondence table representing the correspondence, and transforming, based on the correspondence table, at least one of the two different pieces of skeletal information to generate compatible skeletal information conforming to a unified standard.

Description

INFORMATION PROCESSING SYSTEM AND METHOD FOR PRODUCING A 3D ANIMATIONCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of Japanese Priority Patent Application JP 2024-123249 filed July 30, 2024, the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to an information processing system and an information processing method.

[0003] In recent years, the production of three-dimensional (3D) animations has become popular. Further, technologies for supporting the production of 3D animations have been developed, as disclosed in PTL 1, for example.

[0004] Japanese Patent No. 7061238Summary

[0005] There are cases where skeletal information of 3D models and skeletal information of motion data for causing the 3D models to perform predetermined motions are used in the production of 3D animations. However, the above-mentioned skeletal information is generally defined uniquely by each creator, thus resulting in poor compatibility.

[0006] According to one mode of the present disclosure, there is provided an information processing system for producing a three-dimensional (3D) animation, comprising: circuitry configured to: receive two different pieces of skeletal information, each defining a structure of a 3D model or motion data for the 3D animation, the skeletal information including parts and positional relationships between the parts, wherein the parts include at least one of a joint, an extremity, a bone and / or other skeletal segments; generate, using a machine-learning-based estimator trained on skeletal knowledge, a correspondence table defining a correspondence between the parts of the two different pieces of skeletal information; transform, based on the correspondence table, at least one of the two different pieces of skeletal information to generate compatible skeletal information conforming to a unified standard; and apply the compatible skeletal information to produce a 3D animation by adapting the motion data to the 3D model or interpolating between pieces of the motion data.

[0007] According to another mode of the present disclosure, there is provided an information processing method for producing a three-dimensional (3D) animation, comprising: receiving, by a circuitry, two different pieces of skeletal information, each defining a structure of a 3D model or motion data for the 3D animation, the skeletal information including parts and positional relationships between the parts, wherein the parts include joint, extremity, bones and / or other skeletal segments; generating, by the circuitry using a machine-learning-based estimator trained on skeletal knowledge, a correspondence table defining a correspondence between the parts of the two different pieces of skeletal information; transforming, by the circuitry based on the correspondence table, at least one of the two different pieces of skeletal information to generate compatible skeletal information conforming to a unified standard; and applying, by the circuitry, the compatible skeletal information to produce a 3D animation by adapting the motion data to the 3D model or interpolating between pieces of the motion data, thereby automating compatibility in the 3D animation production process.

[0008] FIG. 1 is a diagram for illustrating an overview of one embodiment of the present disclosure.FIG. 2 is a diagram illustrating a specific example of skeletal information 60A according to the embodiment.FIG. 3 is a diagram illustrating a specific example of skeletal information 60B according to the embodiment.FIG. 4 is a diagram illustrating a specific example of a correspondence table 70 according to the embodiment.FIG. 5 is a block diagram illustrating an example of the functional configuration of an information processing system 1 that performs an information processing method according to the embodiment.FIG. 6 is a diagram for illustrating learning performed by an estimator 125 according to the embodiment.FIG. 7 is a diagram for illustrating an efficiency improvement of remapping according to the embodiment.FIG. 8 is a diagram for illustrating output of the correspondence table 70 based on the skeletal information 60A of a bipedal walking type and skeletal information 60C of a quadrupedal walking type according to the embodiment.FIG. 9 is a diagram for illustrating output of the correspondence table 70 based on skeletal information that defines some parts of the entire body according to the embodiment.FIG. 10 is a diagram for illustrating standardization of skeletal information 60 according to the embodiment.FIG. 11 is a diagram for illustrating a search based on standardized skeletal information 60AS according to the embodiment.FIG. 12 is a diagram for illustrating an example of the configuration of the information processing system 1 in the case of performing a motion-to-motion search according to the embodiment.FIG. 13 is a diagram for illustrating motion interpolation based on the correspondence table 70 according to the embodiment.FIG. 14 is a block diagram illustrating an example of the hardware configuration of an information processing apparatus 90 according to the embodiment.Description of Embodiment

[0009] Hereinafter, a preferred embodiment of the present disclosure will be described in detail with reference to the accompanying drawings. It should be noted that, in the present specification and the drawings, components having substantially the same functional configuration are denoted by the same reference sign, and redundant description will be omitted.

[0010] In addition, in the present specification and drawings, in the case where multiple components of the same type are described separately, an alphabet or the like may be added to the end of the reference sign. On the other hand, in the case where there is no need to distinguish between the multiple components of the same type, the above-mentioned alphabet or the like may be omitted, and a description common to all of the multiple components of the same type may be given.

[0011] Note that the description will be given in the following order. 1. Embodiment 1.1. Overview 1.2. Example of functional configuration 1.3. Learning by estimator 125 1.4. Processing using correspondence table 70 2. Example of hardware configuration 3. Summary

[0012] <1. Embodiment> <<1.1. Overview>> First, an overview of one embodiment of the present disclosure will be described.

[0013] As described above, there are cases where skeletal information is used in the production of 3D animations.

[0014] Skeletal information is information that defines the structure of the body (skeleton) of a 3D model.

[0015] The skeletal information includes information regarding parts corresponding to joints, extremities, bones, or other skeletal segments, names of the parts, relations between the parts, postures defined by the positional relations of the multiple parts, and the like, for example.

[0016] Note that the above-mentioned relations between the parts include positional relations between the parts and connection relations between the parts, for example.

[0017] In the skeletal information, defining the positional relations between the multiple parts and the connection relations between the parts makes it possible to define the posture that the 3D model should take and the changes in that posture, that is, the motion.

[0018] However, at present, there is no unified standard for skeletal information, and skeletal information is generally defined uniquely by each creator.

[0019] For example, there are cases where the number of parts, names of the parts, etc., differ between skeletal information of one 3D model imitating a human and skeletal information of another 3D model imitating a human.

[0020] As an example, there are cases where a part defined by the name “hips” in one 3D model is defined by the name “pelvis” in another 3D model.

[0021] In this way, pieces of skeletal information specially created by different creators are often not compatible as they are, which reduces the efficiency of various types of processing and operations.

[0022] The technical idea according to the one embodiment of the present disclosure has been conceived in terms of the above-mentioned points, and aims at improving the efficiency of various types of processing and operations that use skeletal information. The technical idea according to the present embodiment addresses the challenge of incompatible skeletal information in 3D animation production, which traditionally requires manual remapping of parts, leading to inefficiencies and errors. By automating part correspondence estimation using a machine-learning-based estimator and transforming skeletal information to a unified standard, the invention significantly reduces processing time and enhances animation accuracy, enabling seamless integration of diverse 3D models and motion data in real-time production environments.

[0023] For this reason, as one feature of an information processing method according to the one embodiment of the present disclosure, the method uses an estimator 125 to estimate, on the basis of two different pieces of skeletal information, the correspondence between parts defined in the two respective pieces of skeletal information.

[0024] FIG. 1 is a diagram for illustrating an overview of the one embodiment of the present disclosure.

[0025] The skeletal information according to the present embodiment may be extracted from a 3D model, or may be extracted from motion data.

[0026] In FIG. 1, it is assumed that skeletal information 60A is skeletal information extracted from motion data 50A, and skeletal information 60B is skeletal information extracted from a 3D model 55B.

[0027] In addition, it is assumed that the skeletal information 60A and the skeletal information 60B illustrated in FIG. 1 are compliant with different standards from each other.

[0028] The estimator 125 according to the present embodiment estimates, on the basis of the two pieces of skeletal information compliant with the different standards as described above, the correspondence between parts defined in the two respective pieces of skeletal information and outputs the result of the estimation.

[0029] In the case of the example illustrated in FIG. 1, the estimator 125 outputs a correspondence table 70 as the result of the estimation based on the skeletal information 60A and the skeletal information 60B that have been input.

[0030] The correspondence table 70 is an example of information indicating the correspondence between the parts defined in the two respective pieces of skeletal information having been input. In order words, The correspondence table 70 maps corresponding parts between the two pieces of skeletal information and indicates 'None' for parts with no corresponding part, ensuring comprehensive handling of all parts regardless of compatibility.

[0031] Here, with reference to FIGS. 2 to 4, more specific examples of the skeletal information 60A, the skeletal information 60B, and the correspondence table 70 will be illustrated. Each element in the skeletal information 60A, the skeletal information 60B, and the correspondence table 70 illustrated in FIGS. 2 to 4 indicates the name of a part.

[0032] When the skeletal information 60A illustrated in FIG. 2 is compared with the skeletal information 60B illustrated in FIG. 3, it can be seen that the number of parts is different between the skeletal information 60A and the skeletal information 60B.

[0033] Further, when the skeletal information 60A illustrated in FIG. 2 is compared with the skeletal information 60B illustrated in FIG. 3, it can also be seen that the name of a part included in one is not included in the other.

[0034] Even in the case where the number of parts differs or the names of the parts differ between pieces of skeletal information as described above, the estimator 125 according to the present embodiment can estimate the correspondence between the parts.

[0035] For example, in an example of the correspondence table 70 illustrated in FIG. 4, the part “LeftUpLeg” in the skeletal information 60A and the part “LeftHip” in the skeletal information 60B are estimated to be corresponding parts.

[0036] Also, for example, in the example of the correspondence table 70 illustrated in FIG. 4, the parts “Spine4,” “Spine5,” “Spine6,” etc., in the skeletal information 60A are associated with “None,” and it is estimated that there are no corresponding parts.

[0037] As illustrated above, the estimator 125 according to the present embodiment can automatically output the correspondence between parts in pieces of skeletal information compliant with different standards, thereby improving the efficiency of various types of processing and operations that use skeletal information.

[0038] The functional configuration including the estimator 125, learning performed by the estimator 125, various types of processing using the correspondence table 70, etc., will be described in detail below.

[0039] <<1.2. Example of Functional Configuration>> FIG. 5 is a block diagram illustrating an example of the functional configuration of an information processing system 1 that performs the information processing method according to the present embodiment.

[0040] As illustrated in FIG. 5, the information processing system 1 according to the present embodiment includes at least an information processing apparatus 10.

[0041] (Information Processing Apparatus 10) The information processing apparatus 10 may include a storage section 110, an estimation section 120, a conversion section 130, a search section 140, an interpolation section 150, and an interface control section 160, for example.

[0042] (Storage Section 110) The storage section 110 stores various types of information used by the information processing apparatus 10.

[0043] The storage section 110 stores motion data 50, a 3D model 55, skeletal information 60, the correspondence table 70, standard skeletal information 65 (see FIG. 10) to be described later, and the like, for example.

[0044] (Estimation Section 120) The estimation section 120 uses the estimator 125 to estimate, on the basis of two different pieces of skeletal information 60, the correspondence between the parts defined in the two respective pieces of skeletal information 60.

[0045] The functions of the estimation section 120 are implemented by cooperation of various processors and memories. The functions of the estimation section 120 will be described in detail later.

[0046] (Conversion Section 130) The conversion section 130 makes the two different pieces of skeletal information 60 compatible with each other on the basis of the result of estimation by the estimation section 120.

[0047] Here, making the two pieces of skeletal information 60 compatible with each other includes converting one of the two different pieces of skeletal information 60 such that the one conforms to the standard of the other. A unified standard refers to a predefined skeletal information format (e.g., a standard skeletal information specification as shown in FIG. 10) that ensures consistent part naming, structure, and relational data across different 3D models or motion data."

[0048] Further, making the two pieces of skeletal information 60 compatible with each other also includes converting (standardizing) both of the two different pieces of skeletal information 60 such that the two conform to a given standard.

[0049] The functions of the conversion section 130 are implemented by cooperation between various processors and memories. The functions of the conversion section 130 will be described in detail later.

[0050] (Search Section 140) The search section 140 performs a search on the basis of the skeletal information 60 standardized by the conversion section 130.

[0051] The functions of the search section 140 are implemented by cooperation of various processors and memories. The functions of the search section 140 will be described in detail later.

[0052] (Interpolation Section 150) On the basis of the two different pieces of skeletal information 60 that have been made compatible with each other by the conversion section 130, the interpolation section 150 performs interpolation between two pieces of motion data 50 relating to the two pieces of skeletal information 60.

[0053] The functions of the interpolation section 150 are implemented by cooperation of various processors and memories. The functions of the interpolation section 150 will be described in detail later.

[0054] (Interface Control Section 160) The interface control section 160 controls a user interface related to various functions that the information processing system 1 provides to a user.

[0055] The functions of the interface control section 160 are implemented by cooperation between various processors and memories.

[0056] The example of the functional configuration of the information processing system 1 according to the present embodiment has been described above. However, the functional configuration described above with reference to FIG. 5 is merely an example, and the functional configuration of the information processing system 1 according to the present embodiment is not limited to this example.

[0057] For example, the storage section 110, the estimation section 120, the conversion section 130, the search section 140, the interpolation section 150, and the interface control section 160 described above do not necessarily need to be provided in a single apparatus.

[0058] The storage section 110, the estimation section 120, the conversion section 130, the search section 140, the interpolation section 150, and the interface control section 160 may be provided in a distributed manner in multiple apparatuses.

[0059] In addition, the information processing system 1 according to the present embodiment may further include an operation receiving section that receives operations made by a user, a display section that displays various types of information, and the like.

[0060] The information processing system 1 according to the present embodiment can flexibly be modified according to the specifications, operations, and the like.

[0061] <<1.3. Learning by Estimator 125>> Next, the learning performed by the estimator 125 according to the present embodiment will be described. FIG. 6 is a diagram for illustrating the learning performed by the estimator 125 according to the present embodiment.

[0062] The estimator 125 according to the present embodiment is generated by machine learning.

[0063] The estimator 125 according to the present embodiment may include both a natural language processing model and a graph neural network (GNN) 127, or either the natural language processing model or the GNN.

[0064] Transformer 129 is an example of the natural language processing model.

[0065] The learning performed by the estimator 125 according to the present embodiment is roughly divided into pre-learning and additional learning.

[0066] First, in pre-learning, the estimator 125 performs self-supervised learning on the basis of a skeletal knowledge 67 having been input thereto.

[0067] The skeletal knowledge 67 is information including knowledge regarding skeletons of people, animals, and the like. That is, skeletal knowledge 67 includes anatomical data (e.g., bone names, joint structures) and linguistic data (e.g., synonyms for skeletal terms like 'pelvis' and 'hips') collected from sources such as the Internet or electronic books.

[0068] The skeletal knowledge 67 may be collected from the Internet, or may be extracted from electronic books, for example.

[0069] The estimator 125 according to the present embodiment learns the meaning of words related to the skeletons through the learning based on the skeletal knowledge 67.

[0070] For example, in the self-supervised learning, the estimator 125 learns to output, when receiving such input as “(?) is the name of the bone located in the hips,” ““pelvis,” “Kotsuban (in Japanese)”” as the above (?).

[0071] Through the above-mentioned task, the estimator 125 can acquire such concepts as ““pelvis” and “Kotsuban (in Japanese)” have the same meaning” and ““pelvis” is closely related to “hips,”” for example.

[0072] In the subsequent additional learning, the estimator 125 performs supervised learning to output the correspondence table 70 appropriate for the two pieces of skeletal information 60 having been input thereto.

[0073] For example, as illustrated in FIG. 6, the estimator 125 performs learning such that the correspondence table 70 to be output for the skeletal information 60A and 60B having been input to the estimator 125 approaches a correspondence table 70G, which is the given correct answer (Ground Truth).

[0074] Through such supervised learning, the estimator 125 can acquire the ability to estimate, on the basis of the two pieces of skeletal information 60 having been input thereto, the correspondence between the parts defined in the two respective pieces of skeletal information 60.

[0075] The learning performed by the estimator 125 according to the present embodiment has been described above by giving the example.

[0076] According to the above-mentioned learning flow, the concept base acquired in the pre-learning can further be tuned by real tasks in the additional learning, and thus, more effective learning can be performed.

[0077] However, the above-mentioned pre-learning is not essential, and the estimator 125 may only perform the supervised learning which has been described above as the additional learning.

[0078] Note that the supervised learning described above as the additional learning can be performed by using only either the natural language processing model or the GNN 127.

[0079] <<1.4. Processing Using Correspondence Table 70>> Next, each piece of processing using the correspondence table 70 according to the present embodiment will be described.

[0080] As described above, the estimator 125 according to the present embodiment estimates the correspondence between the parts defined in the two respective pieces of skeletal information 60 having been input to the estimator 125, and outputs the correspondence table 70 as the result of the estimation.

[0081] Further, the conversion section 130 according to the present embodiment makes the two different pieces of skeletal information 60 compatible with each other on the basis of the correspondence table 70.

[0082] According to the processing, the efficiency of each piece of processing in which the standards of the skeletal information 60 are assumed to be unified can be improved.

[0083] First, retargeting will be described. The retargeting is processing of adapting the motion data 50 to the 3D model 55.

[0084] Here, in the case where the standards of the skeletal information 60 of the motion data 50 and the skeletal information 60 of the 3D model 55 are different, the retargeting requires processing (operation) called remapping for associating the parts defined in the two respective pieces of skeletal information 60 with each other.

[0085] FIG. 7 is a diagram for illustrating the efficiency improvement of the remapping according to the present embodiment.

[0086] As illustrated on the left side of FIG. 7, existing remapping requires a user to select, one by one, correspondence between a certain part in the skeletal information 60 of the motion data 50 (motion skeleton) and a certain part in the skeletal information 60 of the 3D model 55 (model skeleton), which is cumbersome.

[0087] On the other hand, with the correspondence table 70 according to the present embodiment, as illustrated on the right side of FIG. 7, the above-mentioned work can be automated, and the result of the estimation of the correspondence between the parts can be presented to the user, thereby greatly improving the efficiency of the remapping.

[0088] Note that the user may check the presented estimation result and correct the result, if necessary. In addition, the correction history may be used for re-learning the estimator 125, or the like.

[0089] In the case where the user gives an instruction by, for example, pressing a button after the user checks and corrects the estimation result, the conversion section 130 according to the present embodiment may perform the retargeting on the basis of the correspondence table 70.

[0090] Note that, as described with reference to FIGS. 2 to 4, the estimator 125 according to the present embodiment can also estimate that there is no corresponding part. That is, the two pieces of skeletal information 60 used for the remapping and retargeting according to the present embodiment may have different numbers of parts.

[0091] In addition, in the above example, the case where the motion data 50 and the 3D model 55 are of a bipedal walking type is illustrated, but the types of the motion data 50 and the 3D model 55 according to the present embodiment are not limited to this example.

[0092] FIG. 8 is a diagram for illustrating output of the correspondence table 70 based on the skeletal information 60A of the bipedal walking type and skeletal information 60C of a quadrupedal walking type according to the present embodiment.

[0093] The skeletal information 60A illustrated in FIG. 8 is the skeletal information 60 extracted from the motion data 50A of the bipedal walking type, and the skeletal information 60C is the skeletal information 60 extracted from a 3D model 55C of the quadrupedal walking type.

[0094] The estimator 125 according to the present embodiment is also capable of estimating the correspondence between parts defined in the two respective pieces of skeletal information 60 having different walking types as described above, and outputting the correspondence table 70 as the result of the estimation.

[0095] Further, as illustrated in FIG. 9, the skeletal information 60 according to the present embodiment may define some parts of the entire body.

[0096] Skeletal information 60D illustrated in FIG. 9 is the skeletal information 60 extracted from motion data 50D of the upper body of a human body, and skeletal information 60E is the skeletal information 60 extracted from a 3D model 55E of the lower body of a human body.

[0097] The estimator 125 according to the present embodiment is also capable of estimating the correspondence between parts defined in the two respective pieces of skeletal information 60, the two pieces of skeletal information 60 being different from each other and defining some parts of the entire body as described above, and outputting the correspondence table 70 as the result of the estimation.

[0098] For example, in the example illustrated in FIG. 9, the motions of both arms can be applied to both legs.

[0099] Next, standardization of the skeletal information 60 according to the present embodiment and a search for the motion data 50 based on the standardized skeletal information 60 will be described.

[0100] In recent years, a technology for searching for, with respect to a certain motion, another motion similar to the certain motion, which is known as a motion-to-motion search, has been developed.

[0101] As the motion-to-motion search, for example, there is a method for calculating feature quantities for each period of time from time-series data of the skeletal information 60 of the motion data 50 to be searched for, applying weighting parameters to the feature quantities, and obtaining, on the basis of the processed feature quantities, motion data 50 similar to the search target from a database (DB).

[0102] The above-mentioned weighting parameters can be determined by an estimator obtained by learning the relation between the above-mentioned feature quantities for each period of time and weighting parameters for each period of time.

[0103] However, the above-mentioned search method is performed on the assumption that the skeletal information 60 of the motion data 50 to be searched for and the skeletal information 60 of the motion data 50 stored in the database have a unified standard.

[0104] For this reason, in the past, in the case where the standard of skeletal information 60 of the motion data 50 to be searched for differs from the standard of the skeletal information 60 of the motion data 50 stored in the database, the search could not be performed.

[0105] The information processing method according to the present embodiment solves the above-mentioned problem by standardizing the standard of the skeletal information 60 of the motion data 50 to be searched for so as to match the standard with the standard of the skeletal information 60 of the motion data 50 stored in the database.

[0106] FIG. 10 is a diagram for illustrating the standardization of the skeletal information 60 according to the present embodiment.

[0107] The motion data 50A illustrated in FIG. 10 is the motion data 50 to be searched for. Further, the skeletal information 60A is the skeletal information 60 extracted from the motion data 50A, and is an example of target skeletal information.

[0108] The estimator 125 outputs the correspondence table 70 on the basis of the skeletal information 60A and the standard skeletal information 65 having been input thereto.

[0109] The standard skeletal information 65 is skeletal information that is compliant with the same standard as the standard (standard specification) of the skeletal information 60 of the motion data 50 stored in a motion DB 20 (see FIG. 11).

[0110] That is, the correspondence table 70 output by the estimator 125 in this example is an estimation result indicating the correspondence between parts that are compliant with any standard in the target skeletal information and parts that are compliant with the standard specification.

[0111] The conversion section 130 according to the present embodiment converts the skeletal information 60A on the basis of the correspondence table 70. That is, the conversion section 130 standardizes the skeletal information 60A such that the skeletal information 60A matches the standard skeletal information 65, and obtains standardized skeletal information 60AS.

[0112] Next, a search based on the standardized skeletal information 60AS will be described with reference to FIG. 11.

[0113] The search section 140 according to the present embodiment performs a search on the basis of the standardized skeletal information 60AS.

[0114] The search section 140 according to the present embodiment calculates feature quantities for each period of time from time-series data of the standardized skeletal information 60AS, applies weighting parameters to the feature quantities, and obtains a search result 52 from the motion DB 20 on the basis of the processed feature quantities, for example.

[0115] The search result 52 may be the motion data 50 including the skeletal information 60 having feature quantities that are highly similar to the standardized skeletal information 60AS.

[0116] Next, with reference to FIG. 12, an example of the configuration of the information processing system 1 in the case of performing the above-mentioned motion-to-motion search will be described.

[0117] In this case, as illustrated in FIG. 12, the information processing system 1 includes the motion DB 20 and a client 30 in addition to the information processing apparatus 10.

[0118] The client 30 is a computer terminal used by a user to perform a search, and may be a personal computer (PC), for example.

[0119] In the client 30, the user uploads the motion data 50 that is a search target to the information processing apparatus 10 by using a user interface controlled by the interface control section 160.

[0120] The information processing apparatus 10 performs a series of steps of processing described with reference to FIGS. 10 and 11 on the motion data 50 uploaded from the client 30, to obtain the search result 52, and sends the search result 52 back to the client 30.

[0121] The information processing apparatus 10 and the motion DB 20 may be constructed on a cloud, for example. Further, the number of clients 30 is not limited to a specific number.

[0122] It should be noted that all or part of the series of steps of processing described with reference to FIGS. 10 and 11 may be performed by the client 30.

[0123] For example, the client 30 may extract the skeletal information 60 from the motion data 50 to be searched for, and upload the extracted skeletal information 60 to the information processing apparatus 10.

[0124] Also, for example, the client 30 may perform up to the standardization based on the correspondence table 70 and upload the standardized skeletal information 60 to the information processing apparatus 10.

[0125] The standardization of the skeletal information 60 according to the present embodiment and the search for the motion data 50 based on the standardized skeletal information 60 have been described above.

[0126] According to the standardization method as described above, it is possible not only to perform the search but also to apply, to a system conforming to only a certain standard, the motion data 50 or the 3D model 55 compliant with another standard.

[0127] Next, motion interpolation based on the correspondence table 70 according to the present embodiment will be described.

[0128] In production of 3D animations, in the case of connecting one piece of motion data 50 with another piece of motion data 50, interpolation processing (motion interpolation) may be performed.

[0129] Techniques for the motion interpolation include motion linking, motion blending, generation of interpolated motion data, and the like, for example.

[0130] The motion linking is processing of simply linking one piece of motion data 50 with another piece of motion data 50. At this time, in order to link the pieces of motion data together more naturally, there are cases where processing is performed in which part of a section included in the one piece of motion data 50 is modified to match the other piece of motion data 50.

[0131] The motion blending is processing of linearly and geometrically combining one piece of motion data 50 with another piece of motion data 50. In the motion blending, in an interpolation section where the one piece of motion data 50 overlaps with the other piece of motion data 50, interpolation is performed such that a movement included in the one piece of motion data 50 gradually changes to a movement included in the other piece of motion data 50.

[0132] Further, the generation of interpolated motion data is a technique for newly generating motion data (interpolated motion data) for interpolation between one piece of motion data 50 and another piece of motion data 50. The interpolated motion data may be generated by supervised learning using deep learning or the like, for example.

[0133] However, any of the above-mentioned techniques is performed on the assumption that the standards of the skeletal information 60 of the two pieces of motion data 50 are unified.

[0134] Therefore, the information processing method according to the present embodiment implements interpolation between pieces of motion data 50 compliant with different standards by making pieces of skeletal information 60 of the two pieces of target motion data 50 compatible with each other on the basis of the correspondence table 70.

[0135] FIG. 13 is a diagram for illustrating the motion interpolation based on the correspondence table 70 according to the present embodiment.

[0136] In the case of the example illustrated in FIG. 13, the skeletal information 60A extracted from the motion data 50A and the skeletal information 60B extracted from motion data 50B are input to the estimator 125.

[0137] The estimator 125 outputs, on the basis of the skeletal information 60A and the skeletal information 60B having been input thereto, the correspondence table 70 indicating the correspondence between the parts defined in the skeletal information 60A and the parts defined in the skeletal information 60B.

[0138] Next, in the case of this example, the motion data 50B, the skeletal information 60B, and the correspondence table 70 are input to the conversion section 130.

[0139] The conversion section 130 converts the skeletal information 60B on the basis the correspondence table 70 such that the skeletal information 60B conforms to the standard of the skeletal information 60A. Further, the conversion section 130 converts the motion data 50B by using skeletal information 60Ba (not illustrated) obtained by converting the skeletal information 60B to conform to the standard of the skeletal information 60A, thereby obtaining motion data 50Ba.

[0140] The motion data 50Ba is the motion data 50 having the skeletal information 60Ba that is compliant with the same standard as the skeletal information 60A of the motion data 50A.

[0141] The motion data 50Ba is input to the interpolation section 150 together with the motion data 50A.

[0142] The interpolation section 150 performs interpolation between the motion data 50A and the motion data 50Ba having been input thereto.

[0143] The motion interpolation performed by the interpolation section 150 may be any of the motion linking, the motion blending, and the generation of interpolated motion data.

[0144] For example, the interpolation section 150 may generate interpolated motion data 75AB for interpolation between the motion data 50A and the motion data 50Ba having been input thereto.

[0145] It is to be noted that, although FIG. 13 illustrates a case where the motion data 50B is converted such that the motion data 50B matches the motion data 50A, the motion data 50A may also be converted to match the motion data 50B, or the motion data 50A and the motion data 50B may be converted (standardized) by using the standard skeletal information 65.

[0146] <2. Example of Hardware Configuration> Next, an example of the hardware configuration of an information processing apparatus 90 according to the one embodiment of the present disclosure will be described. FIG. 14 is a block diagram illustrating the example of the hardware configuration of the information processing apparatus 90 according to the one embodiment of the present disclosure.

[0147] The information processing apparatus 90 may be an apparatus having the hardware configuration equivalent to that of the information processing apparatus 10.

[0148] As illustrated in FIG. 14, the information processing apparatus 90 has a processor 871, a read-only memory (ROM) 872, a random-access memory (RAM) 873, a host bus 874, a bridge 875, an external bus 876, an interface 877, an input device 878, an output device 879, a storage 880, a drive 881, a connection port 882, and a communication device 883, for example. Note that the hardware configuration illustrated here is just an example, and some of the components may be omitted. Further, the information processing apparatus 90 may further include components other than those illustrated here. The functionality of the elements disclosed herein may be implemented using circuitry or processing circuitry which includes general purpose processors, special purpose processors, integrated circuits, ASICs (“Application Specific Integrated Circuits”), FPGAs (“Field-Programmable Gate Arrays”), conventional circuitry and / or combinations thereof which are programmed, using one or more programs stored in one or more memories, or otherwise configured to perform the disclosed functionality. Processors and controllers are considered processing circuitry or circuitry as they include transistors and other circuitry therein. In the disclosure, the circuitry, units, or means are hardware that carry out or are programmed to perform the recited functionality. The hardware may be any hardware disclosed herein which is programmed or configured to carry out the recited functionality. There is a memory that stores a computer program which includes computer instructions. These computer instructions provide the logic and routines that enable the hardware (e.g., processing circuitry or circuitry) to perform the method disclosed herein. This computer program can be implemented in known formats as a computer-readable storage medium, a computer program product, a memory device, a record medium such as a CD-ROM or DVD, and / or the memory of a FPGA or ASIC.

[0149] (Processor 871) The processor 871 functions as an arithmetic processing device or a control device, for example, and controls all or some of the operations of the respective components on the basis of various programs recorded in the ROM 872, the RAM 873, the storage 880, or a removable storage medium 901.

[0150] (ROM 872 and RAM 873) The ROM 872 is a device for storing programs to be loaded into the processor 871, data to be used for calculations, etc. The RAM 873 temporarily or permanently stores programs to be loaded into the processor 871, various parameters that change appropriately when the programs are executed, and the like, for example.

[0151] (Host Bus 874, Bridge 875, External Bus 876, and Interface 877) The processor 871, the ROM 872, and the RAM 873 are connected to one another via the host bus 874 capable of high-speed data transmission, for example. Meanwhile, the host bus 874 is connected via the bridge 875 to the external bus 876 having a relatively low data transmission speed, for example. In addition, the external bus 876 is connected to various components via the interface 877.

[0152] (Input Device 878) As the input device 878, a mouse, a keyboard, a touch panel, a button, a switch, or a lever is used, for example. Further, as the input device 878, a remote controller (hereinafter referred to as a remocon) capable of transmitting control signals by using infrared rays or other radio waves may also be used. In addition, the input device 878 includes an audio input device such as a microphone.

[0153] (Output Device 879) The output device 879 is a device capable of visually or audibly notifying a user of obtained information. The output device 879 is, for example, a display device such as a cathode ray tube (CRT), a liquid crystal display (LCD), or organic electroluminescent (EL), or an audio output device such as a speaker or headphones, or is a printer, a mobile phone, a facsimile, or the like. Additionally, the output device 879 according to the embodiment of the present disclosure includes various vibration devices capable of outputting tactile stimuli.

[0154] (Storage 880) The storage 880 is a device for storing various types of data. As the storage 880, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like is used, for example.

[0155] (Drive 881) The drive 881 is, for example, a device for reading information recorded in the removable storage medium 901 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, or writing information to the removable storage medium 901.

[0156] (Removable Storage Medium 901) The removable storage medium 901 is, for example, a digital versatile disc (DVD) medium, a Blu-ray (registered trademark) medium, a high-definition (HD) DVD medium, any of various semiconductor storage media, or the like. Needless to say, the removable storage medium 901 may be an integrated circuit (IC) card equipped with a contactless IC chip, an electronic device, or the like, for example.

[0157] (Connection Port 882) The connection port 882 is a port for the connection to an externally connected device 902, such as a universal serial bus (USB) port, an Institute of Electrical and Electronics Engineers (IEEE)1394 port, a small computer system interface (SCSI), a Recommended Standard (RS)-232C port, or an optical audio terminal.

[0158] (Externally Connected Device 902) The externally connected device 902 is a printer, a portable music player, a digital camera, a digital video camera, an IC recorder, or the like, for example.

[0159] (Communication Device 883) The communication device 883 is a communication device for the connection to a network and is, for example, a wired or wireless local area network (LAN), Bluetooth (registered trademark), a communication card for a wireless USB (WUSB), a router for optical communication, a router for an asymmetric digital subscriber line (ADSL), a modem for various types of communication, or the like.

[0160] <3. Summary> As described above, the information processing system 1 according to the one embodiment of the present disclosure includes the estimation section 120 that estimates, on the basis of two different pieces of skeletal information 60, the correspondence between parts defined in the two respective pieces of skeletal information 60, and the conversion section 130 that makes the two different pieces of skeletal information 60 compatible with each other on the basis of the result of the estimation by the estimation section 120.

[0161] According to the above-mentioned configuration, the efficiency of various types of processing and operations that use skeletal information can be improved.

[0162] Although the preferred embodiment of the present disclosure has been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to this example. It is clear that a person with ordinary knowledge in the technical field of the present disclosure can think of various modified or altered examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0163] For example, in the above-mentioned example, the case where the skeletal information 60 is related to a 3D animation is illustrated, but the skeletal information 60 may be related to a robot that is present in the real world.

[0164] Further, the technical idea according to the one embodiment of the present disclosure is applicable to various types of information described in a graph structure, in addition to the skeletal information 60.

[0165] In addition, the steps related to the processing described in the present disclosure do not necessarily have to be performed in chronological order according to the order depicted in the flowcharts or sequence diagrams. For example, the steps related to the processing by each of the apparatuses may be performed in an order different from that described herein, or may be performed in parallel.

[0166] Moreover, the series of steps of processing performed by each of the apparatuses described in the present disclosure may be executed by a program stored in a non-transitory computer readable storage medium. Each program is loaded into a RAM when executed by a computer, and is executed by a processor such as a central processing unit (CPU), for example. The above-mentioned storage medium is a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like, for example. Also, the above-mentioned program may be distributed via a network, for example, without using a storage medium.

[0167] Furthermore, the effects described in the present specification are merely explanatory or exemplary, and are not restrictive. In other words, the technology according to the present disclosure may produce other effects that are apparent to a person skilled in the art from the description of the present specification, in addition to or in place of the above-mentioned effects.

[0168] Note that the following configurations also fall within the technical scope of the present disclosure. (1) An information processing system for producing a three-dimensional (3D) animation, comprising: circuitry configured to: receive two different pieces of skeletal information, each defining a structure of a 3D model or motion data for the 3D animation, the skeletal information including parts and positional relationships between the parts, wherein the parts include at least one of a joint, an extremity, a bone and / or other skeletal segments; generate, using a machine-learning-based estimator trained on skeletal knowledge, a correspondence table defining a correspondence between the parts of the two different pieces of skeletal information; transform, based on the correspondence table, at least one of the two different pieces of skeletal information to generate compatible skeletal information conforming to a unified standard; and apply the compatible skeletal information to produce a 3D animation by adapting the motion data to the 3D model or interpolating between pieces of the motion data. (2) The information processing system according to (1), wherein the machine-learning-based estimator includes at least one of a natural language processing model or a graph neural network trained to process skeletal information. (3) The information processing system according to (1), wherein the circuitry is configured to train the machine-learning-based estimator using self-supervised learning based on the skeletal knowledge comprising information about skeletons of people or animals. (4) The information processing system according to (1), wherein the circuitry is further configured to estimate the correspondence between parts defined in skeletal information of a bipedal walking type and skeletal information of a quadrupedal walking type. (5) The information processing system according to (1), wherein the circuitry is further configured to estimate the correspondence between parts defined in skeletal information representing only a portion of an entire body. (6) The information processing system according to (1), wherein the circuitry transforms both of the two different pieces of skeletal information to conform to a predefined standard skeletal information specification. (7) The information processing system according to (1), wherein the circuitry is further configured to search for the motion data in a database based on the compatible skeletal information. (8) The information processing system according to (7), wherein the circuitry is further configured to calculate feature quantities from time-series data of the compatible skeletal information, apply weighting parameters to the feature quantities, and retrieve the motion data from the database based on the feature quantities. (9) The information processing system according to (1), wherein the circuitry is further configured to perform interpolation between two pieces of the motion data by generating interpolated motion data based on the compatible skeletal information. (10) The information processing system according to (9), wherein the circuitry is further configured to perform the interpolation by motion linking, motion blending, or generating interpolated motion data using a deep learning model. (11) The information processing system according to (1), wherein the skeletal information includes names of the parts, and the correspondence table maps part names between the two different pieces of skeletal information. (12) The information processing system according to (1), wherein the correspondence table indicates when a part in one piece of skeletal information has no corresponding part in an other piece of skeletal information. (13) The information processing system according to (1), wherein the two different pieces of skeletal information are extracted from the 3D model and the motion data, respectively. (14) An information processing method for producing a three-dimensional (3D) animation, comprising: receiving, by a circuitry, two different pieces of skeletal information, each defining a structure of a 3D model or motion data for the 3D animation, the skeletal information including parts and positional relationships between the parts, wherein the parts include joint, extremity, bones and / or other skeletal segments; generating, by the circuitry using a machine-learning-based estimator trained on skeletal knowledge, a correspondence table defining a correspondence between the parts of the two different pieces of skeletal information; transforming, by the circuitry based on the correspondence table, at least one of the two different pieces of skeletal information to generate compatible skeletal information conforming to a unified standard; and applying, by the circuitry, the compatible skeletal information to produce a 3D animation by adapting the motion data to the 3D model or interpolating between pieces of the motion data, thereby automating compatibility in the 3D animation production process. (15) The information processing method according to (14), further comprising performing, by the circuitry, a search for the motion data in a motion database based on the compatible skeletal information by calculating feature quantities and applying weighting parameters. (16) The information processing method according to (14), further comprising performing, by the circuitry, interpolation between two pieces of the motion data by generating interpolated motion data based on the compatible skeletal information. (17) The information processing method according to (16), further comprising, by the circuitry, performing the interpolation by motion linking, motion blending, or generating interpolated motion data using a deep learning model. (18) The information processing method according to (14), further comprising, by the circuitry: searching for the motion data in a database based on the compatible skeletal information; and calculating feature quantities from time-series data of the compatible skeletal information, applying weighting parameters to the feature quantities, and retrieving the motion data from the database based on the feature quantities. (19) A non-transitory computer-readable storage medium storing a program that, when executed by a processor, causes a computer to perform a method for producing a three-dimensional (3D) animation, the method comprising: receiving two different pieces of skeletal information, each defining a structure of a 3D model or motion data for the 3D animation, the skeletal information including parts and positional relationships between the parts, wherein the parts include joint, extremity, bones and / or other skeletal segments; generating, using a machine-learning-based estimator trained on skeletal knowledge, a correspondence table defining a correspondence between the parts of the two different pieces of skeletal information; transforming, based on the correspondence table, at least one of the two different pieces of skeletal information to generate compatible skeletal information conforming to a unified standard; and applying the compatible skeletal information to produce a 3D animation by adapting the motion data to the 3D model or interpolating between pieces of motion data, thereby automating compatibility in the 3D animation production process. (20) The non-transitory computer-readable storage medium according to (19), wherein the method further comprises: searching for motion data in a database based on the compatible skeletal information; and calculating feature quantities from time-series data of the compatible skeletal information, applying weighting parameters to the feature quantities, and retrieving motion data from the database based on the feature quantities. (21) An information processing system including: an estimation section that estimates, on the basis of two different pieces of skeletal information, correspondence between parts defined in the two respective pieces of skeletal information; and a conversion section that makes the two different pieces of skeletal information compatible with each other on the basis of a result of the estimation by the estimation section. (22) The information processing system according to (21) above, further including: a search section that performs a search on the basis of the pieces of skeletal information that have been made compatible with each other by the conversion section. (23) The information processing system according to (22) above, in which the estimation section estimates correspondence between a part defined in standard skeletal information and a part defined in target skeletal information, on the basis of the standard skeletal information and the target skeletal information. (24) The information processing system according to (23) above, in which the conversion section standardizes the target skeletal information so as to match the target skeletal information with the standard skeletal information, on the basis of a result of the estimation by the estimation section. (25) The information processing system according to (24) above, in which the search section performs a search on the basis of the skeletal information standardized by the conversion section. (26) The information processing system according to any one of (21) to (25) above, in which the skeletal information is extracted from motion data of a 3D animation. (27) The information processing system according to (26) above, further including: an interpolation section that performs, on the basis of the two different pieces of skeletal information that have been made compatible with each other by the conversion section, interpolation between two pieces of motion data related to the two pieces of skeletal information. (28) The information processing system according to any one of (21) to (27) above, in which the estimation section estimates correspondence between a part defined in skeletal information extracted from motion data of a 3D animation and a part defined in skeletal information extracted from a 3D model. (29) The information processing system according to (28) above, in which the conversion section performs retargeting on the basis of a result of the estimation by the estimation section. (30) The information processing system according to any one of (21) to (29) above, in which the estimation section estimates the correspondence between the parts defined in the two respective pieces of skeletal information, by using an estimator generated by machine learning. (31) The information processing system according to (30) above, in which the estimator includes at least one of a natural language processing model and a graph neural network. (32) The information processing system according to (31) above, in which the natural language processing model includes Transformer. (33) An information processing method including: estimating, on the basis of two different pieces of skeletal information, correspondence between parts defined in the two respective pieces of skeletal information; and making the two different pieces of skeletal information compatible with each other on the basis of a result of the estimation. (34) The information processing method according to (33) above, further including: performing a search on the basis of the pieces of skeletal information that have been made compatible with each other. (35) The information processing method according to (34) above, in which correspondence between a part defined in standard skeletal information and a part defined in target skeletal information is estimated on the basis of the standard skeletal information and the target skeletal information. (36) The information processing method according to (35) above, in which the target skeletal information is standardized so as to match the target skeletal information with the standard skeletal information, on the basis of a result of the estimation. (37) The information processing method according to any one of (33) to (36) above, in which the skeletal information is extracted from motion data of a 3D animation. (38) The information processing method according to (37) above, further including: performing, on the basis of the two different pieces of skeletal information that have been made compatible with each other by the conversion section, interpolation between two pieces of motion data related to the two pieces of skeletal information. (39) The information processing method according to any one of (33) to (38) above, in which correspondence between a part defined in skeletal information extracted from motion data of a 3D animation and a part defined in skeletal information extracted from a 3D model is estimated. (40) The information processing method according to (39) above, further including: performing retargeting on the basis of a result of the estimation.

[0169] 1: Information processing system 10: Information processing apparatus 110: Storage section 120: Estimation section 125: Estimator 127: GNN 129: Transformer 130: Conversion section 140: Search section 150: Interpolation section 160: Interface control section 20: Motion DB 30: Client 50: Motion data 55: 3D model 60: Skeletal information 65: Standard skeletal information 70: Correspondence table

Claims

1. An information processing system for producing a three-dimensional (3D) animation, comprising: circuitry configured to: receive two different pieces of skeletal information, each defining a structure of a 3D model or motion data for the 3D animation, the skeletal information including parts and positional relationships between the parts, wherein the parts include at least one of a joint, an extremity, a bone and / or other skeletal segments; generate, using a machine-learning-based estimator trained on skeletal knowledge, a correspondence table defining a correspondence between the parts of the two different pieces of skeletal information; transform, based on the correspondence table, at least one of the two different pieces of skeletal information to generate compatible skeletal information conforming to a unified standard; and apply the compatible skeletal information to produce a 3D animation by adapting the motion data to the 3D model or interpolating between pieces of the motion data.

2. The information processing system according to claim 1, wherein the machine-learning-based estimator includes at least one of a natural language processing model or a graph neural network trained to process skeletal information.

3. The information processing system according to claim 1, wherein the circuitry is configured to train the machine-learning-based estimator using self-supervised learning based on the skeletal knowledge comprising information about skeletons of people or animals.

4. The information processing system according to claim 1, wherein the circuitry is further configured to estimate the correspondence between parts defined in skeletal information of a bipedal walking type and skeletal information of a quadrupedal walking type.

5. The information processing system according to claim 1, wherein the circuitry is further configured to estimate the correspondence between parts defined in skeletal information representing only a portion of an entire body.

6. The information processing system according to claim 1, wherein the circuitry transforms both of the two different pieces of skeletal information to conform to a predefined standard skeletal information specification.

7. The information processing system according to claim 1, wherein the circuitry is further configured to search for the motion data in a database based on the compatible skeletal information.

8. The information processing system according to claim 7, wherein the circuitry is further configured to calculate feature quantities from time-series data of the compatible skeletal information, apply weighting parameters to the feature quantities, and retrieve the motion data from the database based on the feature quantities.

9. The information processing system according to claim 1, wherein the circuitry is further configured to perform interpolation between two pieces of the motion data by generating interpolated motion data based on the compatible skeletal information.

10. The information processing system according to claim 9, wherein the circuitry is further configured to perform the interpolation by motion linking, motion blending, or generating interpolated motion data using a deep learning model.

11. The information processing system according to claim 1, wherein the skeletal information includes names of the parts, and the correspondence table maps part names between the two different pieces of skeletal information.

12. The information processing system according to claim 1, wherein the correspondence table indicates when a part in one piece of skeletal information has no corresponding part in an other piece of skeletal information.

13. The information processing system according to claim 1, wherein the two different pieces of skeletal information are extracted from the 3D model and the motion data, respectively.

14. An information processing method for producing a three-dimensional (3D) animation, comprising: receiving, by a circuitry, two different pieces of skeletal information, each defining a structure of a 3D model or motion data for the 3D animation, the skeletal information including parts and positional relationships between the parts, wherein the parts include joint, extremity, bones and / or other skeletal segments; generating, by the circuitry using a machine-learning-based estimator trained on skeletal knowledge, a correspondence table defining a correspondence between the parts of the two different pieces of skeletal information; transforming, by the circuitry based on the correspondence table, at least one of the two different pieces of skeletal information to generate compatible skeletal information conforming to a unified standard; and applying, by the circuitry, the compatible skeletal information to produce a 3D animation by adapting the motion data to the 3D model or interpolating between pieces of the motion data, thereby automating compatibility in the 3D animation production process.

15. The information processing method according to claim 14, further comprising performing, by the circuitry, a search for the motion data in a motion database based on the compatible skeletal information by calculating feature quantities and applying weighting parameters.

16. The information processing method according to claim 14, further comprising performing, by the circuitry, interpolation between two pieces of the motion data by generating interpolated motion data based on the compatible skeletal information.

17. The information processing method according to claim 16, further comprising, by the circuitry, performing the interpolation by motion linking, motion blending, or generating interpolated motion data using a deep learning model.

18. The information processing method according to claim 14, further comprising, by the circuitry: searching for the motion data in a database based on the compatible skeletal information; and calculating feature quantities from time-series data of the compatible skeletal information, applying weighting parameters to the feature quantities, and retrieving the motion data from the database based on the feature quantities.

19. A non-transitory computer-readable storage medium storing a program that, when executed by a processor, causes a computer to perform a method for producing a three-dimensional (3D) animation, the method comprising: receiving two different pieces of skeletal information, each defining a structure of a 3D model or motion data for the 3D animation, the skeletal information including parts and positional relationships between the parts, wherein the parts include joint, extremity, bones and / or other skeletal segments; generating, using a machine-learning-based estimator trained on skeletal knowledge, a correspondence table defining a correspondence between the parts of the two different pieces of skeletal information; transforming, based on the correspondence table, at least one of the two different pieces of skeletal information to generate compatible skeletal information conforming to a unified standard; and applying the compatible skeletal information to produce a 3D animation by adapting the motion data to the 3D model or interpolating between pieces of motion data, thereby automating compatibility in the 3D animation production process.

20. The non-transitory computer-readable storage medium according to claim 19, wherein the method further comprises: searching for motion data in a database based on the compatible skeletal information; and calculating feature quantities from time-series data of the compatible skeletal information, applying weighting parameters to the feature quantities, and retrieving motion data from the database based on the feature quantities.

Images