Information processing system and information processing method

Abstract

To improve the efficiency of various processing and work using skeleton information.SOLUTION: There is provided an information processing system including an estimation unit configured to estimate, based on two different pieces of skeleton information, correspondence between parts defined in the respective two pieces of skeleton information, and a conversion unit configured to make the two different pieces of skeleton information compatible with each other based on a result of estimation by the estimation unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing system and an information processing method. [Background technology]

[0002] In recent years, the production of 3D (three-dimensional) animation has become popular, and as disclosed in Patent Document 1, for example, technology for supporting the production of 3D animation has been developed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7061238 Summary of the Invention [Problem to be solved by the invention]

[0004] The creation of 3D animations often involves the use of skeletal information for 3D models and motion data to make the 3D models perform specific motions. However, such skeletal information is generally defined independently by each creator, resulting in a lack of compatibility.

[0005] According to one aspect of the present disclosure, there is provided an information processing system including an estimation unit that estimates correspondences between parts defined in two different pieces of skeletal information based on the two pieces of skeletal information, and a conversion unit that makes the two different pieces of skeletal information mutually compatible based on the results of the estimation by the estimation unit.

[0006] According to another aspect of the present disclosure, there is provided an information processing method including: estimating correspondences between parts defined in two different pieces of skeletal information based on the two pieces of skeletal information; and making the two different pieces of skeletal information mutually compatible based on the results of the estimation. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram for explaining an overview of an embodiment of the present disclosure. [Figure 2] FIG. 10 is a diagram showing a specific example of skeleton information 60A according to the embodiment. [Figure 3] FIG. 10 is a diagram showing a specific example of skeletal information 60B according to the embodiment. [Figure 4] FIG. 10 is a diagram showing a specific example of a correspondence table 70 according to the embodiment. [Figure 5] 2 is a block diagram showing an example of a functional configuration of an information processing system 1 that implements an information processing method according to the embodiment. FIG. [Figure 6] 10 is a diagram for explaining learning of an estimator 125 according to the embodiment. FIG. [Figure 7] 10A and 10B are diagrams for explaining efficiency of remapping according to the embodiment; [Figure 8] 10 is a diagram for explaining output of a correspondence table 70 based on skeletal information 60A of a bipedal walking type and skeletal information 60C of a quadrupedal walking type according to the embodiment. FIG. [Figure 9] 10 is a diagram for explaining output of a correspondence table 70 based on skeletal information that defines some parts of the whole body according to the embodiment. FIG. [Figure 10] 10 is a diagram illustrating standardization of skeletal information 60 according to the embodiment. FIG. [Figure 11] 10 is a diagram illustrating a search based on standardized skeletal information 60AS according to the embodiment. FIG. [Figure 12] 10 is a diagram illustrating an example of the configuration of the information processing system 1 when performing a motion 2 motion search according to the embodiment. FIG. [Figure 13] 10 is a diagram illustrating motion interpolation based on a correspondence table 70 according to the embodiment. FIG. [Figure 14] FIG. 2 is a block diagram showing an example of the hardware configuration of an information processing device 90 according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0009] In addition, in this specification and drawings, when multiple identical components are to be described separately, letters or the like may be added to the end of the reference numerals. On the other hand, when it is not necessary to distinguish between multiple identical components, the letters or the like may be omitted and a description common to all of the multiple identical components may be given.

[0010] The explanation will be given in the following order. 1. Embodiment Overview 1.2. Functional configuration example 1.3. Training the estimator 125 1.4. Processing using the correspondence table 70 2. Hardware configuration example 3. Summary

[0011] <1. Embodiment> <<1.1. Overview>> First, an outline of one embodiment of the present disclosure will be described.

[0012] As mentioned above, skeletal information may be used in the production of 3D animation.

[0013] Skeletal information is information that defines the structure of the body (skeleton) of a 3D model.

[0014] The skeletal information includes, for example, information on parts corresponding to joints or extremities, names of parts, relationships between parts, and postures defined by the positional relationships of multiple parts.

[0015] The relationships between the above-mentioned parts include, for example, the positional relationships between the parts and the connection relationships between the parts.

[0016] In the skeletal information, by defining the positional relationships between parts and the connection relationships between parts for multiple parts, it is possible to define the posture that the 3D model should take and the changes in that posture, i.e., motion.

[0017] However, currently there is no unified standard for skeletal information, and skeletal information is generally defined independently by each creator.

[0018] For example, the number of body parts, names of body parts, etc. may differ between the skeletal information of one 3D model that resembles a human and the skeletal information of another 3D model that resembles a human.

[0019] As an example, a part defined by the name "hips" in one 3D model may be defined by the name "pelvis" in another 3D model.

[0020] As such, skeletal information created by different creators is often incompatible as is, which reduces the efficiency of various processes and tasks.

[0021] The technical idea according to an embodiment of the present disclosure was conceived with attention to the above points in mind, and aims to improve the efficiency of various processes and operations that use skeletal information.

[0022] For this reason, one of the features of an information processing method according to one embodiment of the present disclosure is that it uses an estimator 125 to estimate the correspondence between parts defined in each of two different pieces of skeletal information based on the two pieces of skeletal information.

[0023] FIG. 1 is a diagram illustrating an overview of an embodiment of the present disclosure.

[0024] The skeleton information according to this embodiment may be extracted from a 3D model or may be extracted from motion data.

[0025] In FIG. 1, it is assumed that skeletal information 60A is skeletal information extracted from motion data 50A, and skeletal information 60B is skeletal information extracted from 3D model 55B.

[0026] Furthermore, the skeleton information 60A and the skeleton information 60B illustrated in FIG. 1 are assumed to be compliant with different standards.

[0027] The estimator 125 according to this embodiment estimates the correspondence between the parts defined in each of the two pieces of skeletal information based on the two pieces of skeletal information conforming to the different standards as described above, and outputs the estimation result.

[0028] In the example shown in FIG. 1, the estimator 125 outputs a correspondence table 70 as a result of estimation based on the input skeletal information 60A and skeletal information 60B.

[0029] The correspondence table 70 is an example of information indicating the correspondence between the parts defined in each of the two pieces of input skeletal information.

[0030] 2 to 4, more specific examples of skeletal information 60A, skeletal information 60B, and correspondence table 70 are shown. Each element in skeletal information 60A, skeletal information 60B, and correspondence table 70 shown in FIGS. 2 to 4 indicates the name of a body part.

[0031] When the skeletal information 60A illustrated in FIG. 2 is compared with the skeletal information 60B illustrated in FIG. 3, it can be seen that the number of parts included in the skeletal information 60A and the skeletal information 60B is different.

[0032] Furthermore, when comparing the skeletal information 60A illustrated in FIG. 2 with the skeletal information 60B illustrated in FIG. 3, it is clear that there are cases where the name of a part that exists in one does not exist in the other.

[0033] In this way, even if the number of parts differs between pieces of skeletal information, or even if the names of the parts differ, the estimator 125 according to this embodiment can estimate the correspondence between the parts.

[0034] For example, in the example of the correspondence table 70 shown in FIG. 4, the part "LeftUpLeg" in the skeletal information 60A and the part "LeftHip" in the skeletal information 60B are estimated to be corresponding parts.

[0035] Also, for example, in the example of the correspondence table 70 shown in FIG. 4, the parts "Spine4", "Spine5", "Spine6", etc. in the skeletal information 60A are associated with "None", and it is presumed that there are no corresponding parts.

[0036] As illustrated above, the estimator 125 according to this embodiment can automatically output the correspondence between parts between skeletal information of different standards, thereby making it possible to improve the efficiency of various processes and operations that use skeletal information.

[0037] The functional configuration including the estimator 125, the learning of the estimator 125, various processes using the correspondence table 70, etc. will be described in detail below.

[0038] <<1.2. Functional configuration example>> FIG. 5 is a block diagram showing an example of the functional configuration of an information processing system 1 that implements the information processing method according to this embodiment.

[0039] As shown in FIG. 5, the information processing system 1 according to this embodiment includes at least an information processing device 10.

[0040] (Information processing device 10) The information processing device 10 may include, for example, a storage unit 110, an estimation unit 120, a conversion unit 130, a search unit 140, an interpolation unit 150, and an interface control unit 160.

[0041] (Storage unit 110) The storage unit 110 stores various types of information used by the information processing device 10.

[0042] The storage unit 110 stores, for example, motion data 50, a 3D model 55, skeleton information 60, a correspondence table 70, standard skeleton information 65 (see FIG. 10) described later, and the like.

[0043] (Estimation part 120) The estimation unit 120 uses the estimator 125 to estimate the correspondence between the parts defined in each of the two different pieces of skeletal information 60 based on the two pieces of skeletal information 60 .

[0044] The functions of the estimation unit 120 are realized by cooperation of various processors and memories. The details of the functions of the estimation unit 120 will be described later.

[0045] (Conversion unit 130) The conversion unit 130 makes two different pieces of skeletal information 60 compatible with each other based on the results of estimation by the estimation unit 120.

[0046] Here, making them mutually compatible includes converting one of the two different skeleton information 60 to conform to the standard of the other.

[0047] Furthermore, making them mutually compatible includes converting (standardizing) two different sets of skeletal information 60 to conform to a defined standard.

[0048] The functions of the conversion unit 130 are realized by the cooperation of various processors and memories. The functions of the conversion unit 130 will be described in detail later.

[0049] (Search unit 140) The search unit 140 performs a search based on the skeleton information 60 standardized by the conversion unit 130 .

[0050] The functions of the search unit 140 are realized by the cooperation of various processors and memories. The functions of the search unit 140 will be described in detail later.

[0051] (Interpolation unit 150) The interpolation unit 150 interpolates between two pieces of motion data 50 relating to the different pieces of skeleton information 60 that have been made mutually compatible by the conversion unit 130.

[0052] The functions of the interpolation unit 150 are realized by the cooperation of various processors and memories. The functions of the interpolation unit 150 will be described in detail later.

[0053] (Interface control unit 160) The interface control unit 160 controls the user interfaces related to the various functions that the information processing system 1 provides to the user.

[0054] The functions of the interface control unit 160 are realized by the cooperation of various processors and memories.

[0055] The functional configuration example of the information processing system 1 according to this embodiment has been described above. However, the functional configuration described above with reference to Fig. 5 is merely an example, and the functional configuration of the information processing system 1 according to this embodiment is not limited to this example.

[0056] For example, the storage unit 110, estimation unit 120, conversion unit 130, search unit 140, interpolation unit 150, and interface control unit 160 described above do not necessarily need to be included in a single device.

[0057] The storage unit 110, the estimation unit 120, the conversion unit 130, the search unit 140, the interpolation unit 150, and the interface control unit 160 may be provided in a distributed manner in multiple devices.

[0058] The information processing system 1 according to this embodiment may further include an operation receiving unit that receives operations by a user, a display unit that displays various types of information, and the like.

[0059] The information processing system 1 according to this embodiment can be flexibly modified according to specifications, operation, and the like.

[0060] <<1.3. Training of Estimator 125>> Next, a description will be given of the learning of the estimator 125 according to this embodiment. Fig. 6 is a diagram for explaining the learning of the estimator 125 according to this embodiment.

[0061] The estimator 125 according to this embodiment is generated by machine learning.

[0062] The estimator 125 according to this embodiment may be configured to include both a natural language processing model and a graph neural network (GNN) 127, or either the natural language processing model or the GNN.

[0063] Transformer129 is an example of a natural language processing model.

[0064] The learning of the estimator 125 according to this embodiment is roughly divided into pre-learning and additional learning.

[0065] First, in pre-learning, the estimator 125 performs self-supervised learning based on the input skeleton knowledge 67 .

[0066] The skeleton knowledge 67 is information including knowledge about the skeletons of people, animals, etc.

[0067] The skeleton knowledge 67 may be collected from the Internet, or extracted from electronic books, for example.

[0068] The estimator 125 according to this embodiment learns the meaning of words related to skeletons through learning based on skeleton knowledge 67.

[0069] For example, in self-supervised learning, the estimator 125 learns to output "pelvis, pelvis (in Japanese)" as the (?) in response to the input "(?) is the name of the bone located in the hips."

[0070] According to the above task, the estimator 125 can acquire concepts such as, for example, "'pelvis' and 'bonnet (in Japanese)' have the same meaning" or "'pelvis' is closely related to 'hips'."

[0071] In the subsequent additional learning, the estimator 125 performs supervised learning to output an appropriate correspondence table 70 for the two pieces of skeleton information 60 input.

[0072] For example, as shown in FIG. 6, the estimator 125 performs learning so that the correspondence table 70 output for the input skeletal information 60A and 60B approaches a correspondence table 70G that is a given ground truth.

[0073] Through such supervised learning, the estimator 125 can acquire the ability to estimate the correspondence between the parts defined in each of the two pieces of skeletal information 60 based on the two pieces of skeletal information 60 that are input.

[0074] The learning of the estimator 125 according to this embodiment has been described above using an example.

[0075] According to the learning flow described above, the concept base acquired in the preliminary learning can be further tuned using actual tasks in the additional learning, thereby realizing more effective learning.

[0076] However, the above-described pre-learning is not essential, and the estimator 125 may perform only the supervised learning described above as the additional learning.

[0077] Note that the supervised learning described above as additional learning can be realized using only either the natural language processing model or GNN127.

[0078] <<1.4. Processing using Correspondence Table 70>> Next, each process using the correspondence table 70 according to this embodiment will be described.

[0079] As described above, the estimator 125 according to this embodiment estimates the correspondence between the parts defined in each of the two input pieces of skeletal information 60, and outputs the correspondence table 70 as the result of the estimation.

[0080] Furthermore, the conversion unit 130 according to this embodiment makes two different pieces of skeleton information 60 compatible with each other based on the correspondence table 70.

[0081] These processes make it possible to improve the efficiency of each process, which is based on the premise that the standards of the skeleton information 60 are unified.

[0082] First, we will explain retargeting, which is the process of adapting motion data 50 to a 3D model 55.

[0083] Here, if the standards of the skeletal information 60 of the motion data 50 and the skeletal information 60 of the 3D model 55 are different, retargeting requires a process (task) called remapping, which associates the parts defined in each of the two pieces of skeletal information 60.

[0084] FIG. 7 is a diagram for explaining the efficiency of remapping according to this embodiment.

[0085] As shown on the left side of Figure 7, conventional remapping required the user to select the correspondence between a certain part in the skeletal information 60 (motion skeleton) of the motion data 50 and a certain part in the skeletal information 60 (model skeleton) of the 3D model 55 one by one, which was cumbersome.

[0086] On the other hand, according to the correspondence table 70 of this embodiment, as shown on the right side of Figure 7, the above-mentioned work can be automated and the results of the estimated correspondence between parts can be presented to the user, making it possible to significantly improve the efficiency of remapping.

[0087] The user may check the presented estimation results and correct them as necessary. The correction history may be used for re-training the estimator 125, etc.

[0088] After the user confirms and corrects the estimation result, if the user issues an instruction by, for example, pressing a button, the conversion unit 130 according to this embodiment may perform retargeting based on the correspondence table 70.

[0089] 2 to 4, the estimator 125 according to this embodiment can also estimate that no corresponding part exists. That is, the two pieces of skeletal information 60 used for remapping and retargeting according to this embodiment may have different numbers of parts.

[0090] Furthermore, in the above example, the motion data 50 and the 3D model 55 are of a bipedal walking type, but the types of the motion data 50 and the 3D model 55 according to this embodiment are not limited to this example.

[0091] FIG. 8 is a diagram for explaining output to a correspondence table 70 based on skeletal information 60A of a bipedal walking type and skeletal information 60C of a quadrupedal walking type according to this embodiment.

[0092] Skeletal information 60A illustrated in FIG. 8 is skeletal information 60 extracted from bipedal walking type motion data 50A, and skeletal information 60C is skeletal information 60 extracted from quadrupedal walking type 3D model 55C.

[0093] The estimator 125 according to this embodiment is also capable of estimating the correspondence between parts defined in each of the two pieces of skeletal information 60 having different walking styles as described above, and outputting a correspondence table 70 as the result of the estimation.

[0094] As shown in FIG. 9, the skeleton information 60 according to this embodiment may define only a part of the entire body.

[0095] Skeletal information 60D illustrated in FIG. 9 is skeletal information 60 extracted from motion data 50D of the upper body of a human body, and skeletal information 60E is skeletal information 60 extracted from a 3D model 55E of the lower body of a human body.

[0096] The estimator 125 according to this embodiment is also capable of estimating the correspondence between the parts defined in each of two different pieces of skeletal information 60 that define parts of the whole body, as described above, and outputting a correspondence table 70 as the result of the estimation.

[0097] For example, in the example shown in FIG. 9, it is possible to adapt the motion of both arms to both legs.

[0098] Next, standardization of the skeleton information 60 according to this embodiment and retrieval of the motion data 50 based on the standardized skeleton information 60 will be described.

[0099] In recent years, a technology called motion-2-motion search has been developed, which searches for motions similar to one another.

[0100] An example of a method for motion 2 motion search is to calculate time-specific features from the time series data of the skeletal information 60 of the motion data 50 to be searched, and then obtain motion data 50 similar to the search target from a database based on the processed features obtained by applying weighting parameters to the features.

[0101] The weighting parameters can be determined by an estimator obtained by learning the relationship between the feature amounts for each time period and the weighting parameters for each time period.

[0102] However, the above search method is based on the premise that the skeletal information 60 of the motion data 50 to be searched and the skeletal information 60 of the motion data 50 stored in the database have the same standard.

[0103] For this reason, in the past, if the standard of the skeleton information 60 of the motion data 50 to be searched differs from the standard of the skeleton information 60 of the motion data 50 stored in the database, the search could not be performed.

[0104] The information processing method according to this embodiment solves the above problem by standardizing the standard of the skeletal information 60 of the motion data 50 to be searched to match the standard of the skeletal information 60 of the motion data 50 stored in the database.

[0105] FIG. 10 is a diagram for explaining the standardization of the skeleton information 60 according to this embodiment.

[0106] 10 is the motion data 50 to be searched. Furthermore, skeletal information 60A is skeletal information 60 extracted from the motion data 50A, and is an example of target skeletal information.

[0107] The estimator 125 outputs a correspondence table 70 based on the input skeleton information 60A and standard skeleton information 65.

[0108] The standard skeleton information 65 is skeleton information that conforms to the same standard as the standard of the skeleton information 60 of the motion data 50 stored in the motion DB 20 (see FIG. 11).

[0109] That is, in this example, the correspondence table 70 output by the estimator 125 is an estimation result showing the correspondence between parts conforming to an arbitrary standard in the target skeletal information and parts conforming to the standard standard.

[0110] The conversion unit 130 according to this embodiment converts the skeletal information 60A based on the correspondence table 70. That is, the conversion unit 130 standardizes the skeletal information 60A in accordance with the standard skeletal information 65, and obtains standardized skeletal information 60AS.

[0111] Next, a search based on the standardized skeletal information 60AS will be described with reference to FIG.

[0112] The search unit 140 according to this embodiment performs a search based on standardized skeleton information 60AS.

[0113] The search unit 140 in this embodiment, for example, calculates time-dependent features from the time series data of the standardized skeletal information 60AS, and obtains search results 52 from the motion DB 20 based on processed features obtained by applying weighting parameters to the features.

[0114] The search result 52 may be motion data 50 including skeleton information 60 having features highly similar to the standardized skeleton information 60AS.

[0115] Next, with reference to FIG. 12, a configuration example of the information processing system 1 for performing the above-mentioned motion 2 motion search will be described.

[0116] In this case, as shown in FIG. 12, the information processing system 1 includes, in addition to the information processing device 10, a motion DB 20 and a client 30.

[0117] The client 30 is a computer terminal used by a user who performs a search, and may be, for example, a PC (Personal Computer).

[0118] The user uploads the motion data 50 to be searched to the information processing device 10 using a user interface controlled by the interface control unit 160 in the client 30.

[0119] The information processing device 10 performs the series of processes described with reference to Figures 10 and 11 on the motion data 50 uploaded from the client 30, obtains the search results 52, and returns the search results 52 to the client 30.

[0120] The information processing device 10 and the motion DB 20 may be built on, for example, a cloud. Furthermore, the number of clients 30 is not limited.

[0121] All or part of the series of processes described with reference to FIGS. 10 and 11 may be performed by the client 30.

[0122] For example, the client 30 may extract the skeleton information 60 from the motion data 50 to be searched, and upload the extracted skeleton information 60 to the information processing device 10.

[0123] Furthermore, for example, the client 30 may perform standardization based on the correspondence table 70 and upload the standardized skeleton information 60 to the information processing device 10.

[0124] The standardization of the skeleton information 60 according to this embodiment and the search for the motion data 50 based on the standardized skeleton information 60 have been described above.

[0125] According to the standardization method described above, it is possible not only to perform searches but also to apply motion data 50 or 3D models 55 that comply with a certain standard to a system that only complies with another standard.

[0126] Next, motion interpolation based on the correspondence table 70 according to this embodiment will be described.

[0127] In the production of 3D animation, when connecting one motion data 50 with another motion data 50, an interpolation process (motion interpolation) may be performed.

[0128] Techniques for motion interpolation include, for example, motion linking, motion blending, and generation of interpolated motion data.

[0129] Motion linking is a process of simply linking one piece of motion data 50 with another piece of motion data 50. In this case, to achieve a more natural linking, a process of correcting a portion of the section included in one piece of motion data 50 to match the other piece of motion data 50 may be performed.

[0130] Motion blending is a process that linearly and geometrically combines one piece of motion data 50 with another piece of motion data 50. In motion blending, in an interpolation section where one piece of motion data 50 overlaps the other piece of motion data 50, interpolation is performed such that the movement included in one piece of motion data 50 gradually switches to the movement included in the other piece of motion data 50.

[0131] Furthermore, generation of interpolated motion data is a method of generating new motion data (interpolated motion data) that interpolates between one motion data 50 and another motion data 50. Generation of the interpolated motion data may be achieved by supervised learning using, for example, deep learning.

[0132] However, in any of the above methods, it is assumed that the standards of the skeleton information 60 of the two motion data 50 are unified.

[0133] Therefore, the information processing method of this embodiment realizes interpolation between motion data 50 that conform to different standards by making the skeletal information 60 of the two target motion data 50 mutually compatible based on a correspondence table 70.

[0134] FIG. 13 is a diagram for explaining motion interpolation based on the correspondence table 70 according to this embodiment.

[0135] In the example shown in FIG. 13, the estimator 125 receives as input skeletal information 60A extracted from motion data 50A and skeletal information 60B extracted from motion data 50B.

[0136] Based on the input skeletal information 60A and skeletal information 60B, the estimator 125 outputs a correspondence table 70 indicating the correspondence between the parts defined in each of the skeletal information 60A and skeletal information 60B.

[0137] Next, in this example, the conversion unit 130 receives the motion data 50B, the skeleton information 60B, and the correspondence table 70.

[0138] The conversion unit 130 converts the skeleton information 60B to conform to the standard of the skeleton information 60A based on the correspondence table 70. Furthermore, the conversion unit 130 converts the motion data 50B using the skeleton information 60Ba (not shown) converted to conform to the standard of the skeleton information 60A, thereby obtaining the motion data 50Ba.

[0139] The motion data 50Ba is motion data 50 having skeleton information 60Ba that conforms to the same standard as the standard of the skeleton information 60A of the motion data 50A.

[0140] The motion data 50Ba is input to the interpolation unit 150 together with the motion data 50A.

[0141] The interpolation unit 150 interpolates between the input motion data 50A and motion data 50Ba.

[0142] The motion interpolation performed by the interpolator 150 may be any of motion concatenation, motion blending, and generation of interpolated motion data.

[0143] For example, the interpolation section 150 may generate interpolated motion data 75AB that interpolates between the input motion data 50A and motion data 50Ba.

[0144] Note that Figure 13 illustrates an example in which motion data 50B is converted to match motion data 50A, but motion data 50A may also be converted to match motion data 50B, or motion data 50A and motion data 50B may be converted (standardized) using standard skeleton information 65.

[0145] <2. Hardware configuration example> Next, a description will be given of an example hardware configuration of the information processing device 90 according to an embodiment of the present disclosure. Fig. 14 is a block diagram showing an example hardware configuration of the information processing device 90 according to an embodiment of the present disclosure.

[0146] The information processing device 90 may be a device having the same hardware configuration as the information processing device 10.

[0147] 14, the information processing device 90 includes, for example, a processor 871, a ROM 872, a RAM 873, a host bus 874, a bridge 875, an external bus 876, an interface 877, an input device 878, an output device 879, a storage 880, a drive 881, a connection port 882, and a communication device 883. Note that the hardware configuration shown here is an example, and some of the components may be omitted. Furthermore, the information processing device 90 may further include components other than those shown here.

[0148] (Processor 871) The processor 871 functions as, for example, an arithmetic processing device or a control device, and controls the overall operation of each component or part of it based on various programs recorded in the ROM 872, the RAM 873, the storage 880, or the removable storage medium 901.

[0149] (ROM872, RAM873) The ROM 872 is a means for storing programs to be read into the processor 871, data to be used for calculations, etc. The RAM 873 temporarily or permanently stores, for example, programs to be read into the processor 871, and various parameters that change as appropriate when the programs are executed.

[0150] (Host bus 874, bridge 875, external bus 876, interface 877) The processor 871, ROM 872, and RAM 873 are connected to one another via, for example, a host bus 874 that is capable of high-speed data transmission. On the other hand, the host bus 874 is connected to, for example, an external bus 876 that has a relatively low data transmission speed via a bridge 875. In addition, the external bus 876 is connected to various components via an interface 877.

[0151] (Input Device 878) The input device 878 may be, for example, a mouse, keyboard, touch panel, button, switch, lever, etc. Furthermore, a remote controller (hereinafter referred to as a remote control) capable of transmitting control signals using infrared rays or other radio waves may also be used as the input device 878. The input device 878 may also include an audio input device such as a microphone.

[0152] (Output Device 879) The output device 879 is a device capable of visually or audibly notifying the user of acquired information, such as a display device such as a CRT (Cathode Ray Tube), LCD, or organic EL, an audio output device such as a speaker or headphones, a printer, a mobile phone, a facsimile, etc. The output device 879 according to the present disclosure also includes various vibration devices capable of outputting tactile stimuli.

[0153] (Storage 880) The storage 880 is a device for storing various types of data. For example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device may be used as the storage 880.

[0154] (Drive 881) The drive 881 is a device that reads information recorded on a removable storage medium 901 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, or writes information to the removable storage medium 901 .

[0155] (Removable storage medium 901) The removable storage medium 901 is, for example, a DVD medium, a Blu-ray (registered trademark) medium, an HD DVD medium, various semiconductor storage media, etc. Of course, the removable storage medium 901 may also be, for example, an IC card equipped with a contactless IC chip, an electronic device, etc.

[0156] (Connection port 882) The connection port 882 is a port for connecting an external device 902, such as a USB (Universal Serial Bus) port, an IEEE1394 port, a SCSI (Small Computer System Interface), an RS-232C port, or an optical audio terminal.

[0157] (External connection device 902) The externally connected device 902 is, for example, a printer, a portable music player, a digital camera, a digital video camera, or an IC recorder.

[0158] (Communication Device 883) The communication device 883 is a communication device for connecting to a network, such as a communication card for wired or wireless LAN, Bluetooth (registered trademark), or WUSB (Wireless USB), a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communication.

[0159] <3. Summary> As described above, the information processing system 1 according to one embodiment of the present disclosure includes an estimation unit 120 that estimates the correspondence between parts defined in each of two different pieces of skeletal information 60 based on the two pieces of skeletal information 60, and a conversion unit 130 that makes the two different pieces of skeletal information 60 mutually compatible based on the result of estimation by the estimation unit 120.

[0160] According to the above configuration, it is possible to improve the efficiency of various processes and operations that use skeletal information.

[0161] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0162] For example, in the above example, the skeleton information 60 is related to a 3D animation, but the skeleton information 60 may be related to a robot that exists in the real world.

[0163] Furthermore, the technical idea according to an embodiment of the present disclosure can be applied to various types of information described in a graph structure in addition to the skeletal information 60.

[0164] Furthermore, the steps of the processes described in this disclosure do not necessarily have to be processed in chronological order according to the order shown in the flowcharts or sequence diagrams. For example, the steps of the processes of each device may be processed in an order different from the order shown, or may be processed in parallel.

[0165] Furthermore, the series of processes performed by each device described in this disclosure may be realized by a program stored in a non-transitory computer-readable storage medium. Each program is, for example, loaded into RAM when executed by a computer and executed by a processor such as a CPU. The storage medium may be, for example, a magnetic disk, an optical disk, a magneto-optical disk, or a flash memory. The program may also be distributed, for example, via a network, without using a storage medium.

[0166] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0167] The following configurations also fall within the technical scope of the present disclosure. (1) an estimation unit that estimates a correspondence between parts defined in each of two different pieces of skeletal information based on the two pieces of skeletal information; a conversion unit that makes the two different pieces of skeletal information compatible with each other based on a result of estimation by the estimation unit; Equipped with Information processing system. (2) a search unit that performs a search based on the skeleton information that has been made mutually compatible by the conversion unit; Further provided with The information processing system according to (1) above. (3) the estimation unit estimates a correspondence between parts defined in the standard skeletal information and the target skeletal information, based on the standard skeletal information and the target skeletal information; The information processing system according to (2) above. (4) The conversion unit standardizes the target skeletal information to the standard skeletal information based on the estimation result by the estimation unit. The information processing system according to (3) above. (5) the search unit performs a search based on the skeleton information standardized by the conversion unit. The information processing system according to (4) above. (6) Skeletal information is extracted from 3D animation motion data. The information processing system according to any one of (1) to (5). (7) an interpolation unit that interpolates between two pieces of motion data related to the different pieces of skeleton information based on the two pieces of skeleton information that have been made mutually compatible by the conversion unit; Further provided with The information processing system according to (6) above. (8) the estimation unit estimates correspondences between parts defined in each of skeletal information extracted from the motion data of the 3D animation and skeletal information extracted from the 3D model; The information processing system according to any one of (1) to (7). (9) The conversion unit performs retargeting based on a result of the estimation by the estimation unit. The information processing system according to (8) above. (10) the estimation unit estimates the correspondence between the parts defined in each of the two pieces of skeletal information using an estimator generated by machine learning; The information processing system according to any one of (1) to (9). (11) The estimator includes at least one of a natural language processing model or a graph neural network. The information processing system according to (10) above. (12) The natural language processing model includes a Transformer. The information processing system according to (11) above. (13) Estimating correspondence between regions defined in each of two different pieces of skeletal information based on the two pieces of skeletal information; Making the two different pieces of skeletal information compatible with each other based on the result of the estimation; Including, Information processing methods. (14) Searching based on mutually compatible skeletal information; further comprising: The information processing method according to (13) above. (15) Estimating correspondence between regions defined in the standard skeletal information and the target skeletal information based on the standard skeletal information and the target skeletal information; The information processing method according to (14) above. (16) standardizing the target skeletal information to the standard skeletal information based on the estimation result; The information processing method according to (15) above. (17) Skeletal information is extracted from 3D animation motion data. The information processing method according to any one of (13) to (16). (18) interpolating between two pieces of motion data related to the different pieces of skeleton information based on the two pieces of skeleton information that are made compatible with each other; further comprising: The information processing method according to (17) above. (19) Estimating correspondences between parts defined in each of the skeletal information extracted from the motion data of the 3D animation and the skeletal information extracted from the 3D model; The information processing method according to any one of (13) to (18). (20) performing retargeting based on the results of said estimation; further comprising: The information processing method according to (19) above. [Explanation of symbols]

[0168] 1. Information Processing Systems 10. Information processing equipment 110 Storage section 120 Estimation part 125 Estimator 127 GNN 129 Transformer 130 Conversion Unit 140 Search Department 150 Interpolation section 160 Interface control section 20 Motion DB 30 clients 50 motion data 55 3D models 60 Skeletal information 65 Standard skeletal information 70 Correspondence Table

Claims

1. an estimation unit that estimates a correspondence between parts defined in each of two different pieces of skeletal information based on the two pieces of skeletal information; a conversion unit that makes the two different pieces of skeletal information compatible with each other based on a result of estimation by the estimation unit; Equipped with Information processing system.

2. a search unit that performs a search based on the skeleton information that has been made mutually compatible by the conversion unit; Further provided with The information processing system according to claim 1 .

3. the estimation unit estimates a correspondence between parts defined in the standard skeletal information and the target skeletal information, based on the standard skeletal information and the target skeletal information; The information processing system according to claim 2 .

4. The conversion unit standardizes the target skeletal information to the standard skeletal information based on the estimation result by the estimation unit. The information processing system according to claim 3 .

5. the search unit performs a search based on the skeleton information standardized by the conversion unit. The information processing system according to claim 4 .

6. Skeletal information is extracted from 3D animation motion data. The information processing system according to claim 1 .

7. an interpolation unit that interpolates between two pieces of motion data related to the different pieces of skeleton information based on the two pieces of skeleton information that have been made mutually compatible by the conversion unit; Further provided with The information processing system according to claim 6.

8. the estimation unit estimates correspondence between parts defined in each of skeletal information extracted from the motion data of the 3D animation and skeletal information extracted from the 3D model; The information processing system according to claim 1 .

9. The conversion unit performs retargeting based on a result of the estimation by the estimation unit. The information processing system according to claim 8 .

10. the estimation unit estimates the correspondence between the parts defined in each of the two pieces of skeletal information using an estimator generated by machine learning; The information processing system according to claim 1 .

11. The estimator includes at least one of a natural language processing model or a graph neural network. The information processing system according to claim 10.

12. The natural language processing model includes a Transformer. The information processing system according to claim 11.

13. Estimating correspondence between regions defined in each of two different pieces of skeletal information based on the two pieces of skeletal information; Making the two different pieces of skeletal information compatible with each other based on the result of the estimation; Including, Information processing methods.

14. Searching based on mutually compatible skeletal information; further comprising: The information processing method according to claim 13.

15. Estimating correspondence between regions defined in the standard skeletal information and the target skeletal information based on the standard skeletal information and the target skeletal information; The information processing method according to claim 14.

16. standardizing the target skeletal information to the standard skeletal information based on the estimation result; The information processing method according to claim 15.

17. Skeletal information is extracted from 3D animation motion data. The information processing method according to claim 13.

18. interpolating between two pieces of motion data related to the different pieces of skeleton information based on the two pieces of skeleton information that are made compatible with each other; further comprising:

18. The information processing method according to claim 17.

19. Estimating correspondence between parts defined in each of the skeletal information extracted from the motion data of the 3D animation and the skeletal information extracted from the 3D model; The information processing method according to claim 13.

20. performing retargeting based on the results of said estimation; further comprising:

20. The information processing method according to claim 19.

Images