Information processing device, information processing method, and program

Abstract

To provide a novel and improved information processing method, information processing apparatus and program capable of improving user convenience.SOLUTION: The information processing apparatus includes: an acquisition unit to acquire a processed feature amount that is a feature amount calculated by applying, to an unprocessed feature amount that is a feature amount of each time or each part of an object calculated from time-series data concerning motion of the object, a weight parameter prepared for each time or each part; and a search unit configured to search for motion data by using the processed feature amount acquired by the acquisition unit.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, animation production and distribution using motion capture to acquire motion information indicating user movements has become popular. For example, the acquired motion information is used to generate motion data that mimics the user's movements, and avatar images based on the motion data are distributed.

[0003] Against this background, the amount of motion data is increasing year by year, and technologies for reusing previously generated motion data are being developed. For example, Patent Document 1 discloses a technology for linking multiple pieces of motion data to create animation data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] US Patent Application Publication No. 2012 / 0038628 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when a user uses the above-mentioned motion data and animation data, the user needs to use a search method such as a text search or a category search to search for the motion data, etc. As the amount of motion data increases and becomes more complex, it may become difficult for the user to search for the motion data, etc. that the user desires.

[0006] Therefore, the present disclosure proposes a new and improved information processing method, information processing device, and program that can improve user convenience. [Means for solving the problem]

[0007] According to the present disclosure, there is provided an information processing device including: an acquisition unit that acquires processed features, which are features calculated by applying weighting parameters prepared for each time or each part of the object to pre-processed features, which are features calculated for each time or each part of the object from time-series data of the movement of the object; and a search unit that searches for motion data using the processed features acquired by the acquisition unit.

[0008] Furthermore, according to the present disclosure, there is provided an information processing method executed by a computer, which includes acquiring processed features, which are features calculated by applying weighting parameters prepared for each time or each body part to pre-processed features, which are features calculated for each time or each body part of the object from time-series data of the movement of the object, and searching for motion data using the acquired post-processed features.

[0009] Furthermore, according to the present disclosure, a program is provided that causes a computer to implement an acquisition function that acquires post-processing features, which are features calculated by applying weighting parameters prepared for each time or each part of the object to pre-processing features, which are features calculated for each time or each part of the object from time-series data of the movement of the object, and a search function that searches for motion data using the post-processing features acquired by the acquisition function. [Brief explanation of the drawings]

[0010] [Figure 1] 10 is an explanatory diagram for explaining an example of an operation process related to a search for motion data of the information processing terminal 10 according to the present disclosure. FIG. [Figure 2] 2 is an explanatory diagram illustrating an example of a functional configuration of an information processing terminal 10 according to the present disclosure. FIG. [Figure 3] 2 is an explanatory diagram illustrating an example of a functional configuration of a server 20 according to the present disclosure. FIG. [Figure 4]FIG. 10 is an explanatory diagram illustrating an example of a GUI (Graphical User Interface) that links a plurality of search results. [Figure 5] FIG. 10 is an explanatory diagram illustrating an example of modifying a section included in existing animation data into motion data. [Figure 6] FIG. 10 is an explanatory diagram illustrating an example of a GUI for modifying an existing animation. [Figure 7] FIG. 10 is an explanatory diagram showing a specific example of a method for generating skeleton data. [Figure 8] FIG. 10 is an explanatory diagram illustrating an example of a method for learning the relationship between time-series data of skeleton data and pre-processing feature amounts using machine learning technology. [Figure 9] 10A and 10B are explanatory diagrams illustrating an example of a method for calculating pre-processing feature amounts for each part according to the present disclosure. [Figure 10] 10A and 10B are explanatory diagrams illustrating an example of a method for calculating a processed feature amount by applying a weight parameter to a pre-processed feature amount. [Figure 11] FIG. 10 is an explanatory diagram illustrating an example of weight parameters prepared for each time period. [Figure 12] FIG. 10 is an explanatory diagram illustrating an example of a weight parameter learning method. [Figure 13] 10A and 10B are explanatory diagrams illustrating an example of a process for correcting a feature amount of motion data. [Figure 14] 10 is an explanatory diagram for explaining an example of an operation process related to a search for motion data of the information processing terminal 10 according to the present disclosure. FIG. [Figure 15] 10 is an explanatory diagram for explaining an example of an operation process related to a search for motion data by the server 20 according to the present disclosure. FIG. [Figure 16] 2 is a block diagram showing a hardware configuration of the information processing terminal 10. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0011] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0012] The "Mode for Carrying Out the Invention" will be described in the following order. 1. Overview of the information processing system 2. Example of functional configuration 2-1. Example of functional configuration of information processing terminal 2-2. Server functional configuration example 3.Details 3-1.Example of user interface 3-2.Pose estimation 3-3. Feature calculation 3-4. Weight parameters 3-5. Similarity evaluation 3-6. Correction 4. Example of operation 4-1. Operation of information processing terminal 4-2. Server operation 5. Examples of effects 6. Hardware Configuration 7. Supplementary Information

[0013] <<1. Overview of the information processing system>> For example, skeleton data expressed by a skeleton structure showing the structure of the body is used as motion data to visualize information on the movements of moving bodies such as humans and animals. Skeleton data includes information such as the positions and postures of body parts. Note that parts in the skeleton structure correspond to, for example, extremity parts and joint parts of the body. Skeleton data may also include bones, which are line segments connecting parts. Bones in the skeleton structure may correspond to, for example, human bones, but the positions and number of bones do not necessarily have to match those of an actual human skeleton.

[0014] The position and posture of each part in the skeleton data can be obtained using a variety of motion capture technologies, such as a camera-based technology in which markers are attached to each part of the body and the positions of the markers are acquired using an external camera, and a sensor-based technology in which motion sensors are attached to each part of the body and the position information of the motion sensors is acquired based on the time-series data acquired by the motion sensors.

[0015] Skeleton data has a wide range of uses. For example, time-series data of skeleton data is used to improve form in sports, or in applications such as virtual reality (VR) or augmented reality (AR). Time-series data of skeleton data is also used to generate avatar images that mimic the movements of a user, and the avatar images are distributed.

[0016] As an embodiment of the present disclosure, the following describes an example configuration of an information processing system that acquires feature amounts of skeleton data calculated from time-series data of a user's entire body movements or feature amounts of the skeleton data for each body part, and searches for motion data using the feature amounts. Note that, although humans will be mainly described below as an example of a moving body, the embodiment of the present disclosure can be similarly applied to other moving bodies such as animals and robots.

[0017] 1 is an explanatory diagram illustrating an information processing system according to an embodiment of the present disclosure. As shown in FIG. 1, the information processing system according to the embodiment of the present disclosure includes six sensor devices S1 to S6 worn by a user U, an information processing terminal 10, and a server 20.

[0018] The information processing terminal 10 and the server 20 are connected via a network 1. The network 1 is a wired or wireless transmission path for information transmitted from devices connected to the network 1. For example, the network 1 may include public network such as the Internet, a telephone network, or a satellite communication network, or various LANs (Local Area Networks) including Ethernet (registered trademark), and WANs (Wide Area Networks). The network 1 may also include a dedicated network such as an IP-VPN (Internet Protocol-Virtual Private Network).

[0019] (Sensor device S) The sensor device S detects the movement of the user U. The sensor device S includes, for example, an inertial sensor (IMU: Inertial Measurement Unit) such as an acceleration sensor that acquires acceleration and a gyro sensor (angular velocity sensor) that acquires angular velocity.

[0020] The sensor device S may also be any of various sensor devices equipped with a sensor that detects the movement of the user U, such as an imaging sensor, a ToF (Time of Flight) sensor, a magnetic sensor, or an ultrasonic sensor.

[0021] It is desirable that the sensor devices S1 to S6 be attached to reference joints of the body (for example, the waist or head) or near the extremities of the body (wrists, ankles, head, etc.). In the example shown in FIG. 1, the user U has sensor device S1 attached to his waist, sensor devices S2 and S5 attached to both wrists, sensor devices S3 and S4 attached to both ankles, and sensor device S6 attached to his head. Note that, hereinafter, the body parts to which the sensor devices S are attached may also be referred to as attachment parts. Furthermore, the number and attachment positions of the sensor devices S (positions of the attachment parts) are not limited to the example shown in FIG. 1, and the user U may have more or fewer sensor devices S attached to them.

[0022] Such a sensor device S acquires acceleration or angular velocity of the part where the sensor device S is attached as time-series data, and transmits the time-series data to the information processing terminal 10.

[0023] Furthermore, the user U does not need to wear the sensor device S. For example, the information processing terminal 10 may detect the movement of the user U using various sensors (for example, an imaging sensor or a ToF sensor) included in the information processing terminal 10.

[0024] (Information processing terminal 10) The information processing terminal 10 is an example of an information processing device. The information processing terminal 10 calculates a feature amount of the movement of the user U from the time-series data received from the sensor device S, and searches for motion data using the calculated feature amount.

[0025] For example, the information processing terminal 10 transmits the processed feature amount as a search request to the server 20. Then, the information processing terminal 10 receives from the server 20 the motion data searched by the server 20 in response to the search request.

[0026] Although FIG. 1 shows a smartphone as the information processing terminal 10, the information processing terminal 10 may be other information processing devices such as a notebook PC (Personal Computer) or a desktop PC.

[0027] (Server 20) The server 20 stores a plurality of motion data and the feature values ​​of each of the plurality of motion data. The server 20 also evaluates the similarity between the feature values ​​of each of the plurality of motion data and the processed feature values ​​received from the information processing terminal 10, and transmits motion data according to the results of the similarity evaluation to the information processing terminal 10.

[0028] The information processing system according to the present disclosure has been outlined above. Next, an example of the functional configuration of the information processing terminal 10 and the server 20 according to the present disclosure will be described.

[0029] <<2. Functional configuration example>> <2-1. Example of functional configuration of information processing terminal> 2 is an explanatory diagram illustrating an example of a functional configuration of the information processing terminal 10 according to the present disclosure. As shown in FIG. 2, the information processing terminal 10 includes an operation display unit 110, a communication unit 120, and a control unit 130.

[0030] (Operation display section 110) The operation display unit 110 functions as a display unit that displays the search results sent by the server 20. The operation display unit 110 also functions as an operation unit that allows the user to input operations.

[0031] The function of the display unit is realized by, for example, a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD) device, or an OLED (Organic Light Emitting Diode) device.

[0032] The function of the operation unit is realized by, for example, a touch panel, a keyboard, or a mouse.

[0033] In FIG. 1, the information processing terminal 10 has a configuration in which the functions of the display unit and the operation unit are integrated, but the functions of the display unit and the operation unit may be separated.

[0034] (Communication unit 120) The communication unit 120 communicates various information with the server 20 via the network 1. For example, the communication unit 120 transmits processed skeleton data calculated from time-series data of the user's movements to the server 20. The communication unit 120 also receives motion data searched by the server 20 according to the transmitted processed feature values.

[0035] (control unit 130) The control unit 130 controls the overall operation of the information processing terminal 10. As shown in FIG.

[0036] The posture estimation unit 131 estimates wearing part information indicating the position and posture of each wearing part based on time series data such as the acceleration or velocity of the wearing part acquired from the sensor device S. The position and posture of each wearing part may be a two-dimensional position or a three-dimensional position.

[0037] Then, the posture estimation unit 131 generates skeleton data including position information and posture information of each part in the skeleton structure based on the wearing part information. The posture estimation unit 131 may also convert the generated skeleton data into reference skeleton data. Details related to posture estimation will be described later.

[0038] The feature calculation unit 135 is an example of an acquisition unit, and calculates pre-processing feature amounts, which are feature amounts of the entire body or feature amounts for each part of the skeleton data, from the time-series data of the skeleton data. Then, the feature calculation unit 135 applies weighting parameters to the pre-processing feature amounts to calculate post-processing feature amounts. Details of the pre-processing feature amounts, weighting parameters, and post-processing feature amounts will be described later.

[0039] The search request unit 139 is an example of a search unit, and causes the communication unit 120 to transmit the processed feature amount calculated by the feature amount calculation unit 135 as a search request.

[0040] The correction unit 143 corrects the feature amounts of the motion data by mixing the processed feature amounts at a set ratio with the feature amounts of the motion data received as search results from the server 20. Details of the correction will be described later.

[0041] The above describes an example of the functional configuration of the information processing terminal 10. Next, an example of the functional configuration of the server 20 will be described with reference to FIG.

[0042] <2-2. Server functional configuration example> 3 is an explanatory diagram illustrating an example of a functional configuration of the server 20 according to the present disclosure. As shown in FIG. 3, the server 20 includes a communication unit 210, a storage unit 220, and a control unit 230.

[0043] (Communication unit 210) The communication unit 210 communicates various types of information with the information processing terminal 10 via the network 1. For example, the communication unit 210 receives from the information processing terminal 10 processed feature amounts of the whole body or each part of skeleton data calculated from time-series data of the user's movements. In addition, the communication unit 210 transmits to the information processing terminal 10 motion data searched for according to the processed feature amounts received from the information processing terminal 10.

[0044] (Storage unit 220) The storage unit 220 stores software and various data. As shown in FIG.

[0045] The motion data storage unit 221 holds a plurality of motion data.

[0046] The motion feature amount storage unit 225 stores the feature amounts of each of the multiple motion data stored in the motion data storage unit 221. More specifically, the motion feature amount storage unit 225 stores the feature amounts of reference motion data, which is motion data obtained by converting each skeleton data included in the motion data into reference skeleton data.

[0047] (control unit 230) The control unit 230 controls the overall operation of the server 20. As shown in FIG. 3, the control unit 230 includes a reference skeleton conversion unit 231, a feature calculation unit 235, a similarity evaluation unit 239, a learning unit 243, and an estimator 247.

[0048] The reference skeleton conversion unit 231 converts the skeleton data included in each of the multiple motion data into reference skeleton data. More specifically, it converts the skeleton of each part included in each of the skeleton data into a reference skeleton including predetermined skeleton information.

[0049] The feature amount calculation unit 235 calculates the feature amounts of the motion data converted into reference skeleton data, and outputs the feature amount calculation results to the motion feature amount storage unit 225. Note that the motion data converted into reference skeleton data is an example of reference motion data.

[0050] The similarity evaluation unit 239 evaluates the similarity between the processed feature received from the information processing terminal 10 and each feature of the plurality of motion data stored in the motion feature storage unit 225. Details of the similarity evaluation will be described later.

[0051] The learning unit 243 generates learning data by machine learning technology using pairs of time-series data for each body part of the skeleton data and feature amounts for each body part of the motion data as training data.

[0052] In addition, the learning unit 243 may acquire weight parameters for each part and weight parameters for each time by using attention in a machine learning technique that uses the time series data of the skeleton data and a set of feature values ​​for each part of the motion data as training data.

[0053] The estimator 247 estimates the pre-processing feature amount of each body part from the skeleton data of the user. The function of the estimator 247 is obtained by the learning data generated by the learning unit 243.

[0054] An example of a functional configuration according to the present disclosure has been described above. Next, details of the system according to the present disclosure will be sequentially described with reference to FIGS.

[0055] <<3.Details>> <3-1. Example of user interface> The user searches for motion data or modifies existing animation data by performing operations on the display screen of the operation display unit 110. In this disclosure, as an example of searching for motion data, an example will be described in which multiple motion data searched in accordance with the user's movements are linked to generate one piece of animation data. Also, as an example of modifying animation data, an example will be described in which a portion of a section included in existing animation data is modified to the motion data searched in accordance with weighting parameters.

[0056] (Concatenation of search results) 4 is an explanatory diagram illustrating an example of a GUI (Graphical User Interface) that links multiple search results. As shown in FIG. 4, the GUI that links multiple search results may include skeleton data S, a search button s1, sections A1 to A3, a correction section d2, and a seek bar b1.

[0057] The search button s1 is a button that turns on or off the search function that acquires user movement information. Furthermore, sections A1 to A3 are sections into which motion data searched according to the user's movement is inserted, and the correction section d2 is a section that connects two sections into which motion data is inserted. Furthermore, the seek bar b1 is an indication bar for displaying skeleton data s at the timing specified by the cursor.

[0058] The operations and processes performed in the GUI are as follows: (1) First, the user selects the search button s1 through a predetermined operation to turn on the search function. (2) Next, the user performs a motion that includes the information that the user wants to search for as motion data. (3) Next, the user selects the search button s1 again to turn off the search function. (4) Then, the operation display unit 110 displays the searched motion data in accordance with the user's movements. (5) If multiple pieces of motion data are displayed based on the user's movements, the user selects one piece of motion data from the displayed pieces of motion data. (6) Furthermore, the user selects one of the sections A1 to A3 as the insertion section. (7) Then, the operation display unit 110 inserts the motion data into the section selected by the user.

[0059] By repeating the operations and processes (1) to (7) multiple times, animation data is generated in which multiple pieces of motion data are linked together.

[0060] The correction section d2 is arbitrary, and may be filled using any correction method, or the correction section d2 may be eliminated and multiple insertion sections may be joined to generate animation data.

[0061] Furthermore, the operation display unit 110 may display a seek bar b1, allowing the user to check the animation data generated by linking the motion data.

[0062] Furthermore, in (6), the user does not have to specify the insertion interval. For example, motion data may be inserted in order starting from the interval with the earliest time. For example, when the operations and processes of (1) to (5) are performed multiple times, the motion data selected by the user in (5) may be inserted in order starting from interval A1. Then, the information processing terminal 10 may use any correction method to connect the motion data of each interval between intervals A1 and A2 and between intervals A2 and A3.

[0063] 4 shows three sections, A1 to A3, into which motion data is inserted, but the number of sections into which motion data is inserted does not have to be three. The number of sections into which motion data is inserted may be determined depending on the number of times the operations and processes (1) to (5) are performed.

[0064] Furthermore, although details will be described later, the operation display unit 110 may display setting fields for various parameters such as various weighting parameters and setting ratios of processed feature amounts and feature amounts of motion data.

[0065] Next, with reference to FIGS. 5 and 6, an example of modifying a portion of existing animation data into motion data will be described.

[0066] (Modifying existing animation data) 5 is an explanatory diagram illustrating an example of modifying a portion of a section included in existing animation data into motion data. In an embodiment according to the present disclosure, a portion of a section included in animation data obtained by motion capture or manually (hereinafter referred to as existing animation data A) may be modified by replacing it with motion data B.

[0067] For example, the user selects section A2 as the section to be modified from among a plurality of sections A1 to A3 included in the existing animation data.

[0068] Then, the operation display unit 110 may replace the section A2 of the existing animation data with the motion data B found based on the processed feature amount of the time-series data of the skeleton data included in the section A2 and display it.

[0069] For example, as shown in Fig. 5, when two pieces of motion data B are displayed as search results, the user selects one of the pieces of motion data B. When the user selects the left image of motion data B shown in Fig. 5, the operation display unit 110 replaces the left image of motion data B shown in Fig. 5 in section A2 of the existing animation and displays it.

[0070] An example of modifying an existing animation according to the present disclosure will be described more specifically with reference to FIG.

[0071] Fig. 6 is an explanatory diagram illustrating an example of a GUI for modifying an existing animation. As shown in Fig. 6, the GUI for modifying an existing animation may include skeleton data S, a weight parameter setting field w1 for each body part, a weight parameter setting field w2 for each time period, a setting ratio setting field qb, a search button s2, an interval A2, a seek bar b2, and a play command c1.

[0072] The weight parameter setting field w1 for each body part is a setting field for setting weight parameters to be applied to the pre-processing feature values ​​calculated for each body part. The weight parameter setting field w2 for each time is a setting field for setting weight parameters to be applied to the pre-processing feature values ​​calculated for each time. The setting ratio setting field qb is a setting field for setting the ratio at which the post-processing feature values ​​are mixed with the feature values ​​of the motion data for each body part. The weight parameters for each body part, the weight parameters for each time, and the setting ratios will be described in detail below.

[0073] Furthermore, the user can check the modified animation data by operating the playback command c1. The user may also check the modified animation data by operating the seek bar b2.

[0074] First, the user selects section A2 as the correction section, then sets various parameters in the weight parameter setting field for each body part w1, the weight parameter setting field for each time period w2, and the setting ratio setting field qb, and selects the search button s2.

[0075] Then, the operation display unit 110 displays at least one or more motion data searched in response to the user's operation. When one motion data is displayed as a search result, the operation display unit 110 inserts the motion data into section A2 by replacing it with the motion data. When multiple motion data are displayed as a search result, the user selects one motion data from the multiple motion data, and the operation display unit 110 inserts the one motion data selected by the user into section A2 by replacing it with the motion data.

[0076] Although specific examples of user interfaces have been described above, an embodiment according to the present disclosure is not limited to such examples. For example, while an example has been described in which a user selects a section to be corrected when correcting an existing animation, the information processing terminal 10 may present candidate sections for correction to the user. For example, the operation display unit 110 may present candidate sections for correction to the user together with displaying the existing animation data. In this case, the user may perform an operation to change the presented candidate sections for correction.

[0077] The candidate correction sections presented by the operation display unit 110 may be, for example, sections with relatively large movements among all sections of the existing animation data, or sections estimated to be particularly important using machine learning techniques such as DNN (Deep Neural Network).

[0078] <3-2. Posture estimation> Fig. 7 is an explanatory diagram showing a specific example of a method for generating skeleton data. Based on the time-series data, the posture estimation unit 131 acquires wearing part information PD including position information and posture information of the wearing parts where the sensor devices S1 to S6 are worn, as shown in the left diagram of Fig. 7.

[0079] Furthermore, based on the wearing part information PD of the wearing parts, the posture estimation unit 131 acquires skeleton data SD including position information and posture information of each part in the skeleton structure, as shown in the right diagram of Fig. 7. The skeleton data SD includes not only information on the wearing part SP1 corresponding to the wearing part of the sensor device S1 and the wearing part SP2 corresponding to the wearing part of the sensor device S2, but also information on the non-wearing part SP7.

[0080] The skeleton data SD may include bone information (position information, posture information, etc.) in addition to information about parts. For example, in the example shown in Fig. 7, the skeleton data SD may include information about bone SB1. The posture estimation unit 131 can identify information about bones between parts based on the position information and posture information of parts in the skeleton structure.

[0081] Furthermore, the user's movement may be detected using an imaging sensor or a ToF sensor provided in the information processing terminal 10. In this case, the posture estimation unit 131 may generate the skeleton data SD of the user using, for example, time-series data of images acquired by photographing a person and an estimator obtained by machine learning technology using a set of skeleton data as training data.

[0082] Furthermore, as will be described in detail later, when evaluating the similarity between the processed features calculated from the time series data of the skeleton data SD generated based on the attachment location information and the features of each of the multiple motion data stored in the motion data storage unit 221, it may be better to convert each skeleton data to the same skeletal information (bone length, bone thickness, etc.).

[0083] Therefore, posture estimation unit 131 may convert the skeleton of each part of skeleton data SD into a reference skeleton, and convert skeleton data SD into reference skeleton data. However, if similarity evaluation is performed using features independent of the skeleton, posture estimation unit 131 does not need to convert skeleton data SD into reference skeleton data. For example, the features independent of the skeleton include posture information of each part.

[0084] The posture estimation unit 131 may convert the skeleton data SD into reference skeleton data using, for example, any method, including copying the posture of each joint, scaling the root position according to the height, and adjusting the end position of each part using IK (Inverse Kinematics).

[0085] Furthermore, learning unit 243 included in server 20 may use DNN to learn to separate skeletal information and movement information of skeleton data. By using estimator 247 obtained by learning, posture estimation unit 131 may omit the process of converting skeleton data SD into reference skeleton data. In the following description, reference skeleton data may be simply referred to as skeleton data.

[0086] <3-3. Feature Calculation> In the present disclosure, feature amounts will be described by dividing them into two types: pre-processed feature amounts and post-processed feature amounts obtained by applying weighting parameters, which will be described later, to pre-processed feature amounts.

[0087] Feature calculation unit 135 calculates pre-processing feature amounts from the time-series data of the skeleton data estimated by posture estimation unit 131.

[0088] For example, the pre-processing feature amount may be the velocity, position, or posture (rotation, etc.) of each joint, or may be ground contact information.

[0089] Furthermore, the learning unit 243 may use a machine learning technique such as DNN to learn the relationship between the time-series data of the skeleton data and the pre-processing feature amounts. In this case, the feature amount calculation unit 135 calculates the pre-processing feature amounts using the estimator 247 obtained by learning. Hereinafter, with reference to FIG. 8 , an example of a method for learning the relationship between the time-series data of the skeleton data and the pre-processing feature amounts using the machine learning technique will be described.

[0090] 8 is an explanatory diagram illustrating an example of a method for learning the relationship between the time-series data of skeleton data and the pre-processing feature amounts using a machine learning technique. For example, the learning unit 243 may learn the relationship between the time-series data of skeleton data and the pre-processing feature amounts using an encoder-decoder model.

[0091] For example, when whole-body posture information of skeleton data in a time interval t to t+T is input, the learning unit 243 estimates pre-processing features using a convolutional neural network (CNN) as an encoder. Then, the learning unit 243 outputs the whole-body posture of skeleton data in a time interval t to t+T using a CNN as a decoder for the estimated pre-processing features.

[0092] 8 shows an example in which the posture of the entire body is input as time-series data of skeleton data, but the input may be, for example, information relating to other movements such as the positions and velocities of joints, or multiple pieces of information may be input. Also, the encoder-decoder model according to the present disclosure may have a more multi-layered or complex structure, or may use other machine learning techniques such as a recurrent neural network (RNN).

[0093] Furthermore, the learning unit 243 may use Deep Metric Learning to learn the relationship between the time-series data of the skeleton data and the pre-processing feature amounts. For example, the learning unit 243 may use Triplet Loss to learn the relationship between the time-series data of the skeleton data and the pre-processing feature amounts.

[0094] When using Triplet Loss, data similar to a certain input (anchor) (positive data) and data dissimilar to the anchor (negative data) may be artificially prepared, or a similarity evaluation method for time series data may be used. Alternatively, data that is close in time may be considered similar, and data that is distant in time may be considered non-similar. Note that the similarity evaluation method for time series data includes, for example, DTW (Dynamic Time Warping).

[0095] Furthermore, class label information (e.g., kick, punch, etc.) may be added to the dataset to be trained. When class label information is added to the dataset to be trained, intermediate features used for class classification may be used as pre-processing features. Furthermore, when class labels are added to part of the dataset to be trained, training may be performed using a machine learning technique based on semi-supervised learning that combines an Encoder-Decoder Model and Triplet Less.

[0096] FIG. 9 is an explanatory diagram illustrating an example of a method for calculating pre-processing feature amounts for each part according to the present disclosure.

[0097] As shown in Figure 9, if the parts of the whole body are divided into five parts: head (Head), torso (Body), right hand (RArm), left hand (LArm), right leg (RLeg), and left leg (Lleg), the learning unit 243 may use a DNN for each part of the skeleton data to learn the relationship between the time series data of each part of the skeleton data and each pre-processing feature.

[0098] For example, the learning unit 243 inputs the posture of the torso of the skeleton data in the time interval t to t+T, and estimates the pre-processing feature amount of the torso of the skeleton data using a DNN as an encoder.

[0099] Then, the feature calculation unit 135 integrates the calculated pre-processing feature of each part using a DNN as a decoder, and outputs the posture of the whole body of the skeleton data in the time interval t to t+T.

[0100] The above describes specific examples of methods for learning inputs and pre-processing features. Note that the learning unit 243 may combine the above-described methods for learning pre-processing features to learn the relationship between inputs and pre-processing features.

[0101] <3-4. Weight parameters> In the present disclosure, when searching for motion data, the user performs an action related to the search for motion data. During the time from when the user selects to start the search on the GUI until when the user selects to end the search, the feature calculation unit 135 calculates feature amounts for each predetermined time interval from the time-series data of the skeleton data indicating the user's movements.

[0102] Furthermore, the feature calculation unit 135 calculates pre-processing feature amounts for each part of the skeleton data that indicates the user's movement. For example, when the user performs a kicking motion, the feature calculation unit 135 calculates not only the pre-processing feature amount of the user's kicking foot, but also the pre-processing feature amount for each part of the body, such as the head and hands.

[0103] However, when searching for motion data, there are cases where feature amounts for all time intervals or feature amounts for all body parts are not necessarily important. Therefore, the feature amount calculation unit 135 according to the present disclosure calculates post-processing feature amounts by applying weighting parameters prepared for each time or each body part to pre-processing feature amounts for each time or each body part calculated from the time-series data of the movement of the skeleton data.

[0104] 10 is an explanatory diagram illustrating an example of a method for calculating a processed feature by applying a weighting parameter to a pre-processing feature. As shown in Fig. 10, the feature calculation unit 135 calculates a processed feature am by applying a weighting parameter wm to each dimension or each time of the pre-processing feature bm of one part j.

[0105] The unprocessed feature value bm of the part j is bm j ∈R M×T Here, M indicates the number of dimensions in the feature direction, and T indicates the number of time intervals obtained by dividing the time direction into predetermined time intervals. That is, FIG. 10 shows an example in which the number of dimensions M in the feature direction and the number of time intervals T in the time direction are 5. Note that the number of dimensions M in the feature direction may be single or multiple. Furthermore, the weight parameter wm and the post-processing feature am are expressed using the same number of rows and columns as the pre-processing feature bm.

[0106] 10, the magnitude of each feature included in the pre-processing feature, each parameter included in the weighting parameters, and each feature included in the post-processing feature is expressed by a shade of color. Note that in Fig. 10, the shade of each feature included in the pre-processing feature bm is expressed by a single value, and the shade of each parameter included in the weighting parameters wm and the shade of each feature included in the post-processing feature am are expressed by a binary value, but various values ​​may be included.

[0107] In addition, if there are multiple parts, other parts may be linked in the feature direction. For example, if the number of parts is N, the weight parameter wm is expressed as wm∈R (M×N)×T It is expressed as the determinant of

[0108] The weighting parameter wm may be set by the user on the GUI, or may be determined using an estimator 247 obtained by machine learning technology. First, an example in which the weighting parameter is set by the user will be described with reference to Fig. 11 .

[0109] Fig. 11 is an explanatory diagram for explaining an example of weighting parameters prepared for each time period. Fig. 11 shows an example in which time series data of foot acceleration acquired by a sensor device S attached to a user's foot is converted into time series data of foot velocity v.

[0110] For example, when a user performs a kicking motion, the sensor device S acquires time-series data before, during, and after the kick. If the kicking motion is found to be distinctive in the motion data search, the user may set the weighting parameters for the time periods before and after the kick to small values ​​or to zero.

[0111] For example, the user may set the weighting parameter wm for each time period using the operation display unit 110 included in the information processing terminal 10. For example, if the hatched sections shown in Fig. 11 are time periods in which the user performed a kicking motion, the user may set the weighting parameter wm for each time period to acquire the feature amount of the hatched sections.

[0112] If the hatched sections are referred to as adopted sections and the sections other than the adopted sections are referred to as non-adopted sections, the weight parameter wm t may be set using Equation 1 below: wm t =1 / L (adopted section) wm t =0 (non-adopted section) Σwm t =1 (Equation 1)

[0113] In addition, L in the formula 1 is the time length of the adopted section.

[0114] The feature calculation unit 135 applies a weight parameter wm set for each time to the pre-processing feature for each time. t By using Equation 1 as above, for example, the feature amount of the time section in which the user performed the kicking motion can be calculated as the processed feature amount.

[0115] Next, the weight parameter wm set for each part j An example of calculating the post-processing feature amount using the following formula will be described.

[0116] For example, when searching for motion data of a kicking motion, the user selects a weight parameter wm Leg the weight parameters wm j may be set to be larger than

[0117] The weighting parameter wm may be set by the user using the operation display unit 110, or may be automatically set by the feature calculation unit 135. For example, assuming that a moving part is important, the feature calculation unit 135 may set the weighting parameter wm of a part whose velocity or velocity change is equal to or greater than a predetermined value. j The weight parameter wm j may be set to a small value.

[0118] Furthermore, the learning unit 243 may learn the relationship between the pre-processing feature amount and the weight parameter wm in addition to learning the relationship between the time-series data of the skeleton data and the pre-processing feature amount.

[0119] Fig. 12 is an explanatory diagram illustrating an example of a weight parameter learning method. The learning unit 243 learns the relationship between the posture of each part of the skeleton data and the pre-processing feature of each part in the time interval t to t+T, using the pre-processing feature calculation method described with reference to Fig. 9.

[0120] Furthermore, the learning unit 243 may input the posture of the whole body and the posture of each part of the skeleton data in the time interval t to t+T, and use the attention of the DNN to learn the relationship between the pre-processing feature for each part and the weight parameter for each part. Similarly, the learning unit 243 may input the posture of the whole body and the posture of each part of the skeleton data, and use the attention of the DNN to learn the relationship between the pre-processing feature for each time and the weight parameter for each time. In this case, the feature calculation unit 235 determines the weight parameter for each time and the weight parameter for each part using the estimator 247 obtained by learning.

[0121] <3-5. Similarity evaluation> The information processing terminal 10 transmits information on the processed feature to the server 20. Then, the similarity evaluation unit 239 included in the server 20 evaluates the similarity between the received processed feature and the feature of the motion data stored in the motion feature storage unit 225.

[0122] The similarity evaluation unit 239 may perform similarity evaluation using, for example, squared error. For example, the unprocessed feature quantity of a region j in a time interval t and a dimension m is query f j t、m The feature of the motion data is dataset f j t、m and the weight parameter is w j t、m and the similarity is s. In this case, the similarity evaluation unit 239 uses Equation 2 to evaluate the similarity between the processed feature amount and the feature amount of the motion data.

[0123] 1 / s=Σ j、t、m w j t、m ( query f j t、m - dataset f j t、m ) (Equation 2)

[0124] Furthermore, the similarity evaluation unit 239 may perform the similarity evaluation using, for example, a correlation coefficient. More specifically, the similarity evaluation unit 239 uses Equation 3 to evaluate the similarity between the processed feature amount and the feature amount of the motion data.

[0125] s=Σ j、m {(Σ j t、m query f j t、m × dataset f j t、m ) / (| query f j m |2×| dataset f j m |2)} (Equation 3)

[0126] Then, the server 20 transmits motion data according to the result of the similarity evaluation by the similarity evaluation unit 239 to the information processing terminal 10. For example, the similarity evaluation unit 239 may calculate the similarity between the received processed feature and each feature of the plurality of motion data, and the server 20 may transmit a predetermined number of motion data with the highest similarity to the information processing terminal 10 as search results.

[0127] The user may also perform an operation to exclude motion data with a high degree of similarity from the search results. In this case, motion data whose similarity is determined by the similarity evaluation unit 239 to be equal to or greater than a predetermined value is excluded from the search results.

[0128] <3-6. Correction> The motion data acquired according to the similarity evaluation may be the user's whole body motion or the motion of a part with a large weight parameter, which may be the motion that the user particularly desires. However, the motion of all parts of the motion data does not necessarily match or resemble the motion that the user desires.

[0129] Therefore, the correction unit 143 may execute a process of correcting the feature amounts of the motion data for at least one or more parts of the motion data acquired as a search result. An example of the process of correcting the feature amounts of the motion data will be described below with reference to FIG.

[0130] Fig. 13 is an explanatory diagram for explaining an example of a process for correcting the feature amount of motion data. In Fig. 13, skeleton data indicating the user's motion acquired by the sensor device S is defined as a query Q(t), and skeleton data of the motion data acquired as a search result is defined as a search result R(t).

[0131] For example, if a user wants to correct the position and movement of the left hand in the search result R(t) to the position and movement of the query Q(t), the correction unit 143 may perform a process to correct the search result based on the setting ratio set by the user as described above.

[0132] For example, the correction unit 143 executes a process of correcting the feature amounts of the motion data by mixing the processed feature amounts with the feature amounts of the motion data for at least one or more parts of the motion data received as a search result from the server 20. As a result, the correction unit 143 acquires a corrected search result R'(t) by mixing the query Q(t) and the search result R(t).

[0133] Furthermore, the correction unit 143 may correct the part designated by the user as the correction target so that the part is positioned in the same position as the query Q(t).

[0134] For example, the correction unit 143 may use the posture of the search result R(t) as an initial value and perform correction processing using IK so that the position of the distal part of the search result R(t) matches the position of the query Q(t). Note that when correcting the position of the part, there is a possibility that the position of the waist between the query Q(t) and the search result R(t) may be misaligned, so for example, the correction unit 143 may perform correction processing based on the relative position from the waist.

[0135] The part to be corrected may be specified by the user using the operation display unit 110, or may be automatically specified by the correction unit 143, for example.

[0136] When the correction unit 143 automatically specifies the parts to be corrected, the correction unit 143 may determine the parts to be corrected based on weight parameters prepared for each part. For example, the correction unit 143 may adopt the feature amounts of the search result R(t) for parts whose weight parameters satisfy a predetermined criterion, and may perform a process of correcting parts whose weight parameters do not satisfy the predetermined criterion to the post-processing feature amounts of the query Q(t).

[0137] Note that even if the user sets a set ratio between the post-processing feature amount of the query Q(t) and the feature amount of the search result R(t) on the GUI, the correction unit 143 may not necessarily perform correction processing based on the set ratio. For example, if performing correction processing according to the set ratio would result in the whole body of the motion data being out of balance, the correction unit 143 may perform processing to correct the feature amounts of parts and other parts according to the positional relationship of each part.

[0138] The details of the present disclosure have been described above. Next, an example of the operation process of the system according to the present disclosure will be described.

[0139] <<4. Example of operation>> <4-1. Example of operation of information processing terminal> FIG. 14 is an explanatory diagram for explaining an example of operation processing related to a search for motion data of the information processing terminal 10 according to the present disclosure.

[0140] As shown in FIG. 14, the information processing terminal 10 acquires time-series data of the movement of the object from the sensor device S (S101).

[0141] Next, the posture estimation unit 131 generates skeleton data from the acquired time-series data of the movement of the object (S105).

[0142] Then, posture estimation unit 131 converts the skeleton of each part of the generated skeleton data into a reference skeleton, and generates reference skeleton data (S109).

[0143] Then, the feature amount calculation unit 135 calculates the pre-processing feature amount of each part of the reference skeleton data from the time-series data of the reference skeleton data (S113).

[0144] Next, the feature calculation unit 135 applies weighting parameters set for each time period or each part to the pre-processing feature to calculate the post-processing feature (S117).

[0145] Next, the communication unit 120, under the control of the search request unit 139, transmits a signal including information on the calculated post-processing feature amount to the server 20 (S121).

[0146] Then, the communication unit 120 receives a signal including information on the motion data searched by the server 20 in accordance with the transmitted information on the processed feature amount (S125).

[0147] Then, the correction unit 143 corrects the feature amount of the motion data based on a set ratio between the processed feature amount and the feature amount of the acquired motion data (S129).

[0148] Then, the operation display unit 110 displays the corrected motion data generated based on the corrected feature amount of the motion data (S133), and the information processing terminal 10 ends the operation processing related to the search for motion data.

[0149] Next, an example of the operation process relating to the search for motion data of the server 20 from S121 to S125 will be described.

[0150] <4-2. Server operation example> FIG. 15 is an explanatory diagram for explaining an example of an operation process related to a motion data search by the server 20 according to the present disclosure.

[0151] First, the communication unit 210 receives the processed feature from the information processing terminal 10 (S201).

[0152] Next, the similarity evaluation unit 239 calculates the similarity between the received processed feature amount and each feature amount of the plurality of motion data stored in the motion feature amount storage unit 225 (S205).

[0153] Then, the similarity evaluation unit 239 acquires a predetermined number of motion data items in descending order of similarity as search results (S209).

[0154] Then, the communication unit 210 transmits the predetermined number of pieces of motion data acquired in S209 to the information processing terminal 10 as search results (S213), and the server 20 ends the operation process related to the search for motion data.

[0155] An example of the operation process of the system according to the present disclosure has been described above. Next, an example of the effects of the present disclosure will be described.

[0156] <<5. Examples of effects>> According to the present disclosure described above, various effects can be obtained. For example, the feature calculation unit 135 calculates post-processing feature values ​​by applying weighting parameters prepared for each body part to pre-processing feature values ​​calculated from time-series data of the user's movements. This may enable a search for motion data that focuses on more important body parts.

[0157] Furthermore, the feature calculation unit 135 calculates processed feature values ​​by applying weighting parameters prepared for each time period to the pre-processing feature values ​​for each time period calculated from the time-series data of the user's movements. This may enable a search for motion data that focuses on time periods with higher importance.

[0158] Furthermore, since the weighting parameters are determined using the estimator 247 obtained by machine learning technology, the user does not need to manually input the weighting parameters, which can improve user convenience.

[0159] Furthermore, the information processing terminal 10 acquires as search results a predetermined number of motion data in order of the degree of similarity between the processed feature calculated from the time-series data of the skeleton data showing the user's movements and the feature of each of the plurality of motion data, thereby allowing the user to select motion data that includes particularly desired motion information from the plurality of presented motion data.

[0160] In one embodiment of the present disclosure, each of the skeleton data representing the user's movements and the skeleton data representing the motion data is converted into reference skeleton data, and feature quantities of the reference skeleton data are compared with each other, thereby reducing the possibility of search errors due to differences between the skeletons of the user and the motion data.

[0161] Furthermore, the correction unit 143 corrects the feature amount of the motion data for at least one body part by mixing the processed feature amount with the feature amount of the motion data at a set ratio. This allows the movement of the body part in the motion data to be corrected to the movement of the body part that the user requires, thereby further improving user convenience.

[0162] <<6. Hardware configuration example>> The embodiments of the present disclosure have been described above. Information processing such as the generation of skeleton data and the extraction of features described above is realized by cooperation between software and the hardware of the information processing terminal 10 described below. Note that the hardware configuration described below can also be applied to the server 20.

[0163] 16 is a block diagram showing the hardware configuration of information processing terminal 10. Information processing terminal 10 includes a CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, a RAM (Random Access Memory) 1003, and a host bus 1004. Information processing terminal 10 also includes a bridge 1005, an external bus 1006, an interface 1007, an input device 1008, an output device 1010, a storage device (HDD) 1011, a drive 1012, and a communication device 1015.

[0164] The CPU 1001 functions as an arithmetic processing unit and a control unit, and controls the overall operation of the information processing terminal 10 in accordance with various programs. The CPU 1001 may also be a microprocessor. The ROM 1002 stores programs used by the CPU 1001, calculation parameters, etc. The RAM 1003 temporarily stores programs used in the execution of the CPU 1001, parameters that change as appropriate during the execution, etc. These are interconnected by a host bus 1004 that includes a CPU bus, etc. The functions of the posture estimation unit 131 and the feature calculation unit 135 described with reference to FIG. 2 can be realized by cooperation between the CPU 1001, the ROM 1002, the RAM 1003, and software.

[0165] The host bus 1004 is connected to an external bus 1006 such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 1005. It is not necessary to configure the host bus 1004, bridge 1005, and external bus 1006 separately, and these functions may be implemented on a single bus.

[0166] The input device 1008 is composed of input means such as a mouse, keyboard, touch panel, buttons, microphone, switches, and levers for the user to input information, and an input control circuit that generates an input signal based on the user's input and outputs it to the CPU 1001. The user of the information processing terminal 10 can input various data to the information processing terminal 10 and instruct processing operations by operating the input device 1008.

[0167] The output device 1010 includes, for example, a display device such as a liquid crystal display device, an OLED device, and a lamp. Furthermore, the output device 1010 includes an audio output device such as a speaker and a headphone. The output device 1010 outputs, for example, reproduced content. Specifically, the display device displays various information such as reproduced video data as text or images. Meanwhile, the audio output device converts reproduced audio data and the like into audio and outputs the audio.

[0168] The storage device 1011 is a device for storing data. The storage device 1011 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, and a deleting device for deleting data recorded on the storage medium. The storage device 1011 is configured, for example, with an HDD (Hard Disk Drive). This storage device 1011 drives a hard disk and stores programs executed by the CPU 1001 and various data.

[0169] The drive 1012 is a reader / writer for a storage medium, and is built into or externally attached to the information processing terminal 10. The drive 1012 reads information recorded on a removable storage medium 30, such as an attached magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs the information to the RAM 1003. The drive 1012 can also write information to the removable storage medium 30.

[0170] The communication device 1015 is, for example, a communication interface configured with a communication device for connecting to the network 1. The communication device 1015 may be a wireless LAN compatible communication device, a LTE (Long Term Evolution) compatible communication device, or a wired communication device that performs wired communication.

[0171] <<7. Supplementary Information>> Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that a person skilled in the art to which the present disclosure pertains can conceive of various modifications or alterations within the scope of the technical ideas set forth in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0172] For example, the information processing terminal 10 may further include all or part of the functional configuration of the server 20 according to the present disclosure. When the information processing terminal 10 includes all of the functional configuration of the server 20 according to the present disclosure, the information processing terminal 10 may execute a series of search-related processes without communicating via the network 1. Furthermore, when the information processing terminal 10 includes part of the functional configuration of the server 20 according to the present disclosure, for example, the information processing terminal 10 may receive a plurality of motion data from the server 20 in advance by communicating via the network 1. Then, the information processing terminal 10 may evaluate the similarity between the processed feature calculated by the feature calculation unit 135 and the plurality of motion data received in advance from the server 20, and search for motion data according to the similarity evaluation result.

[0173] The steps in the processing of the information processing terminal 10 and the server 20 in this specification do not necessarily have to be processed in chronological order according to the order described in the flowcharts. For example, the steps in the processing of the information processing terminal 10 and the server 20 may be processed in an order different from the order described in the flowcharts.

[0174] It is also possible to create a computer program that causes hardware such as a CPU, ROM, and RAM built into the information processing terminal 10 to perform functions equivalent to those of the above-described components of the information processing terminal 10. A storage medium storing the computer program is also provided.

[0175] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0176] The following configurations also fall within the technical scope of the present disclosure. (1) an acquisition unit that acquires processed feature values, which are feature values ​​calculated by applying weighting parameters prepared for each time or each body part to pre-processed feature values, which are feature values ​​for each time or each body part of the object, calculated from time-series data of the movement of the object; a search unit that searches for motion data using the processed feature amount acquired by the acquisition unit; An information processing device comprising: (2) the weighting parameters to be applied to the unprocessed features are determined by an estimator obtained by learning the relationship between the features for each body part and the weighting parameters for each body part. The information processing device according to (1) above. (3) The weighting parameters to be applied to the unprocessed features are determined by an estimator obtained by learning the relationship between the feature values ​​for each time and the weighting parameters for each time. The information processing device according to (1) or (2). (4) The search unit calculating a similarity between the processed feature amount of the object acquired by the acquisition unit and each feature amount of a plurality of motion data, and searching for motion data based on the calculation result of the similarity; The information processing device according to any one of (1) to (3). (5) The search unit acquiring, as search results, a predetermined number of motion data items in descending order of feature similarity with the processed feature items based on the calculation results of the similarity; The information processing device according to (4) above. (6) The acquisition unit The motion data is searched by comparing processed feature amounts calculated from time-series data of the motion of a reference object obtained by converting the skeleton of the object into a reference skeleton with feature amounts calculated from reference motion data obtained by converting the skeleton of skeleton data into the reference skeleton. The information processing device according to (4) or (5). (7) The information processing device includes: a correction unit that corrects the feature amount of the motion data by mixing the feature amount of the motion data with the processed feature amount at a set ratio for at least one part; The information processing device according to any one of (1) to (6), further comprising: (8) The feature amount for each part of the target includes at least one of velocity, position, or posture. The information processing device according to any one of (1) to (7). (9) acquiring post-processing features, which are features calculated by applying weighting parameters prepared for each time or each body part to pre-processing features, which are features calculated for each time or each body part of the object from time-series data of the movement of the object; retrieving motion data using the acquired processed features; 2. A computer-implemented information processing method, comprising: (10) On the computer, an acquisition function for acquiring processed feature values, which are feature values ​​calculated by applying weighting parameters prepared for each time or each body part to pre-processed feature values, which are feature values ​​for each time or each body part of the object calculated from time-series data of the movement of the object; a search function that searches for motion data using the processed feature amount acquired by the acquisition function; A program that makes this happen. [Explanation of symbols]

[0177] 10 Information processing terminal 20 servers 110 Operation display section 120 Communications Department 130 Control Unit 131 Posture estimation section 135 Feature calculation unit 139 Search Request Section 143 Correction Unit 210 Communications Department 220 Storage section 221 Motion data storage unit 225 Motion feature memory unit 230 Control Unit 231 Reference Skeleton Transformation Unit 235 Feature Calculation Unit 239 Similarity Evaluation Unit 243 Learning Department 247 Estimator

Claims

1. An information processing method executed by an information processing device that searches for motion data, comprising: Obtaining user motion data; acquiring a feature amount of the motion data; Searching for similar motions based on the feature amount; generating animation data using the similar motion; Modifying animation data using the similar motion; Including, Information processing methods.

2. A plurality of the similar motions are linked together to generate animation data. The information processing method according to claim 1 .

3. generating motion data corresponding to a correction section connecting the plurality of similar motions, thereby generating animation data; The information processing method according to claim 1 .

4. The feature amount is extracted for each time period or each part of the body. The information processing method according to claim 1 .

5. applying a weight parameter to the feature amount to obtain a processed feature amount, and searching for the similar motion using the processed feature amount; The information processing method according to claim 4.

6. The feature quantities are acquired using a model obtained by pre-learning the user's motion data and motion class labels. The information processing method according to claim 1 .

7. An information processing system for searching motion data, a motion data acquisition unit that acquires user motion data; a feature acquisition unit that acquires feature amounts of the motion data; a search unit that searches for similar motions based on the feature amount; a generation unit that generates animation data using the similar motion; and The generation unit corrects the animation data using the similar motion. Information processing system.

8. The generation unit generates animation data by connecting a plurality of the similar motions. The information processing system according to claim 7 .

9. the generation unit generates motion data corresponding to a correction section connecting the plurality of similar motions to generate animation data. The information processing system according to claim 7 .

10. The feature amount is extracted for each time period or each part of the body. The information processing system according to claim 7 .

11. the feature amount acquisition unit acquires processed feature amounts by applying weighting parameters to the feature amounts; the search unit searches for the similar motion using the processed feature amount. The information processing system according to claim 10.

12. the feature acquisition unit acquires the feature using a model obtained by pre-learning user motion data and motion class labels; The information processing system according to claim 7 .

Images