Learning device, operational information conversion device, and learning method

Abstract

To provide an operation method that allows intuitive operation by a user even for an operational object having a structure different from a posture input location.SOLUTION: A learning device conducting machine learning for the processing of converting a user's input attitude into an output attitude of an operational object having a structure different from that of the user's posture input location comprises a data acquisition part that acquires teacher data including input motion data that is the time-series variation of the input attitude, and output motion data that is the time-series variation of the output attitude representing an operation corresponding to the input motion data, and a learning processing part that performs learning processing using pair data of the input attitude and the output attitude at corresponding points in time among the input motion data and the output motion data, and generates a learned model that takes the input attitude as an input and the corresponding output attitude as an output.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a learning device, an operation information conversion device, and a learning method. [Background technology]

[0002] There is known a technique for operating an object by a user's motion. Examples of the object include a character in a virtual space of a game or VR (Virtual Reality), and a robot remotely operated in a real space. The user's posture is measured over time, and the measured posture is reflected in the object, thereby operating the object. The object operated by the motion usually has a structure similar to the structure of the user's posture input part. For example, the posture of the entire body of the user is used as an input to operate the entire body of a humanoid character, or the posture of the user's hand is used as an input to operate the hand of a humanoid character. It is difficult to operate an object by motion, which has a structure different from that of the posture input part, such as the number, arrangement, and range of motion of joints. This is because it is difficult to appropriately determine the correspondence relationship (mapping between input and output) between the input motion of the user and the motion of the object as an output. An object having a structure different from that of the posture input part is, for example, an object having a non-humanoid structure, such as an animal having a structure different from that of a human.

[0003] As a method for operating an operation target having a non-humanoid structure, for example, Non-Patent Document 1 describes an operation method for operating a three-dimensional model (avatar) of an animal such as an elephant as the operation target by the hand input postures of both hands of the user. Non-Patent Document 1 proposes a method in which the authors manually associate each posture of the right and left hands with the joints of the avatar, and then optimize the correspondence based on a mathematical formula that expresses the ease of hand movement from the viewpoint of biomechanics. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Y. Jiang, Z. Li, M. He, D. Lindlbauer, and Y. Yan, “HandAvatar: Embodying Non-Humanoid Virtual Avatars through Hands”, Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, USA, Association for Computing Machinery, Apr.2023, p.1-17. [online]<URL:https: / / doi.org / 10.1145 / 3544548.<3581027>, [searched on September 21, 2023]

Summary of the Invention

Problems to be Solved by the Invention

[0005] From the perspective of operability by motion, it is important that the difference between the input-output correspondence assumed by the user and the correspondence actually used by the operation system for calculation is small. In the method for determining the correspondence according to the above Non-Patent Document 1, the correspondence assumed by the user, that is, the intuitive operation image of the user, cannot be considered. When the user uses an operation method different from the intuitive operation image, the time and workload required to acquire the operation method increase. Therefore, it is desired to provide an operation method that enables intuitive operation by the user even for an operation target having a structure different from the posture input part.

[0006] The present invention has been made in view of the above, and an object of the present invention is to provide a learning device, an operation information conversion device, and a learning method capable of providing an operation method that enables intuitive operation by the user even for an operation target having a structure different from the posture input part.

Means for Solving the Problems

[0007] The learning device of the present invention is a learning device that machine-learns the process of converting a user's input posture into an output posture of an object to be operated that has a structure different from the structure of the user's posture input part, and is equipped with a data acquisition unit that acquires teacher data including input motion data which is a time series change in the input posture and output motion data which is a time series change in the output posture which represents an action corresponding to the input motion data, and a learning processing unit that performs a learning process using pair data of an input posture and an output posture at corresponding points in time among the input motion data and the output motion data, and generates a learned model that uses the input posture as input and the corresponding output posture as output.

[0008] The operation information conversion device according to the present invention is an operation information conversion device that converts a user's input posture into an output posture of an operation target having a structure different from the structure of the user's posture input part, and includes an input unit that acquires the user's input posture, an operation information generation unit that generates an output posture from the input posture input to the input unit using paired data of an input posture and an output posture at corresponding points in time out of input motion data which is a time series change in the input posture and output motion data which is a time series change in the output posture which represents an action corresponding to the input motion data, by using a trained model that has learned a process in which the input posture is input and the corresponding output posture is output, and an output unit that outputs the generated output posture.

[0009] The learning method of the present invention is a machine learning method for converting a user's input posture into an output posture of an object to be operated that has a structure different from the structure of the user's posture input part, and includes the steps of acquiring teacher data including input motion data which is a time series change in the input posture and output motion data which is a time series change in the output posture which represents an action corresponding to the input motion data, and performing a learning process using paired data of an input posture and an output posture at corresponding points in time among the input motion data and the output motion data, and generating a learned model which uses the input posture as input and the corresponding output posture as output. Effect of the Invention

[0010] According to the present invention, it is possible to provide an operation method that allows a user to intuitively operate an operation target having a structure different from that of a posture input part. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of a learning device and an operation information conversion device according to the first embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating an example of the configuration of the learning device. [Diagram 3] FIG. 3 is a schematic diagram showing an example of a VR device. [Figure 4] FIG. 4 is a schematic diagram for explaining training data used in machine learning. [Diagram 5] FIG. 5 is a schematic diagram showing an example of an image presented to a training user when teacher data is acquired. [Figure 6] FIG. 6 is an explanatory diagram showing an example of a predefined action of an operation target. [Figure 7] FIG. 7 is an explanatory diagram showing an example of a predefined action of an operation target. [Figure 8] FIG. 8 is a diagram for explaining the processing by the preprocessing unit. [Figure 9] FIG. 9 is a graph showing time series data before (A) and after (B) the synchronization process. [Figure 10] FIG. 10 is an explanatory diagram for explaining an example of machine learning by the learning processing unit. [Figure 11] FIG. 11 is a flowchart showing the learning method according to the first embodiment. [Figure 12] FIG. 12 is a schematic diagram showing an example of an operation system including the operation information conversion device according to the first embodiment. [Figure 13] FIG. 13 is a diagram illustrating a trained model according to the second embodiment. [Figure 14] FIG. 14 is a diagram illustrating a trained model according to the third embodiment. [Figure 15]FIG. 15 is a diagram illustrating a trained model according to the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of a learning device, an operation information conversion device, and a learning method according to the present invention will be described with reference to the drawings. Note that the present invention is not limited to these embodiments. In addition, the components in the following embodiments include those that are replaceable and easy for a person skilled in the art, or those that are substantially the same.

[0013] [First embodiment] (Overview of the learning device and operation information conversion device) Fig. 1 is a diagram illustrating an example of a learning device 100 and an operation information conversion device 200 according to the first embodiment. The learning device 100 illustrated in Fig. 1 is a learning device that performs machine learning on a process of converting an input posture 1 of a user into an output posture 2 of an operation target MO having a structure different from the structure of a posture input part of the user.

[0014] The input posture 1 is information indicating the posture of the posture input part of the user. The posture input part of the user is at least a part of the user's body. The input posture 1 is generated by measuring the posture of the posture input part of the user using a measurement device. The posture input part is a part of the user's body that is a measurement target whose posture is measured by the measurement device. Therefore, the input posture 1 may be information obtained by converting the posture of the entire body of the user into data, or may be information obtained by converting the posture of a part of the user's body into data. The input posture 1 includes the posture of the user's hand HA. In the following, an example will be described in which the user's hand HA is the posture input part and the posture of the user's hand HA is the input posture 1. The input posture 1 can be expressed as the position and / or direction of each joint in a three-dimensional model simulating a humanoid structure. The humanoid structure refers to a body structure that a human has, and in this specification, it particularly means a structure related to human movement. The humanoid structure is a concept that includes the main components of the body, such as one head, one neck, one torso, two arms, and two legs, the number, arrangement, and range of motion of each joint of each part.

[0015] The input posture 1 is information that defines the posture of the user. The input posture 1 includes position information of a plurality of first joints 1A, which are the user's joints. The input posture 1 is acquired sequentially in a time series. The movement (motion) of the user's body is expressed as a time series change in the input posture 1. When the input posture 1 is the posture of the user's hand HA, the first joint 1A includes the joints of each finger and the wrist joint.

[0016] The operation target MO is an object that the user intends to operate with the input posture 1. Examples of the operation target MO include a character in a virtual space of a game or VR, and a robot remotely operated in a real space. Examples of the remotely operated robot include an industrial robot, a rescue robot, an exploration robot, and a surgical robot. Such an operation target MO may be referred to as an avatar below. In the embodiment, the operation target MO has a structure different from the structure of the posture input part of the user. Such an operation target MO has, for example, a non-human structure. That is, the operation target MO has a main body configuration, the number of joints of each part, the arrangement, and a movable range, which are partly or entirely different from the body structure of a human (human structure). In FIG. 1, a frog-shaped character is shown as an example of an operation target MO having a non-human structure. In addition, the operation target MO may have a human structure but a structure different from the posture input part. For example, the user's hand HA may be the posture input part and the face of a humanoid character may be the operation target MO, in which case the facial expression, eyes, mouth, etc. posture (opening and curvature of the eyes and mouth, etc.) are output as output posture 2 for input posture 1 of the hand HA. Also, for example, the user's hand HA may be the posture input part and the entire body of a humanoid character may be the operation target MO, in which case the posture of the entire body is output as output posture 2 for input posture 1 of the hand HA. The operation target MO includes a plurality of second joints 2A as movable parts. The structure of the operation target MO is represented by a three-dimensional model such as a rigid link model that shows the connection relationship of the plurality of second joints 2A.

[0017] The output posture 2 is information that defines the posture of the operation target MO. The output posture 2 includes position information of a plurality of second joints 2A, which are the joints of the operation target MO. By controlling the positions of the plurality of second joints 2A of the operation target MO according to the output posture 2, a posture of the operation target MO corresponding to the user's input posture 1 is realized. The output posture 2 is generated sequentially in a time series. The movement (motion) of the operation target MO is represented as a time series change in the output posture 2.

[0018] The learning device 100 performs a learning process using paired data of an input posture 1 and an output posture 2 at corresponding points in a motion, and generates a trained model 10 in which the input posture 1 is used as an input and the corresponding output posture 2 is used as an output. In other words, the learning device 100 causes the trained model 10 to acquire a mapping function that defines the correspondence between position information of a first joint 1A included in the input posture 1 and position information of a second joint 2A included in the output posture 2, by machine learning.

[0019] In the embodiment, pair data of an input posture 1 and an output posture 2 can be created by actually measuring the input posture 1 from a training user who creates data for machine learning. That is, a predefined motion of the operation target MO created in advance is presented (displayed) to the training user. The predefined motion is a specific motion realized by a time-series change in a predefined output posture 2. In response to this, the training user inputs a movement that he or she imagines as realizing the presented predefined motion of the operation target MO as a series of changes in the input posture 1. This allows pair data of an input posture 1 and an output posture 2 showing corresponding motions to be created.

[0020] The operation information conversion device 200 is a device that converts an input posture 1 of a user into an output posture 2 of an operation target MO having a structure different from the structure of a posture input part of the user. The operation information conversion device 200 is configured by a computer including a processor such as a CPU (Central Processing Unit) and memories such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The operation information conversion device 200 includes an input unit 110, an operation information generation unit 120, and an output unit 130.

[0021] The input unit 110 acquires an input posture 1 of a user. The input unit 110 may be connected to a motion measurement device (motion capture) that measures the movement of at least a part of the user's body (time-dependent change in posture) or may include a motion measurement device. The motion measurement method is not particularly limited. Examples of the motion measurement method include an optical method, an inertial method, an image method, and a magnetic method.

[0022] The optical method is a method in which a reflective marker attached to the measurement target (user) is detected by multiple cameras, and the three-dimensional coordinates of the reflective marker are calculated from the detection results of each camera. The inertial method is a method in which sensing devices such as gyro sensors and acceleration sensors are attached to the measurement target, and the position and orientation of the measurement part are measured. The image method is a method in which the measurement target is photographed by multiple cameras, and the three-dimensional coordinates of each part of the measurement target are calculated by image processing. The magnetic method is a method in which a magnetic sensor is attached to the measurement target, and the three-dimensional coordinates of each part of the user are calculated based on the magnetic measurement results of the magnetic sensor in a magnetic field generated by a magnetic field generator.

[0023] The operation information generation unit 120 uses the trained model 10 to process the input posture 1 input to the input unit 110 and generate an output posture 2. The trained model 10 has learned, by the learning device 100, a process of inputting the input posture 1 and outputting the corresponding output posture 2, using paired data of the input posture 1 and the output posture 2 at corresponding points in time.

[0024] The output unit 130 outputs the output attitude 2 generated by the operation information generation unit 120. The output unit 130 outputs the output attitude 2 to a system (motion control device) that controls the movement of the operation target MO in accordance with the output attitude 2. The output unit 130 may include an interface for inputting and outputting data, a communication device for performing data communication with an external device, and the like. When the operation target MO is a character in a virtual space, an image of the character in the virtual space is generated based on the output attitude 2. The motion control device is, for example, a calculation device that generates moving images of the character. When the operation target MO is a robot in the real space, an actuator provided at each joint of the robot is controlled based on the output attitude 2. The motion control device is, for example, a robot control device that controls the movement of the robot.

[0025] As a result, the operation information conversion device 200 generates an output posture 2 for the operation target MO by inputting the user's input posture 1 to the trained model 10, and outputs the generated output posture 2 to a system that controls the movement of the operation target MO. As a result, the operation target MO is controlled to achieve the posture defined by the output posture 2. The operation information conversion device 200 sequentially processes changes in the user's input posture 1. As a result, the user can operate the operation target MO by motion.

[0026] (Learning device) Fig. 2 is a block diagram showing an example of the configuration of the learning device 100. As shown in Fig. 2, the learning device 100 is configured as a computer including a processor 21, a storage unit 22, and an interface 23 for input / output. The processor 21 is composed of a CPU, a GPU (Graphics Processing Unit), etc. The storage unit 22 includes a main storage device and an auxiliary storage device such as a ROM, a RAM, a HDD (Hard Disk Drive), and an SSD (Solid State Drive). The interface 23 includes a communication device for communication between information device hardware.

[0027] The processor 21 executes a program stored in the storage unit 22 to perform processing for realizing each function of the learning device 100. The learning device 100 includes a data acquisition unit 31, a preprocessing unit 32, and a learning processing unit 33. The data acquisition unit 31, the preprocessing unit 32, and the learning processing unit 33 are functional blocks realized on the program by the processor 21. The storage unit 22 stores, in addition to the program executed by the processor 21, a learning model PT before machine learning.

[0028] The learning device 100 of this embodiment is connected to a motion measurement device 51 and an avatar control device 52 via an interface 23. The motion measurement device 51 measures the movement of at least a part of the user's body and generates an input posture 1 of the user. The avatar control device 52 controls the movement of the operation target MO (i.e., an avatar) according to the output posture 2.

[0029] The avatar control device 52 visualizes the motion of the operation target MO corresponding to the output posture 2. When the operation target MO is a character in a virtual space, the avatar control device 52 causes the presentation device 53 to generate a moving image of the character. The presentation device 53 is a device that presents the operation target MO to the user. The presentation device 53 may be composed of a wearable display device such as an HMD (Head Mounted Display), a stationary display device such as a liquid crystal display device or an EL (Electro-Luminescence) display device, a projector, or the like. When the operation target MO is a robot in the real space, the avatar control device 52 controls the posture of the robot based on the output posture 2. The avatar control device 52 acquires a moving image of the robot using a camera provided in the robot or a camera provided in the installation environment of the robot, and causes the acquired moving image to be displayed on the presentation device 53. When the user can directly view the robot, the presentation device 53 does not need to be provided.

[0030] The motion measuring device 51, the avatar control device 52, and the presentation device 53 may be integrated into a single VR device 50. FIG. 3 is a schematic diagram showing an example of the VR device 50. The VR device 50 is a headset-type HMD device worn in front of the user's eyes. The VR device 50 includes a presentation device 53 having displays for the left and right eyes, a motion measuring device 51 having a plurality of cameras for external photography, and an avatar control device 52 that generates a moving image of an avatar for the presentation device 53. The motion measuring device 51 captures an image of the user's hand HA from the VR device 50, measures the motion of each first joint 1A of the user's hand HA by image processing, and outputs a plurality of input postures 1 in a time series.

[0031] 4 is a schematic diagram for explaining teacher data 5 used in machine learning. In the learning process, a data acquisition unit 31 acquires teacher data 5 including input motion data 3 which is a time series change of an input posture 1, and output motion data 4 which is a time series change of an output posture 2 which represents a movement corresponding to the input motion data 3.

[0032] The input posture 1 includes position information of each first joint 1A measured by the motion measurement device 51. The number of first joints 1A of the user's hand HA is I, and the position information of each first joint 1A is expressed as a vector h i Let (0 <i≦I)。ベクトルh i For example, the position and direction (orientation) of the first joint 1A are included. The representation format of the position and direction is not particularly limited, but for example, a vector h i contains the position in the world coordinate system and the direction expressed in quaternion format. The set of position information of each first joint 1A at a certain time t is expressed as a vector H t Then, the vector H t is expressed by the following equation (1). The input motion data 3 is a vector H t Includes.

number

[0033] A motion executed by a user is represented by changes in each input posture 1 from the start point to the end point of the motion, that is, by input motion data 3. In this way, the input motion data 3 includes data on hand motion, which is the movement of the user's hand HA.

[0034] The output posture 2 includes position information of each second joint 2A of the operation object MO. The number of second joints 2A of the operation object MO is J, and the position information of each second joint 2A is expressed by a vector a j Let (0 <j≦J)。ベクトルa j For example, the vector a includes the position and direction (orientation) of the second joint 2A. j is expressed as a vector h i The vector A is a set of position information of each second joint 2A at a certain time t, and includes, for example, a position in the world coordinate system and a direction expressed in quaternion format. t Then, vector A t is expressed by the following equation (2). The output motion data 4 is the vector A t Includes.

number

[0035] The motion of the target MO is represented by the change in output attitude 2 from the start point to the end point of the motion, that is, by output motion data 4.

[0036] The teacher data 5 includes a pair of input motion data 3 and output motion data 4 that represent corresponding actions. Representing corresponding actions means that the motion of the operation target MO represented by the output motion data 4 is a conversion of the user's motion represented by the input motion data 3. In other words, the output motion data 4 included in the teacher data 5 is a collection of correct answer data when estimating the output posture 2 from the input posture 1.

[0037] As described above, the teacher data 5 can be created by actually measuring the input posture 1 of a training user who creates data for machine learning. Specifically, the data acquisition unit 31 acquires input motion data 3 generated by measuring the time series change of the input posture 1 corresponding to the predefined motion in response to the presentation of the predefined motion of the operation target MO realized by the output motion data 4.

[0038] FIG. 5 is a schematic diagram showing an example of an image presented to a training user when teacher data 5 is acquired. A training image TI as shown in FIG. 5 is presented to a training user wearing the VR device 50 shown in FIG. 3. The training image TI in FIG. 5 shows a virtual space in which a frog avatar, which is an operation target MO, and a three-dimensional model HY of a hand HA of the training user are arranged. A motion measurement device 51 measures position information of each first joint 1A of the training user in time series at a predetermined sampling period. An avatar control device 52 reflects the measured position information in the training image TI in real time to the three-dimensional model HY.

[0039] The data acquisition unit 31 transmits the output motion data 4 indicating the predefined motion of the operation target MO to the avatar control device 52. In response to this, the avatar control device 52 causes the presentation device 53 to display the predefined motion of the operation target MO in the training image TI.

[0040] 6 and 7 are explanatory diagrams showing examples of predefined movements of the control target MO. FIG. 6 and FIG. 7 show each stage of the predefined movements of the control target MO frame by frame using a plurality of images. The predefined movement example shown in FIG. 6 is a movement in which the control target MO jumps from a basic posture of the control target MO toward the front of the control target MO. FIG. 6 shows an image 61A of a basic posture, an image 61B of an intermediate posture of the jump, and an image 61C of a landing posture. The predefined movement example shown in FIG. 7 is a movement in which the control target MO rotates to the side and falls (flips over) from a basic posture of the control target MO. FIG. 7 shows an image 62A of a basic posture, an image 62B of an intermediate posture during rotation, and an image 62C of a posture lying on one's back.

[0041] The data acquisition unit 31 acquires measurement data of the input posture 1 from the motion measurement device 51 while presenting the predefined motion of the operation target MO. While visually recognizing the predefined motion of the operation target MO, the training user moves his / her hand HA in a motion corresponding to the predefined motion. In other words, the training user imitates (mimics) the predefined motion of the operation target MO by the movement of his / her hand HA. As a result, the data acquisition unit 31 acquires input motion data 3 representing a motion that the training user intuitively recognizes as corresponding to the predefined motion.

[0042] The data acquisition unit 31 supplies the pre-processing unit 32 with teacher data 5, which is made up of a pair of input motion data 3 and output motion data 4, each of which represents a corresponding action.

[0043] The pre-processing unit 32 adjusts the time difference between the input motion data 3 and the output motion data 4. When a training user imitates a predefined motion of the operation target MO, the timing at which the user starts the motion is usually delayed in time with respect to the predefined motion of the operation target MO. Furthermore, when the timing at which the user starts the motion is delayed, the user tries to make his / her own motion catch up with the predefined motion of the operation target MO. For this reason, pairs of each input posture 1 constituting the input motion data 3 and each output posture 2 constituting the output motion data 4 may be shifted in time. The pre-processing unit 32 adjusts this time difference.

[0044] FIG. 8 is a diagram for explaining the processing by the pre-processing unit 32. In FIG. 8, the teacher data 5 before processing by the pre-processing unit 32 is referred to as "teacher data 5A", and the teacher data 5 after processing by the pre-processing unit 32 is referred to as "teacher data 5B". In the example of FIG. 8, the pre-processing unit 32 performs a normalization process of the input motion data 3 and the output motion data 4 of the teacher data 5A, and a synchronization process for adjusting the time difference between the input motion data 3 and the output motion data 4. The pre-processing unit 32 calculates the joint velocity of each first joint 1A from the joint position coordinates of each first joint 1A in the input motion data 3. The pre-processing unit 32 normalizes the joint position and joint velocity of each first joint 1A in the input motion data 3 and the joint position of each second joint 2A in the output motion data 4. The normalization is performed for each of the x-axis, y-axis, and z-axis in the coordinate system of the input motion data 3 and the output motion data 4. Normalization involves scaling the data on each axis so that it falls within a given numerical range (for example, the range [-1, 1]).

[0045] The pre-processing unit 32 performs a synchronization process to adjust the time shift for each piece of data after the calculation and normalization of the joint velocity of the first joint 1A.

[0046] The pre-processing unit 32 performs a process of adjusting the time lag between the input motion data 3 and the output motion data 4 based on the maximum value of the joint velocities of the first joint 1A and the second joint 2A. Specifically, the pre-processing unit 32 calculates the joint velocity norm of each first joint 1A and the joint velocity norm of each second joint 2A, and extracts a pair of the first joint 1A and the second joint 2A having the largest joint velocity norm. The pre-processing unit 32 adjusts the time lag using a dynamic time warping method (DTW) for the time series data of the two extracted joints. The dynamic time warping method is a method of calculating the distance between each point of two time series data, and calculating the similarity between the time series data based on the magnitude of the shortest distance from among all combinations of the distances. The pre-processing unit 32 adjusts the time lag between the input motion data 3 and the output motion data 4 by associating the points with the maximum similarity. The points of the time series data to be associated here are the data of the individual input posture 1 constituting the input motion data 3 and the data of the individual output posture 2 constituting the output motion data 4. Therefore, the pre-processing unit 32 adjusts the time lag for each piece of input attitude 1 data and output attitude 2 data.

[0047] The reason for adjusting the time lag based on the maximum value of the joint velocity in this way is that it is considered that when a user operates the operation target MO, the input action (generation of the input posture 1) is performed while being conscious of the part that is moving the most. In this specification, "synchronization" by synchronization processing is a broad concept that includes not only matching the timing between time series data (minimizing the time lag) but also adjusting the timing lag between time series data to a desired level. For example, in applications such as remote operation over a long distance where communication delays are unavoidable, synchronization processing (adjustment of the time lag) may be performed to delay the input motion data 3 by a certain amount in consideration of the communication delays.

[0048] FIG. 9 is a graph showing each time series data before (A) and after (B) synchronization processing. In each of graphs (A) and (B), the vertical axis represents joint velocity and the horizontal axis represents time. In FIG. 9, the joint velocity 71 of each first joint 1A of the input motion data 3 is shown by a solid line, and the joint velocity 72 of each second joint 2A of the output motion data 4 is shown by a dotted line. Here, the joint velocity 71 is the average value of the joint velocity of each first joint 1A, and the joint velocity 72 is the average value of the joint velocity of each second joint 2A. It can be seen that the synchronization processing reduces the deviation in the horizontal axis direction (i.e., the time deviation) between each data of the input motion data 3 and each data of the output motion data 4.

[0049] The pre-processing unit 32 supplies the learning processing unit 33 with the teacher data 5B (the pair of the input motion data 3 and the output motion data 4) that has been subjected to the normalization processing and synchronization processing.

[0050] The learning processing unit 33 performs learning processing using paired data of an input posture 1 and an output posture 2 at corresponding points in time out of the input motion data 3 and output motion data 4 included in the teacher data 5B, and generates a trained model 10 in which the input posture 1 is used as an input and the corresponding output posture 2 is used as an output. That is, the learning processing unit 33 performs learning processing using the paired data (paired data included in the teacher data 5B) after time offset adjustment by the pre-processing unit 32.

[0051] 10 is an explanatory diagram for explaining an example of machine learning by the learning processing unit 33. The learning processing unit 33 receives input postures 1 of the input motion data 3 as inputs to the learning model PT and output postures 2 of the output motion data 4 as outputs of the learning model PT. As an example, each input posture 1 of the input motion data 3 includes data on the position, direction, and speed of each first joint 1A. Each output posture 2 of the output motion data 4 includes data on the position and direction of each second joint 2A.

[0052] In the example of FIG. 10, the learning model PT (trained model 10) has a conversion unit 11 consisting of an encoder-decoder network. That is, the conversion unit 11 includes an encoder 12 that receives an input posture 1 as an input and outputs a latent space 14, and a decoder 13 that receives the latent space 14 as an input and outputs an output posture 2. The encoder 12 includes an input layer, an output layer, and a hidden layer. The total number of hidden layers of the encoder 12 is not particularly limited, but is, for example, 3. The decoder 13 includes an input layer, an output layer, and a hidden layer. The total number of hidden layers of the decoder 13 is not particularly limited, but is, for example, 3. Note that the learning model PT (trained model 10) is not limited to an encoder-decoder model, and may be any machine learning model that can acquire a mapping function that converts an input posture 1 into a corresponding output posture 2.

[0053] The latent space 14 is information obtained by extracting features contained in the input posture 1 as vectors in space. The latent space 14 is an N-dimensional latent space vector z (z1, z2, ..., z N-1 ,z N The number of dimensions N of the latent space 14 is not particularly limited, and may be determined by the number of samples of the training data 5, the number of first joints 1A of the input posture 1 (vector h i It is set appropriately depending on the number of I) etc.

[0054] The learning processing unit 33 performs a learning process using pair data of an input posture 1 and an output posture 2 at the same time t from among the input motion data 3 and the output motion data 4 that represent corresponding motions. Then, the learning processing unit 33 performs a learning process using each pair data at each time t included in the input motion data 3 and the output motion data 4, with the input motion data 3 as input and the output motion data 4 as output. The learning processing unit 33 determines each parameter of the encoder 12 and the decoder 13 of the learning model PT by machine learning. As a result, the learning processing unit 33 generates a learned model 10 that has acquired a mapping function that defines the correspondence between the input posture 1 and the output posture 2.

[0055] As described above, the input posture 1 and output posture 2 used in machine learning represent motions corresponding to each other by the imitated movements of the training user, and are synchronized based on the movement that the user is most conscious of through preprocessing (synchronization processing). As a result of machine learning, the trained model 10 learns a process of inputting the user's input posture 1 and outputting the output posture 2 representing the movement of the operation target MO that is estimated to be imitated by the input posture 1. By performing machine learning using multiple teacher data 5 created for various predefined movements, the trained model 10 can learn the conversion from the input posture 1 to the output posture 2 for various postures included in the predefined movements and intermediate postures between them.

[0056] The learning processing unit 33 generates a trained model 10 that converts the joints of the input posture 1 and the joints of the output posture 2 in correspondence with each other. In one example, the learning processing unit 33 performs a learning process so that the position information of the first joints 1A included in the input posture 1 and the position information of the second joints 2A included in the output posture 2 corresponding to the input posture 1 are in a non-one-to-one correspondence. When the joint structure of the user and the joint structure of the operation target MO match, the correspondence between the input posture 1 and the output posture 2 is usually one-to-one correspondence between the same joints. When the joint structure of the user and the joint structure of the operation target MO are different, particularly when the number of joints of the operation target MO is greater than that of the user, if the joints are made to correspond one-to-one with each other, the postures that the operation target MO can take are restricted. When the first joint 1A of the user and the second joint 2A of the operation target MO have a non-one-to-one correspondence relationship, for example, the position of one first joint 1A affects the positions of multiple second joints 2A, or the relative positional relationship of multiple first joints 1A determines the position of one of the second joints 2A of the operation target MO. This enables the operation of various motions even when the posture vector space that can be expressed by the output posture 2 is larger than the posture vector space that can be expressed by the input posture 1. However, in operating an avatar, it is not necessary that all postures that the operation target MO can take can be realized by the input posture 1 of the user, and the input posture 1 and output posture 2 may be associated with each other by limiting it to only a part of the posture vector space of the operation target MO depending on the purpose of the operation.

[0057] (How to learn) Next, the learning method according to the first embodiment will be described. FIG. 11 is a flowchart showing the learning method according to the first embodiment. The learning method according to the embodiment is a learning method for machine learning a process of converting an input posture 1 of a user into an output posture 2 of an operation target MO having a structure different from the structure of a posture input part of the user. The learning method according to the embodiment is performed by a learning device 100. As shown in FIG. 11, the learning method according to the embodiment includes a step S10 of acquiring teacher data 5, a step S20 of performing preprocessing, and a step S30 of generating a trained model 10 by a learning process.

[0058] Step S10 of acquiring teacher data 5 acquires teacher data 5 including input motion data 3 which is a time series change in an input posture 1, and output motion data 4 which is a time series change in an output posture 2 which represents an action corresponding to the input motion data 3. That is, the data acquisition unit 31 acquires teacher data 5 which includes a pair of input motion data 3 and output motion data 4 which represent motions corresponding to each other.

[0059] In pre-processing step S20, normalization and synchronization are performed on the acquired teacher data 5 (teacher data 5A). That is, the pre-processing unit 32 calculates the joint velocities and performs normalization on the input motion data 3 and the output motion data 4, and performs synchronization to adjust the time shift on each of the obtained time-series data. In this way, the pre-processing unit 32 generates teacher data 5 after normalization and synchronization (teacher data 5B).

[0060] Step S30 of generating the trained model 10 performs a learning process using pair data of an input posture 1 and an output posture 2 at corresponding points in time among the input motion data 3 and the output motion data 4 included in the teacher data 5B, and generates the trained model 10 having the input posture 1 as an input and the corresponding output posture 2 as an output. That is, the learning processing unit 33 performs machine learning of the training model PT using a pair of the input posture 1 as an input to the training model PT and the output posture 2 as an output of the training model PT. As a result, the learning processing unit 33 generates the trained model 10 having the input posture 1 as an input and the corresponding output posture 2 as an output.

[0061] (Operation information conversion device) Next, a specific example of the operation information conversion device 200 according to the embodiment will be described. Fig. 12 is a schematic diagram showing an example of an operation system OS including the operation information conversion device 200 according to the first embodiment. Fig. 12 shows an example in which the operation information conversion device 200 according to the embodiment is applied to an operation system OS based on a client-server model.

[0062] The operation system OS is composed of a client terminal CL having a motion measurement device 51 that generates an input posture 1, an avatar control device 52 that controls the operation target MO according to an output posture 2, and a presentation device 53, and a server device SV having an operation information conversion device 200.

[0063] In this example, the client terminal CL is configured with a VR device 50 as shown in Fig. 3. The server device SV includes a processor 210, a storage unit 220, and an interface 230. The processor 210 executes a program stored in the storage unit 220 to realize a function as the operation information conversion device 200 according to the embodiment. That is, the processor 210 includes an input unit 110, an operation information generation unit 120, and an output unit 130. The input unit 110, the operation information generation unit 120, and the output unit 130 are functional blocks realized on the program by the processor 210.

[0064] The server device SV can communicate with the learning device 100 and the client terminal CL via a network. The server device SV acquires pre-processing parameters, the trained model 10 and its parameters from the learning device 100, and stores them in the storage unit 220. In addition to the trained model 10 and various parameters, the storage unit 220 stores information on three-dimensional models of various operation target MOs identified by avatar IDs. The trained model 10 is created separately for each operation target MO (for each avatar ID).

[0065] A user who operates an operation target MO selects the operation target MO for the client terminal CL. The client terminal CL transmits an avatar ID corresponding to the selected operation target MO to the server device SV. The server device SV reads out data of the operation target MO corresponding to the avatar ID and data of the trained model 10 from the storage unit 220.

[0066] The user utilizes the motion measuring device 51 of the client terminal CL to generate the input posture 1. The client terminal CL transmits the generated input posture 1 to the server device SV.

[0067] In the server device SV, the input unit 110 acquires the input posture 1 from the client terminal CL and supplies it to the operation information generating unit 120. The input posture 1 is generated every moment in accordance with the user's movement and transmitted to the server device SV. The operation information generating unit 120 performs pre-processing, conversion processing, and post-processing. The pre-processing includes calculation of joint velocities and normalization processing similar to the pre-processing unit 32 of the learning device 100. The conversion processing includes a process of generating an output posture 2 of the operation target MO by inputting the normalized input posture 1 to the trained model 10. The post-processing includes a process of normalizing (denormalizing) the output posture 2 generated by the conversion processing according to the data processing in the client terminal CL. The operation information generating unit 120 supplies the normalized output posture 2 to the output unit 130. The output unit 130 transmits the output posture 2 to the client terminal CL as a response to the input posture 1. The server device SV converts the input posture 1 transmitted from time to time into an output posture 2 one by one and transmits it to the client terminal CL.

[0068] The client terminal CL generates an image of the operation target MO corresponding to the output posture 2 using the avatar control device 52, and moves the operation target MO by outputting the image to the presentation device 53. That is, the avatar control device 52 controls the movement of the operation target MO in the virtual space based on the position and direction information included in the output posture 2. In this way, the user's motion is visualized as the motion of the operation target MO.

[0069] Although not shown in the figure, if the operation target MO is a robot in the real space, the avatar control device 52 is provided outside the client terminal CL. In this case, the server device SV transmits the output posture 2 to the avatar control device 52. The avatar control device 52 controls the operation target MO based on the output posture 2, and transmits an image showing the movement of the operation target MO to the client terminal CL. The client terminal CL presents the image acquired from the avatar control device 52 to the user by the presentation device 53. As a result, the user's motion is visualized as the motion of the operation target MO.

[0070] As described above, the learning device 100 of the first embodiment is a learning device 100 that performs machine learning to convert a user's input posture 1 into an output posture 2 of an operation target MO having a structure different from the structure of the user's posture input part, and is equipped with a data acquisition unit 31 that acquires teacher data 5 including input motion data 3 which is a time series change in the input posture 1 and output motion data 4 which is a time series change in the output posture 2 which represents an action corresponding to the input motion data 3, and a learning processing unit 33 that performs a learning process using pair data of an input posture 1 and an output posture 2 at corresponding points in time among the input motion data 3 and the output motion data 4, and generates a learned model 10 that uses the input posture 1 as an input and the corresponding output posture 2 as an output.

[0071] According to this configuration, the correspondence between the input posture 1 and the output posture 2 is not designed based on some mathematical formula, but the correspondence is learned using the input posture 1 itself by the user, thereby realizing an operation method (conversion from the input posture 1 to the output posture 2) that allows the user to intuitively operate. For this purpose, by using pair data of the input posture 1 and the output posture 2 that represent corresponding actions and correspond to each other at the time of each other in the learning process, the correspondence between the input posture 1 and the output posture 2 can be appropriately learned even for an operation target MO having a structure different from the posture input part.

[0072] In the learning device 100 according to the first embodiment, the input posture 1 includes position information of the user's joints, the output posture 2 includes position information of the joints of the operation target MO, and the learning processing unit 33 generates a trained model 10 that converts the joints of the input posture 1 in association with the joints of the output posture 2. This makes it possible to operate the operation target MO so as to realize a variety of motions in response to changes in the input posture 1, without being limited to predetermined motions.

[0073] In the learning device 100 of the first embodiment, the data acquisition unit 31 acquires input motion data 3 generated by measuring the time-series change in an input posture 1 representing a movement corresponding to a predefined movement of the operation target MO realized by output motion data 4.

[0074] According to this configuration, by actually executing the input posture 1 that the user envisions when making the target MO perform the predefined motion while viewing the predefined motion, a pair of corresponding input motion data 3 and output motion data 4 can be obtained. Typically, a user who views a predefined motion moves in a way that imitates (mimics) the motion, thereby easily generating the input posture 1 for intuitively operating the target MO.

[0075] The learning device 100 according to the first embodiment further includes a pre-processing unit 32 that adjusts the time offset between the input motion data 3 and the output motion data 4, and the learning processing unit 33 performs learning processing using the pair data after the time offset has been adjusted by the pre-processing unit 32.

[0076] According to this configuration, even if the user does not take the input posture 1 so as to sufficiently match in time with the presented predefined action, pair data of the input posture 1 and the output posture 2 at corresponding times can be generated. For example, it becomes unnecessary for the user to repeatedly play back the predefined action and input the action until the input posture 1 corresponding in time is obtained, and the workload of the training user in generating the teacher data 5 can be reduced.

[0077] In the learning device 100 according to the first embodiment, the input posture 1 includes position information of a first joint 1A which is a joint of the user, and the output posture 2 includes position information of a second joint 2A which is a joint of the operation target MO, and the pre-processing unit 32 performs processing to adjust the time lag between the input motion data 3 and the output motion data 4 based on the maximum values ​​of the joint velocities of the first joint 1A and the second joint 2A.

[0078] According to this configuration, when operating an operation object MO having a structure different from the posture input part, the time lag can be adjusted based on the posture change of the part that the user is most aware of, that is, the part that moves the most. As a result, the accuracy of the temporal correspondence in the pair data of the input posture 1 and the output posture 2 can be improved.

[0079] In the learning device 100 according to the first embodiment, the input motion data 3 includes data on hand motion, which is the movement of the user's hand HA.

[0080] According to this configuration, when the target MO is non-humanoid in structure, and it is difficult to associate parts such as the head, limbs, etc. with each other, it is possible to realize a variety of movements of the target MO by inputting operations using hand motions, which are capable of expressing the most complex and diverse movements.

[0081] In the learning device 100 of the first embodiment, the input posture 1 includes position information of multiple first joints 1A, which are the user's joints, and the output posture 2 includes position information of multiple second joints 2A, which are the joints of the operation object MO, and the learning processing unit 33 performs learning processing so that the multiple first joints 1A included in the input posture 1 and the multiple second joints 2A included in the output posture 2 corresponding to the input posture 1 have a non-one-to-one correspondence.

[0082] With this configuration, even if the body structures of the user and the operation target MO are significantly different, it is possible to prevent the operable postures of the operation target MO from being restricted by the postures that can be expressed by the input posture 1. In particular, even if the posture space that can be expressed by the output posture 2 is larger than the posture space that can be expressed by the input posture 1, a variety of motion controls are possible.

[0083] The operation information conversion device 200 according to the first embodiment is an operation information conversion device 200 that converts an input posture 1 of a user into an output posture 2 of an operation target MO having a structure different from the structure of the posture input part of the user, and includes an input unit 110 that acquires the input posture 1 of the user, an operation information generation unit 120 that generates an output posture 2 from an input posture 1 input to the input unit 110 using a trained model 10 that has learned a process of inputting the input posture 1 and outputting the corresponding output posture 2 by using pair data of an input posture 1 and an output posture 2 at corresponding points in time out of input motion data 3 which is a time series change in the input posture 1 and output motion data 4 which is a time series change in the output posture 2 that represents an action corresponding to the input motion data 3, and an output unit 130 that outputs the generated output posture 2.

[0084] According to this configuration, the correspondence between the input posture 1 and the output posture 2 is not designed based on some mathematical formula, but the trained model 10 that has learned the correspondence using the input posture 1 itself by the user can realize an operation method (conversion from the input posture 1 to the output posture 2) that allows the user to intuitively operate. For this reason, by using the trained model 10 that represents corresponding actions and has been trained using pair data of the input posture 1 and the output posture 2 at the corresponding times, appropriate correspondence between the input and output motions can be achieved even for an operation target MO having a structure different from the posture input part.

[0085] The learning method of the first embodiment is a learning method that machine-learns the process of converting a user's input posture 1 into an output posture 2 of an operation target MO having a structure different from the structure of the user's posture input part, and includes a step S10 of acquiring teacher data 5 including input motion data 3 which is a time series change in the input posture 1 and output motion data 4 which is a time series change in the output posture 2 which represents an action corresponding to the input motion data 3, and a step S30 of performing a learning process using pair data of an input posture 1 and an output posture 2 at corresponding points in time among the input motion data 3 and the output motion data 4, to generate a learned model 10 which has the input posture 1 as input and the corresponding output posture 2 as output.

[0086] According to this configuration, the correspondence between the input posture 1 and the output posture 2 is not designed based on some mathematical formula, but the correspondence is learned using the input posture 1 itself by the user, thereby realizing an operation method (conversion from the input posture 1 to the output posture 2) that allows the user to intuitively operate. For this reason, by using pair data of the input posture 1 and the output posture 2 that represent corresponding actions and correspond to each other at the time points in the learning process, the correspondence between the input and output motions can be appropriately learned even for an operation target MO that has a structure different from the posture input part.

[0087] [Second embodiment] A second embodiment will be described below. In the following description, components that are the same as or equivalent to those in the first embodiment described above are given the same reference numerals, and descriptions of those components will be simplified or omitted.

[0088] 13 is a diagram showing a trained model 10A according to the second embodiment. After being generated by the learning device 100, the trained model 10A according to the second embodiment can be additionally trained to adapt to operations of individual users when used in the operation information conversion device 200.

[0089] In the second embodiment, the trained model 10A includes a conversion unit 11 that converts an input posture 1 into an output posture 2, and an adjustment unit 15 that generates adjusted input data to be subjected to the conversion process together with the input posture 1. In the second embodiment, the adjustment unit 15 generates adjusted input data according to each individual user.

[0090] The configuration of the conversion unit 11 is the same as that of the first embodiment. The conversion unit 11 has fixed parameters determined by a learning process. The fixed parameters are parameters that are not changed in additional learning described below. The fixed parameters are parameters determined when the trained model 10A is generated by the learning device 100. The fixed parameters include parameters of each layer (input layer, hidden layer, output layer) of the encoder 12 and parameters of each layer (input layer, hidden layer, output layer) of the decoder 13. In FIG. 13, for convenience, the nodes to which the fixed parameters are set are hatched.

[0091] The adjustment unit 15 according to the second embodiment has an embedding layer that receives identification information (user ID) indicating a user and outputs a user vector 16 corresponding to the identification information. lA user vector 16 represented by the following expression is added to the latent space 14, which is the output of the encoder 12. Therefore, the decoder 13 of the conversion unit 11 receives the latent space 14 to which the user vector 16 has been added, and generates an output posture 2 of the operation target MO. By adding the user vector 16 corresponding to each individual user, input-output conversion adapted to the characteristics and tendencies of each user becomes possible. For the sake of convenience, the user who generated the input posture 1 for the teacher data 5 is referred to as a training user, and the user who uses the trained model 10A after it has been generated is referred to as a user.

[0092] The adjustment unit 15 has variable parameters that can be additionally learned to generate adjustment input data after the trained model 10A is generated by the learning process. In the example of FIG. 13, the variable parameters are parameters of the output layer of the adjustment unit 15. The variable parameters are pre-learned by the identification information of the training user in the learning process by the learning device 100. Then, when the trained model 10A is used in the operation information conversion device 200, the variable parameters are additionally learned by using the calibration data 6 by a new user. The variable parameters are determined individually for each user ID of a user.

[0093] The calibration data 6 includes pair data of a calibration input 6A and a calibration output 6B corresponding to the calibration input 6A. The calibration output 6B represents an output posture that is a correct answer to the calibration input 6A by the user himself. The calibration data 6 is generated, for example, as a calibration operation when the user starts using the operation information conversion device 200. The calibration data 6 can be acquired by the user imitating (copying) the predefined action of the operation target MO, as in the case of generating the teacher data 5. The calibration data 6 may be acquired by a method other than the imitation input. The calibration data 6 can be acquired by the user taking the input posture 1 so as to operate the operation target MO to perform a predetermined task. When the operation target MO is a character in a virtual space, the user can be made to perform a task in a manner such as a simple mini-game. When the operation target MO is a remote-controlled robot, the user can be made to perform a simple task that can be performed by the robot.

[0094] When the operation information generating unit 120 of the operation information conversion device 200 receives the calibration data 6 generated by the user, the operation information generating unit 120 performs a learning process (additional learning) of the adjustment unit 15 using the calibration data 6. The adjustment unit 15 additionally learns variable parameters for each individual user based on the input posture 1 of the user and the output posture 2 of the operation target MO. In the additional learning, the variable parameters of the adjustment unit 15 are determined so as to output a user vector 16 that minimizes the error between the output posture 2 of the conversion unit 11 when the calibration input 6A is input and the calibration output 6B. This enables the trained model 10A to adapt to the difference in the input posture 1 of the user using the user vector 16.

[0095] As described above, in the learning device 100 of the second embodiment, the trained model 10A includes a conversion unit 11 that converts an input posture 1 into an output posture 2, and an adjustment unit 15 that generates adjusted input data to be used in the conversion process together with the input posture 1, and the conversion unit 11 has fixed parameters determined by the learning process, and the adjustment unit 15 has variable parameters that allow additional learning to generate adjusted input data after the trained model 10A is generated by the learning process.

[0096] According to this configuration, in a usage scenario after the generation of the trained model 10A, the trained model 10A can be adapted to an individual user or a specific operation target MO by additional learning. As a result, an operation method corresponding to individual differences and individual differences of the operation target MO can be provided.

[0097] In the learning device 100 according to the second embodiment, the adjustment unit 15 additionally learns variable parameters for each individual user, based on the input posture 1 of that user and the output posture 2 of the operation target MO.

[0098] According to this configuration, it is possible to additionally learn the characteristics and tendencies of the input posture 1 of each user, and adapt the conversion process from the input posture 1 to the output posture 2 to the individual user. As a result, it is possible to improve the operability of the operation target MO for each individual user.

[0099] The other effects of the second embodiment are similar to those of the first embodiment.

[0100] [Third embodiment] A third embodiment will now be described. In the following description, components that are the same as or equivalent to those in the first and second embodiments described above are given the same reference numerals, and descriptions of those components will be simplified or omitted.

[0101] 14 is a diagram showing a trained model 10B according to the third embodiment. After being generated by the learning device 100, the trained model 10B according to the third embodiment can be subjected to additional learning so as to adapt to differences between individual operation targets MO when used in the operation information conversion device 200.

[0102] In the third embodiment, the trained model 10B includes a conversion unit 11 and an adjustment unit 15. In the third embodiment, the adjustment unit 15 generates adjusted input data according to each operation target MO.

[0103] For example, there may be multiple variations in the operation target MO. The variations in the operation target MO may include, for example, robots or characters that have approximately the same structure but differ in size or performance. These operation target MOs are distinguished by individual identification information (avatar ID). As shown in FIG. 12, when using the operation information conversion device 200, the user selects the operation target MO to be used, and the avatar ID of the selected operation target MO is transmitted from the client terminal CL to the server device SV.

[0104] The adjustment unit 15 according to the third embodiment has an embedding layer that receives identification information (avatar ID) indicating the operation target MO and outputs an avatar vector 17 corresponding to the identification information. The variable parameters of the adjustment unit 15 are determined individually for each avatar ID of the operation target MO. The adjustment unit 15 calculates the vector g l The avatar vector 17 represented by the following expression is added to the latent space 14, which is the output of the encoder 12. Therefore, the decoder 13 of the conversion unit 11 receives the latent space 14 to which the avatar vector 17 has been added, and generates the output posture 2 of the operation target MO. The addition of the avatar vector 17 corresponding to each operation target MO enables input-output conversion that is adapted to the differences between each operation target MO.

[0105] The variable parameters of the adjustment unit 15 are parameters of the output layer of the adjustment unit 15. The calibration data 6 is generated, for example, as a calibration operation at the start of use of an operation target MO identified by certain identification information (avatar ID). The method of acquiring the calibration data 6 is similar to that of the second embodiment, and the data can be acquired by the user actually operating the operation target MO of the identification information (avatar ID) in the input posture 1.

[0106] When the operation information generating unit 120 of the operation information conversion device 200 receives the calibration data 6 generated using an operation target MO identified by certain identification information (avatar ID), the operation information generating unit 120 performs a learning process (additional learning) of the adjustment unit 15 using the calibration data 6. The adjustment unit 15 additionally learns variable parameters for each individual operation target MO based on the output posture 2 of the operation target MO and the input posture 1 of the user. In the additional learning, the variable parameters of the adjustment unit 15 are determined so as to output an avatar vector 17 that minimizes the error between the output posture 2 of the conversion unit 11 when the calibration input 6A is input and the calibration output 6B. This enables the learned model 10B to adapt to individual differences of each avatar (operation target MO) using the avatar vector 17.

[0107] The second and third embodiments may be combined so that the adjustment unit 15 outputs the user vector 16 and the avatar vector 17, making it possible to adapt to both the user and the operation target MO.

[0108] As described above, in the learning device 100 according to the third embodiment, the adjustment unit 15 additionally learns variable parameters for each individual operation object MO, based on the output posture 2 of the operation object MO and the input posture 1 of the user.

[0109] According to this configuration, it is possible to additionally learn the differences between each individual MO to be operated, and to adapt the conversion process from the input posture 1 to the output posture 2 for each individual MO to be operated. As a result, it is possible to improve the operability of each individual MO to be operated.

[0110] The other effects of the third embodiment are similar to those of the first embodiment.

[0111] [Fourth embodiment] A fourth embodiment will be described below. In the following description, components that are the same as or equivalent to those in the first embodiment described above are given the same reference numerals, and the description of those components will be simplified or omitted.

[0112] FIG. 15 is a diagram showing a trained model 10C according to the fourth embodiment. A trained model 10C is created individually for operation targets MO with different structures (e.g., number of joints). In this case, a learning process is performed for each trained model 10C using teacher data 5. The trained model 10C (trained model PT) according to the fourth embodiment can efficiently perform learning processes by utilizing the results of past learning processes by switching the output layer 13A of the decoder 13 when performing learning processes for operation targets MO with different structures.

[0113] The trained model 10C (trained model PT) of the fourth embodiment includes an output layer 13A that outputs an output posture 2 converted based on an input posture 1, and is capable of switching between multiple output layers 13A according to multiple types of operation objects MO with different structures.

[0114] In the storage unit 22 of the learning device 100, data of the output layer 13A of each of a plurality of types of operation target MOs is stored in association with identification information (avatar ID). FIG. 13 illustrates three output layers 13A, namely, "avatar ID: 001", "avatar ID: 002", and "avatar ID: 003". Each output layer 13A outputs a different output posture 2 according to the structure of the operation target MO (such as the number of second joints 2A). When the learning processing unit 33 receives input of identification information during the learning process, it switches the output layer 13A of the decoder 13 to data of the output layer 13A corresponding to the identification information. In other words, the data obtained by switching the output layer 13A of the already created trained model 10C is used as the learning model PT to be used in a new learning process.

[0115] As a result, for example, in the learning process for generating a trained model 10C of "avatar ID: 002", it is possible to use a part of the encoder 12 and the decoder 13 (parts other than the output layer 13A) of the trained model 10C of "avatar ID: 001" that has already been created. By reusing the part of the trained model 10C other than the output layer 13A of the decoder 13 from an existing model, the learning process for generating a new trained model 10C ("avatar ID: 002") can be quickly converged. When the structures of the MOs to be operated are completely different, the output layer 13A of the decoder 13 may be completely different reflecting the difference in structure, whereas the process of the encoder 12 and the process from the latent space 14 of the decoder 13 to the output layer 13A have many common parts even if the structures of the MOs to be operated are different.

[0116] As described above, in the learning device 100 of the fourth embodiment, the learned model 10C includes an output layer 13A that outputs an output posture 2 converted based on an input posture 1, and is capable of switching between multiple output layers 13A according to multiple types of operation objects MO with different structures.

[0117] According to this configuration, a part or all of the parts of the trained model 10C other than the output layer 13A can be reused among a plurality of types of operation targets MO having different structures and different output postures 2 defined by the number of second joints 2A, etc. As a result, the amount of teacher data 5 required for the learning process can be reduced and the learning process time can be shortened.

[0118] The other effects of the fourth embodiment are similar to those of the first embodiment.

[0119] The technical scope of the present invention is not limited to the above-mentioned embodiment, and appropriate modifications can be made without departing from the spirit of the present invention. For example, the above-mentioned embodiment has been described by taking an example of an operation using a hand motion. The user's input measured as the input motion data 3 is not limited to a hand motion, and may be the movement of any part of the user's body.

[0120] In the above embodiment, the pre-processing unit 32 adjusts the time difference between the input motion data 3 and the output motion data 4, but the time difference does not have to be adjusted. In this case, the training user's motion may be repeatedly measured until the measured input motion data 3 sufficiently corresponds in time to the output motion data 4, or the training user may manually adjust the time difference while comparing the measured input motion data 3 with the output motion data 4.

[0121] In the above embodiment, an example has been described in which the pre-processing unit 32 adjusts the time lag based on the maximum values ​​of the joint velocities of the first joint 1A and the second joint 2A. However, the pre-processing unit 32 may adjust the time lag based on values ​​other than the maximum values ​​of the joint velocities.

[0122] In addition, the configurations of the second to fourth embodiments can be combined with each other. The second or third embodiment may be combined with the fourth embodiment, or the second, third, and fourth embodiments may all be combined. [Explanation of symbols]

[0123] 1...input posture, 1A...first joint, 2...output posture, 2A...second joint, 3...input motion data, 4...output motion data, 5, 5A, 5B...teaching data, 6...calibration data, 6A...calibration input, 6B...calibration output, 10, 10A, 10B, 10C...trained model, 11...conversion unit, 12...encoder, 13...decoder, 13A...output layer, 14...latent space, 15...adjustment unit, 16...user vector, 17...avatar vector, 21...processor, 22...storage unit, 23...interface Face, 31...data acquisition unit, 32...preprocessing unit, 33...learning processing unit, 50...VR device, 51...motion measurement device, 52...avatar control device, 53...presentation device, 100...learning device, 110...input unit, 120...operation information generation unit, 130...output unit, 200...operation information conversion device, 210...processor, 220...memory unit, 230...interface, CL...client terminal, HA...user's hand, HY...3D model, PT...learning model, MO...operation target, OS...operation system, SV...server device, TI...training image.

Claims

1. A learning device that performs machine learning on a process of converting an input posture of a user into an output posture of an operation object having a structure different from a structure of a posture input part of the user, a data acquisition unit that acquires teacher data including input motion data that is a time series change in the input posture and output motion data that is a time series change in the output posture that represents an action corresponding to the input motion data; a learning processing unit that performs a learning process using pair data of the input posture and the output posture at corresponding time points among the input motion data and the output motion data, and generates a learned model in which the input posture is an input and the corresponding output posture is an output. Learning device.

2. the input posture includes position information of the user's joints; the output attitude includes position information of a joint of the operation target, The learning processing unit generates a learned model that converts the joints of the input posture and the joints of the output posture in association with each other. The learning device according to claim 1 .

3. the data acquisition unit acquires the input motion data generated by measuring a time-series change in the input posture representing a motion corresponding to a predefined motion of the operation target realized by the output motion data; The learning device according to claim 1 .

4. a pre-processing unit that adjusts a time lag between the input motion data and the output motion data; The learning processing unit performs the learning process using the pair data after the time offset adjustment by the pre-processing unit. The learning device according to claim 3 .

5. the input posture includes position information of a first joint which is a joint of the user, the output attitude includes position information of a second joint which is a joint of the operation target, the pre-processing unit performs a process of adjusting a time lag between the input motion data and the output motion data based on a maximum value of joint velocities of the first joint and the second joint. The learning device according to claim 4.

6. The learned model includes a conversion unit that converts the input attitude into the output attitude, and an adjustment unit that generates adjustment input data to be subjected to the conversion process together with the input attitude, the conversion unit has fixed parameters determined by a learning process; The adjustment unit has a variable parameter that can be additionally learned to generate the adjustment input data after the trained model is generated by a learning process. The learning device according to any one of claims 1 to 5.

7. the adjustment unit additionally learns the variable parameter for each individual user based on the input posture of the user and the output posture of the operation target. The learning device according to claim 6.

8. the adjustment unit additionally learns the variable parameter for each individual operation object based on the output posture of the operation object and the input posture of the user. The learning device according to claim 6.

9. The trained model includes an output layer that outputs the output posture converted based on the input posture, and is capable of switching between a plurality of the output layers according to a plurality of types of the operation objects having different structures. The learning device according to any one of claims 1 to 5.

10. The input motion data includes hand motion data which is a hand movement of the user. The learning device according to any one of claims 1 to 5.

11. the input posture includes position information of a plurality of first joints which are joints of the user; the output attitude includes position information of a plurality of second joints which are joints of the operation target, the learning processing unit performs a learning process such that the first joints included in the input posture and the second joints included in the output posture corresponding to the input posture have a non-one-to-one correspondence. The learning device according to claim 2 .

12. 1. An operation information conversion device that converts an input posture of a user into an output posture of an operation object having a structure different from a structure of a posture input part of the user, an input unit that acquires the input posture of the user; an operation information generation unit that uses pair data of the input posture and the output posture at corresponding time points among input motion data that is a time series change of the input posture and output motion data that is a time series change of the output posture that represents an operation corresponding to the input motion data, and generates the output posture from the input posture input to the input unit using a trained model that has learned a process of inputting an input posture and outputting a corresponding output posture; and an output unit that outputs the generated output attitude. Operation information conversion device.

13. A learning method for machine learning a process of converting an input posture of a user into an output posture of an operation object having a structure different from a structure of a posture input part of the user, the method comprising: acquiring teacher data including input motion data that is a time series change of the input posture, and output motion data that is a time series change of the output posture that represents a movement corresponding to the input motion data; and performing a learning process using pair data of the input posture and the output posture at corresponding time points among the input motion data and the output motion data, to generate a learned model in which the input posture is used as an input and the corresponding output posture is used as an output. How to learn.

Images