Method and control device for generating stylized motion of an object such as a robot or virtual avatar

Abstract

To provide a method for controlling motion of an object such as a robot or a virtual avatar.SOLUTION: A method includes: a step for receiving input motion information describing present motion of a present motion type and a present style, and a target style for target motion of the present motion type and a target style; a step for converting the input motion information into output motion information on the basis of the target style by using a first feed-forward neural network (FFNN) of a style statistic predictor (1), the output motion information describing the target motion; and a step for controlling motion of an object on the basis of the output motion information such that a motion type of motion of the object corresponds to the present motion type and that a style of motion of the object corresponds to the target style. Further, the present invention relates to a control device configured so as to perform a method by the method of the present invention to control motion of the object, and a robot including such a control device and respective computer programs.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a method and a controller for controlling the movement of an object, such as a robot or a virtual avatar, the controller being configured to perform the method. The present disclosure further relates to a robot comprising a controller configured to control the movement of the robot, and a computer program comprising program code for performing such a method when implemented on a processor. [Background technology]

[0002] Given a current motion of an object, such as a robot or virtual avatar, the styling method can be used to change the style of the current motion by converting the style of the current motion from the current style to a goal style. For example, a neutral walking motion (i.e., the motion type of the current motion is "walking" and the current style is "neutral") can be converted to a fun walking motion (i.e., the goal style of the current motion, which is motion type "walking", is "fun"). Converting the style of the current motion from the current style to a goal style can also be expressed as styling the current motion to the goal style.

[0003] Current state-of-the-art styling approaches have various disadvantages. The motion style of the current motion of an object, such as a robot or virtual avatar, to be styled into a target style must be very similar to the motion type of the data used as the styling reference to produce meaningful styling results. Furthermore, a user who wants to style the current motion of an object must provide examples of various styles for the motion type of the current motion to be styled. For example, to style the current motion of the motion type "walking," the user needs to provide examples of the current motion type in various styles, i.e., examples of walking motion in various walking styles. The current motion can then be styled into any of these various styles.

[0004] Furthermore, current state-of-the-art styling techniques are typically applied only to virtual avatars, and therefore cannot be directly used to control robot motion, since the robot's physical constraints must be taken into account when generating its motion. Summary of the Invention [Problem to be solved by the invention]

[0005] It is therefore an object of the present invention to provide a method for controlling the movement of an object, such as a robot or a virtual avatar, that overcomes at least one of the above-mentioned disadvantages. [Means for solving the problem]

[0006] These and other objects that will become apparent on reading the following description are solved by the subject matter of the independent claims. The dependent claims refer to advantageous implementations of embodiments of the invention.

[0007] A first aspect of the present disclosure provides a method for controlling the movement of an object, such as a robot or a virtual avatar, the method comprising: - receiving input motion information describing a current motion of a current motion type and a current style, and receiving a target style for a target motion of the current motion type and a target style; - converting input motion information into output motion information based on a target style using a first feedforward neural network (FFNN) of a style statistics predictor, the output motion information describing the target motion; - controlling the motion of the object based on the output motion information such that the motion type of the motion of the object corresponds to the current motion type and the style of the motion of the object corresponds to the target style.

[0008] In other words, the first aspect of the present disclosure proposes a method for using a first FFNN of a style statistics predictor to enable the current style of a current movement of an object, such as a robot or a virtual avatar, to be transformed into a target style without changing the movement type of the current movement.

[0009] For example, if the current motion type is "walking" and thus the current motion of the object corresponds to the object walking, the current style "happy" can be changed to a target style "sad." If the object is a robot, the robot's motion according to a motion type of, for example, "walking" and a style of, for example, "happy" means that the robot walks happily. In other words, the robot walks in a way that makes the robot look happy. If the robot's motion style is changed to a target style of, for example, "sad" without changing the motion type "walking," the robot adapts its walking style from walking happily to walking sadly. In other words, the robot walks in a way that makes the robot look sad. The same applies to objects that are virtual avatars.

[0010] The use of the first FFNN in the style statistics predictor overcomes the above-described disadvantage that the motion type of the current motion of the object to be styled in the target style must be very similar to the motion type of the data used as the stylization reference. Furthermore, the above-described disadvantage that the user must provide all different styles (including the target style) for the motion type of the current motion is overcome. That is, the first FFNN in the style statistics predictor can predict how input motion information describing the current motion type and the current style should be converted into output motion information describing the current motion type, which is the target style. The first FFNN does not need to know the current style of the input motion information. Furthermore, the current motion type and current style of the input motion information do not need to be similar to the training data used to train the first FFNN. Thus, the method according to the first aspect is advantageous because it can convert the current style of a current motion of any motion type into a different style without having examples of motion information of the same motion type in different styles.

[0011] Therefore, the method according to the first aspect has several advantages:

[0012] The motion repertoire of a robot or virtual avatar may be expanded. That is, the number of different styles in which a robot's or virtual avatar's motion can be styled may be expanded. That is, the user does not need to provide different style examples for each motion type. This leads to more expressive robots and virtual avatars. As a result, a robot or virtual avatar may be able to convey its intentions or information to a person by changing the style of its current motion. For example, changing the current motion from a current style of "neutral" to a style of "angry" may be used to communicate danger. Furthermore, the robot or virtual avatar may be easily personalized by the user, leading to an improved user experience. Furthermore, the robot or virtual avatar may reflect the emotional state of a person (human) to make the person feel more comfortable or the robot, respectively, the virtual avatar, appear more caring.

[0013] As a result, engineers do not need to create different styles of motion information for different motion types in a time-consuming process. In contrast, engineers only need to create motion information for a certain motion type (e.g., a single hand-waving motion, a single walking motion, ...) once. The method of the first aspect can be used to style motion information. That is, the method of the first aspect can be used to convert motion information of a certain motion type into output motion information of the same motion type and target style.

[0014] Furthermore, the method of the first aspect enables the current style of a robot or virtual avatar's current motion, which corresponds to the current motion type and current style, to be automatically changed to a different target style. In other words, input motion information describing the current motion of the current motion type and current style can be automatically converted into output motion information by using the first FFNN of the style statistics predictor based on the target style, and the output motion information describes motion of the current motion type and target style (target motion). Thus, the method of the first aspect enables the automatic conversion of the style of motion to a target style. The automatically converted output motion information can be used for motion editing. That is, if the motion of a robot or virtual avatar controlled based on the output motion information does not match an engineer's expectations regarding the motion type and / or target style, the output motion information can serve as a starting point for manually editing the motion.

[0015] Therefore, the method of the first aspect can be used not only for controlling the movement of a robot or virtual avatar, but also as a design tool during the development process of a robot or virtual avatar. In particular, the method of the first aspect can be used to control the movement of a robot or virtual avatar so that the robot or virtual avatar appears more human-like.

[0016] In particular, the method for controlling the object's movement can be referred to as a styling method for styling the object's current movement to a target style.

[0017] The robot may be a humanoid robot. The virtual avatar may be a humanoid virtual avatar. The virtual avatar is sometimes called an "animated character."

[0018] The style statistics predictor and the first FFNN may each be implemented by hardware and / or software.

[0019] The terms "movement" and "behavior" are sometimes used synonymously with the term "exercise." The terms "exercise category" and "exercise content" are sometimes used synonymously with the term "exercise type." The term "exercise type" is sometimes abbreviated to the term "type." The term "style" is sometimes referred to as "exercise style."

[0020] Examples of movement types for a robot or virtual avatar, such as a humanoid robot or virtual avatar, include walking, dancing, running, jumping, making facial expressions, gesturing, and the like.

[0021] Examples of movement styles for a robot or virtual avatar include happy, sad, angry, old, young, etc.

[0022] Movement style may also be referred to as an emotional state, such as sad, happy, or angry, or a human characteristic, such as young or old, that is evident to a person when they exercise according to that style.

[0023] The input motion information may comprise or correspond to data, specifically xyz positions, each xyz position corresponding to each degree of freedom of the object being controlled. An xyz position is defined by its xyz coordinates. Thus, an xyz position may also be referred to as a respective xyz coordinate. The terms "three-dimensional position" and "position in three-dimensional space" are sometimes used synonymously with the term "xyz position."

[0024] According to one embodiment, the input motion information is a time series matrix X 1:N The data are xyz positions corresponding to the degrees of freedom of the object to be controlled, and the matrix X has dimensions (k×3), where k is the number of degrees of freedom of the object. That is, for each time point t=1 to N, the matrix X 1:N comprises xyz positions corresponding to the degrees of freedom of the robot or virtual avatar.

[0025] The degrees of freedom of the object are itself controllable to generate object motion according to a motion type and motion style; - Parts of the object, and / or - Possible elementary movements of one or more parts of an object It may be.

[0026] Therefore, the degrees of freedom of a robot or virtual avatar are: itself capable of being controlled to generate a movement of the robot or virtual avatar according to a movement type and a movement style; - Part of a robot or virtual avatar (e.g., a robot arm with six joints as six degrees of freedom, each joint being part of the body of the robot or virtual avatar), and / or - Possible basic movements of one or more parts of the robot or virtual avatar (e.g., a robot with five degrees of freedom: eye tilt, eye roll, eye pop, body tilt forward and backward, and basic rotations) It may be.

[0027] The first type, FFNN, is sometimes called a "deep neural network."

[0028] The first FFNN is configured to convert input motion information of any motion type and any style into output motion information of the same motion type and a target style among a plurality of different styles, without having examples of the same motion type in different styles. Thus, the first FFNN is configured to convert input motion information of any motion type and any style into output motion information of the same motion type and a different style corresponding to the target style.

[0029] According to one implementation of the first aspect, the first FFNN is a probabilistic neural network (PNN). Such a probabilistic neural network may also be referred to as a "deep probabilistic neural network."

[0030] According to one implementation of the first aspect, a method includes: - generating input statistics based on the input exercise information using a first exercise statistics module, the input statistics indicating a current exercise type and a current style; - generating target statistics based on the input statistics and the target style using a first FFNN, the target statistics indicating a current motion type and a target style; - converting the input athletic information into output athletic information based on the input statistics and the target statistics.

[0031] In other words, the first FFNN of the style statistics predictor may be configured to map the input statistics generated by the first motion statistics module based on the input motion information and the target style to the target statistics, and the input motion information can be converted into output motion information based on the input statistics and the target statistics.

[0032] The first athletic statistics module may be implemented by hardware and / or software.

[0033] The input statistics are statistics about the current exercise, and therefore about the current exercise type and the current style.

[0034] The input statistics may be the mean and / or variance of the data of the input motion information. Alternatively or additionally, the input statistics may be defined as a Gram matrix of the data. If the data are xyz positions corresponding to degrees of freedom of the object to be controlled, the input statistics may be the mean and / or variance of the xyz positions and / or the input statistics may be defined as a Gram matrix of the xyz positions.

[0035] According to one embodiment, the input motion information is a time series of data X of length N as described above. 1:N If the input statistics correspond to or comprise:

number

[0036] As already outlined above, the matrix X is interpreted as a vector of length k×3, where k is the number of degrees of freedom of the object.

[0037] Target statistics are statistics about the target movement, and thus about the current movement type and target style. The target style may be defined by the statistical properties of the movement observed according to the target style.

[0038] The target statistics may be the mean and / or variance of the data and describe a target motion that has the same motion type (current motion type) as the input motion information and a style that corresponds to the target style. If the data are xyz positions corresponding to the degrees of freedom of the object to be controlled, the target statistics may be the mean and / or variance of the xyz positions.

[0039] According to one embodiment, the target statistic may be the average of the cross products of the xyz coordinates of each xyz position for all degrees of freedom over the time domain (average similarity of the xyz positions).

[0040] If the first FFNN is a PPN, the target statistics generated by the PPN may be a Gaussian probability distribution with a mean and variance.

[0041] If the first FFNN is a PNN, the target statistics may correspond to a range distribution, such as a distribution of means and / or variances of xyz positions corresponding to degrees of freedom, where the distribution describes the probability that the movement indicated by the target statistics corresponds to the target movement depending on the current movement type and target style.

[0042] According to one implementation of the first aspect, a method includes: - using a first FFNN to predict, based on the input statistics and the target style, how one or more statistics, such as the mean and / or variance of the input statistics, must be changed to transform the input statistics into the target statistics; - using the first FFNN to transform the input statistics into target statistics based on the results of the prediction.

[0043] According to one implementation of the first aspect, a method includes: - generating a motion primitive (MP) according to the target style based on the target statistics using a second feedforward neural network (FFNN) of the motion primitive (MP) generator; - controlling the movement of the object based on the MP according to the target style.

[0044] The second FFNN and the MP generator may each be implemented by hardware and / or software.

[0045] The first athletic statistics module, the style statistics predictor including the first FFNN, and the MP generator including the second FFNN may constitute an athletic style setting framework system configured to change a current MP (according to the current style) to a modified MP according to the target style.

[0046] A movement primitive (MP) is sometimes referred to as a basic elemental movement shape. Thus, an object's MP represents all possible individual movements that can be performed by the object for one particular style (e.g., "fun"). Multiple MPs according to a particular style can create the object's movement, and the style of the movement is defined by the particular style of the MPs. The style of the movement may additionally be defined by which MPs are enabled, i.e., which MPs are performed. In other words, an object's movement, such as "walking," is created by activating each MP, and thus each individual movement that forms the walking movement.

[0047] The term "motion primitive" is sometimes used as a synonym for the term "motion primitive."

[0048] The MPs may be stored as data in a matrix G of size r×d, where r is the number of MPs and d is the dimension of the MP. For example, if the data describes one xyz position per degree of freedom, the dimension d of the MP is 3, or if the data describes a quaternion that represents the orientation of a degree of freedom relative to its predecessor, the dimension d of the MP is 4.

[0049] Different representations are possible for an MP. For example, an MP may generate one or more xyz positions corresponding to one or more degrees of freedom. Alternatively or additionally, an MP may generate one or more Euler angles for the orientation of one or more degrees of freedom. Alternatively or additionally, an MP may generate one or more quaternions for the orientation of one or more degrees of freedom. Alternatively or additionally, an MP may generate one or more joint angles for one or more degrees of freedom. The representation of an MP may be defined based on the control variables of a given object, such as a robot or virtual avatar. Examples of control variables for a given object are one or more Euler angles, quaternions, joint angles, and / or xyz positions of the object.

[0050] For example, if a robot or virtual avatar is controlled via joint angles (i.e., the control variables of the robot are joint angles), then MP is used to describe the trajectory of the joint angles of the robot, respectively the virtual avatar.

[0051] The second FFNN may map target statistics to parameters of a predetermined number of MPs. For fixed input motion information, the MP generator, specifically the second FFNN, may generate different MPs for different target statistics, and therefore different target styles. The MP generator may correspond to a motion manifold (all possible motions that the object can perform).

[0052] According to one implementation of the first aspect, a method includes: - using a second FFNN to generate an MP according to the target style based on the input statistics and the target statistics.

[0053] The second FFNN may determine a current exercise type of the current exercise to be styled to the target style based on the input statistics and the target statistics. The second FFNN may determine a target style based on the target statistics. The second FFNN may generate an MP according to the target style based on the determination result.

[0054] The second FFNN may map the input statistics and target statistics to parameters of a predetermined number of MPs.

[0055] According to one implementation of the first aspect, a method includes: - changing the MP according to the current style to an MP according to the target style based on the target statistics, in particular based on the target statistics and the input statistics, using the second FFNN.

[0056] That is, the second FFNN may modify a current MP according to a current style, which may be used to generate a motion of a current style (such as a current motion of a current motion type and a current style), to a modified MP according to a target style based on the target statistics. The modified MP may be used to generate a motion of a target style, such as a target motion of a current motion type and a target style. The target statistics indicate the target style.

[0057] For example, the current MP is an MP according to the current style "young," i.e., the current MP can be used to create the motion of an object of the style "young," so that the motion of the object appears to be performed by a young person. If the target style is "old," the second FFNN can change the current MP according to the current style "young" to a modified MP according to the target style "old." The modified MP can be used to create the motion of an object of the style "old," so that the motion of the object appears to be performed by an old person.

[0058] For example, an MP according to the style "young" can generate a motion of the motion type "walking" such that the humanoid robot or virtual avatar walks with, for example, a straight back and stable shoulders. As a result of the straight back and stable shoulders, the walking humanoid robot or virtual avatar appears young to a person watching the gait. In contrast, an MP according to the style "old" can generate a motion of the motion type "walking" such that the humanoid robot or virtual avatar walks with, for example, a hunched back and slumped shoulders. As a result of the hunched back and slumped shoulders, the walking humanoid robot or virtual avatar appears old to a person watching the gait.

[0059] According to one embodiment, the second FFNN may change the current MP according to the current style to a changed MP according to the target style based on the target statistics and the input statistics.

[0060] According to one implementation of the first aspect, a method includes: - using a motion generator to generate output motion information based on the MP and MP activation according to the target style.

[0061] The motion generator may be implemented by software and / or hardware.

[0062] The MP activation defines which of the possible individual motions represented by the MPs should be activated, and thus defines the object motions produced by the individual motions represented by the MPs.

[0063] Thus, the MP activation may define which of the possible individual movements represented by the MP should be activated so that the output movement information describes the target movement of the current movement type and target style.

[0064] In particular, the MP validation may define the motion type of the motion described by the output motion information.

[0065] Given the MPs and the time series of MP activations generated by the MP generator, the motion generator may generate output motion information of the same dimensionality as the input motion information.

[0066] According to a first embodiment of the first aspect, a method comprises: - generating a loss value based on the input motion information and the output motion information using a loss function module, the loss value indicating a difference between a motion type of the output motion information and a motion type of the input motion information; - optimizing the MP activation based on the loss value using an optimizer.

[0067] In particular, the loss function module may determine, based on the input motion information and the output motion information, a difference between a current motion type described by the input motion information and a motion type described by the output motion information.

[0068] The loss function module and the optimizer may be implemented by software and / or hardware.

[0069] In particular, the optimizer uses one or more optimization algorithms, such as a gradient descent algorithm, to generate optimized MP activations based on the loss values. The optimizer may be an Adam optimizer.

[0070] According to a second embodiment of the first aspect, a method comprises: - generating a loss value based on the input motion information, the output motion information, and the target style using a loss function module, wherein the loss value indicates a difference between the motion type of the output motion information and the motion type of the input motion information, and a difference between the style of the output motion information and the target style; - optimizing the MP activation based on the loss value using an optimizer.

[0071] In particular, the loss function module may determine, based on the input motion information and the output motion information, a difference between a current motion type described by the input motion information and a motion type described by the output motion information. The loss function module may determine a difference between a style described by the output motion information and a target style.

[0072] In particular, the optimizer uses one or more optimization algorithms, such as a gradient descent algorithm, to generate optimized MP activations based on the loss values. The optimizer may be an Adam optimizer.

[0073] According to a third embodiment of the first aspect, when the target statistics are generated using the first FFNN based on the input statistics and the target style, the method includes: - generating output statistics based on the output athletic information using a second athletic statistics module; - generating a loss value based on the input motion information, the output motion information, the target statistics, and the output statistics using a loss function module, wherein the loss value indicates a difference between the motion type of the output motion information and the motion type of the input motion information, and a difference between the style of the output motion information and the target style; - optimizing the MP activation based on the loss value using an optimizer.

[0074] In particular, the loss function module may determine, based on the input motion information and the output motion information, a difference between a current motion type described by the input motion information and a motion type described by the output motion information. The loss function module may determine, based on the target statistics and the output statistics, a difference between a style described by the output motion information and a target style.

[0075] In particular, the optimizer uses one or more optimization algorithms, such as a gradient descent algorithm, to generate optimized MP activations based on the loss values. The optimizer may be an Adam optimizer.

[0076] According to one implementation of the first aspect, - generating output motion information based on MP and MP activation according to a target style, as well as the steps of the first embodiment of the first aspect described above; or - generating output motion information based on MP and MP activation according to the target style, as well as the steps of the second embodiment of the first aspect described above; or - generating output motion information based on MP and MP activation according to the target style, as well as the steps of the third embodiment of the first aspect described above, Iterations are performed until the loss value is less than a threshold, and in the first iteration, output motion information is generated based on MPs and any MP activations according to the target style.

[0077] The above steps are repeated until the loss value is smaller than the threshold, so that the output motion information generated in the final iteration is close to the input motion information in terms of motion content, i.e., the motion type of the motion described by the output motion information (generated in the final iteration) corresponds to the motion type of the motion described by the input motion information. The output motion information generated in the final iteration is close to the target style in terms of style, i.e., the style of the motion described by the output motion information (generated in the final iteration) corresponds to the target style.

[0078] The loss function module may reward similarity of the motion type of the output motion information with the motion type of the input motion information. The loss function module may additionally penalize deviation of the style of the output motion from the target style.

[0079] According to one implementation of the first aspect, a method includes: - generating target motion information based on the MP and any MP activation according to the target style using a motion generator; - using an inverse and / or forward kinematics module to generate output motion information based on the target motion information and physical constraints of the object, such as one or more joint angles and one or more bone lengths of the object.

[0080] The desired motion information describes a desired motion of a current motion type and a target style. The output motion information generated based on the desired motion information describes a desired motion of a current motion type and a target style while taking into account physical constraints of the object.

[0081] As a result of using the inverse and / or forward kinematics modules, the method according to the first aspect can be directly used to control the motion of an object (robot or virtual avatar), since the physical constraints of the object are taken into account by the inverse and / or forward kinematics modules when generating the object's motion. That is, the inverse and / or forward kinematics modules make it possible to generate physically realizable, respectively kinematically aware, output motion information of the object.

[0082] The inverse and / or forward kinematics modules enable the satisfaction of physical constraints of the object, such as one or more joint angles, particularly joint constraints, and one or more bone lengths of the object, when generating output kinematic information.

[0083] The inverse kinematics module can determine the joint angles of the object to achieve the desired motion, and the forward kinematics module can determine the positions and orientations of various points from the joint angles of the object.

[0084] According to one implementation of the first aspect, the motion of the object is controlled by controlling one or more control variables of the object, such as one or more Euler angles, quaternions, joint angles and / or xyz positions.

[0085] According to one implementation of the first aspect, when one or more xyz positions are controlled as control variables, an inverse kinematics module is used to generate output motion information, and when one or more Euler angles, quaternions, and / or joint angles are controlled as control variables, a forward kinematics module is used to generate output motion information.

[0086] The forward kinematics module may incorporate one or more joint constraints when using one or more joint angles as control variables, and the inverse kinematics module may ensure that physical constraints, such as bone length constraints of an object, are satisfied when using one or more xyz positions as the MP representation.

[0087] The inverse kinematics module may be a forward and backward reaching inverse kinematics (FABRIK) module.

[0088] Some or all of the above implementations and optional features may be combined with each other to realize the method according to the first aspect.

[0089] A second aspect of the present disclosure provides a control device for controlling the movement of an object, such as a robot or a virtual avatar, the control device being configured to perform a method according to the first aspect or any of its implementations described above.

[0090] The control device includes a style statistics predictor having a first feedforward neural network (FFNN), the first FFNN configured to convert input motion information describing a current motion of a current motion type and a current style into output motion information based on a target style, the output motion information describing a target motion of the current motion type and a target style, and the control device configured to control the motion of the object based on the output motion information such that the motion type of the motion of the object corresponds to the current motion type and the style of the motion of the object corresponds to the target style.

[0091] A controller may correspond to or comprise a processor, microprocessor, controller, microcontroller, application specific integrated circuit (ASIC), or any combination of these elements.

[0092] The controller may control the degrees of freedom of the object. For example, if the object corresponds to a robot with an arm having six different joint angles, the six joint angles are six degrees of freedom and may be controlled by the controller. In particular, the controller may control actuators of the object that govern the movement of the object.

[0093] The first FFNN may be a probabilistic neural network (PNN).

[0094] According to one implementation of the second aspect, the control device includes a first motion statistics module configured to generate input statistics based on input motion information, the input statistics indicating a current motion type and a current style. The first FFNN is configured to generate target statistics based on the input statistics and the target style, the target statistics indicating the current motion type and the target style. The control device is configured to convert the input motion information into output motion information based on the input statistics and the target statistics.

[0095] According to one implementation form of the second aspect, the first FFNN is configured to predict how one or more statistical values, such as the mean and / or variance of the input statistics, should be changed based on the input statistics and the target style in order to convert the input statistics into the target statistics, and to convert the input statistics into the target statistics based on the results of the prediction.

[0096] According to one implementation form of the second aspect, the control device includes a motion primitive (MP) generator having a second feedforward neural network (FFNN), the second FFNN is configured to generate a motion primitive (MP) according to a goal style based on the goal statistics, and the control device is configured to control the motion of the object based on the MP according to the goal style.

[0097] According to one implementation form of the second aspect, the second FFNN is configured to generate an MP according to a target style based on the input statistics and the target statistics.

[0098] According to one implementation form of the second aspect, the second FFNN is configured to change the MP according to the current style to an MP according to the target style based on the target statistics, specifically based on the target statistics and the input statistics.

[0099] According to one implementation of the second aspect, the control device includes a motion generator configured to generate output motion information based on an MP and MP activation according to a target style.

[0100] According to a first embodiment of the second aspect, the control device includes a loss function module configured to generate a loss value based on the input motion information and the output motion information, the loss value indicating a difference between a motion type of the output motion information and a motion type of the input motion information, and an optimizer configured to optimize MP activation based on the loss value.

[0101] According to a second embodiment of the second aspect, the control device includes a loss function module configured to generate a loss value based on the input motion information, the output motion information, and a target style, the loss value indicating a difference between the motion type of the output motion information and the motion type of the input motion information, and a difference between the style of the output motion information and the target style. In addition, the control device includes an optimizer configured to optimize MP activation based on the loss value.

[0102] According to a third embodiment of the second aspect, the control device includes a second motion statistics module configured to generate output statistics based on the output motion information, and a loss function module configured to generate a loss value based on the input motion information, the output motion information, the target statistics, and the output statistics, the loss value indicating a difference between the motion type of the output motion information and the motion type of the input motion information, and a difference between the style of the output motion information and the target style. In addition, the control device includes an optimizer configured to optimize MP activation based on the loss value.

[0103] According to one implementation form of the second aspect, the control device generating output motion information by a motion generator, generating loss values ​​by a loss function module according to the first embodiment of the second aspect, and optimizing the MP activation by an optimizer according to the first embodiment of the second aspect; or generating output motion information by a motion generator, generating loss values ​​by a loss function module according to a second embodiment of the second aspect, and optimizing the MP activation by an optimizer according to a second embodiment of the second aspect; or generating output motion information by a motion generator, generating loss values ​​by a loss function module according to the third embodiment of the second aspect, and optimizing MP activation by an optimizer according to the third embodiment of the second aspect. is configured to iterate until the loss value is smaller than a threshold, and in the first iteration, output motion information is generated based on the MP according to the target style and any MP activation.

[0104] According to one implementation form of the second aspect, the motion generator is configured to generate target motion information based on MP and MP activation according to the target style, and the control device includes an inverse and / or forward kinematics module configured to generate output motion information based on physical constraints of the object, such as one or more joint angles and one or more bone lengths of the object, and the target motion information.

[0105] According to one implementation of the second aspect, the control device is configured to control the movement of the object by controlling one or more control variables of the object, such as one or more Euler angles, quaternions, joint angles and / or xyz positions.

[0106] According to one implementation of the second aspect, when one or more xyz positions are controlled as control variables, an inverse kinematics module is used to generate output motion information, and when one or more Euler angles, quaternions, and / or joint angles are controlled as control variables, a forward kinematics module is used to generate output motion information.

[0107] The first FFNN, style statistics predictor, first motion statistics module, second FFNN, MP generator, motion generator, loss function module, optimizer, second motion statistics module, and inverse and / or forward kinematics module may each be implemented by hardware and / or software.

[0108] The above description of the method according to the first aspect and its implementations is also valid for the control device according to the second aspect and its implementations.

[0109] Some or all of the above embodiments and optional features may be combined with each other to realize a control device according to the second aspect.

[0110] A third aspect of the present disclosure provides a robot comprising a control device according to the second aspect or any of its implementations, as described above.

[0111] The robot may be a humanoid robot. The robot may include one or more actuators for effecting movement according to one or more degrees of freedom of the robot. The controller is configured to control the actuators to create and control the movement of the robot.

[0112] The above description regarding the method of the first aspect and its implementations, and the above description regarding the control device according to the second aspect and its implementations, are also valid for the robot according to the third aspect.

[0113] Some or all of the above optional features may be combined with each other to realize a robot according to the third aspect.

[0114] A fourth aspect of the present disclosure provides a computer program comprising a program code for performing the method according to the first aspect or any of its implementations.

[0115] In particular, a fourth aspect of the present disclosure provides a computer program comprising a program code for, when executed on a processor, performing a method according to the first aspect or any of its implementations.

[0116] A fifth aspect of the present disclosure provides a computer program product comprising program code for, when executed on a processor, performing a method according to the first aspect or any of its implementations.

[0117] A sixth aspect of the present disclosure provides a non-transitory storage medium having stored thereon executable program code which, when executed by a processor, causes a method according to the first aspect or any of its implementations to be performed.

[0118] A seventh aspect of the present disclosure provides a computer comprising a memory and a processor configured to store and execute program code for performing a method according to the first aspect or any of its implementations.

[0119] The invention will now be described, by way of example only, with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0120] [Figure 1]FIG. 1 is an exemplary diagram illustrating one embodiment of a portion of a control device for controlling the movement of an object, such as a robot or virtual avatar. [Figure 2] FIG. 1 is an exemplary diagram illustrating one embodiment of a portion of a control device for controlling the movement of an object, such as a robot or virtual avatar. [Figure 3] FIG. 1 is an exemplary diagram illustrating one embodiment of a portion of a control device for controlling the movement of an object, such as a robot or virtual avatar. [Figure 4] FIG. 1 is an exemplary diagram illustrating one embodiment of a portion of a control device for controlling the movement of an object, such as a robot or virtual avatar. [Figure 5] FIG. 1 exemplarily illustrates a flow diagram of an embodiment of a method for training a first feed-forward neural network (FFNN) of a style statistics predictor used by a method according to the first aspect or any of its implementations; [Figure 6] FIG. 1 exemplarily illustrates a flow diagram of one embodiment of functionality of a first feedforward neural network (FFNN) of a style statistics predictor after training, the style statistics predictor being used by a method according to the first aspect or any of its implementations. [Figure 7] FIG. 2 exemplarily illustrates a flow diagram of an embodiment of a method for training a second feed-forward neural network (FFNN) of a motion primitive (MP) generator used by an embodiment of the method according to the first aspect; [Figure 8] FIG. 10 is an exemplary flowchart of an embodiment of the functionality of a second feedforward neural network (FFNN) of a trained motion primitive (MP) generator, which is used by an implementation of the method according to the first aspect. [Figure 9] 3 exemplarily shows a flow diagram of method steps of an optimization process according to an embodiment of an implementation of the method according to the first aspect; DETAILED DESCRIPTION OF THE INVENTION

[0121] In the figures, corresponding elements are designated by the same reference numerals.

[0122] FIG. 1 illustrates an exemplary embodiment of a portion of a control device for controlling the movement of an object, such as a robot or virtual avatar.

[0123] As shown in FIG. 1, a control device for controlling the movement of an object such as a robot or a virtual avatar may include a style statistics predictor 1 and a movement statistics module 2 .

[0124] The above description of the method according to the first aspect and its implementation relating to the style statistics predictor and the first motion statistics module, and the above description of the control device according to the second aspect and its implementation relating to the style statistics predictor and the first motion statistics module, are correspondingly valid for the style statistics predictor 1 and motion statistics module 2 in Fig. 1. Motion statistics module 2 may also be referred to as the first motion statistics module.

[0125] 1, the exercise statistics module 2 is configured to receive input exercise information describing a current exercise type and a current style, and generate input statistics based on the input exercise information. The input statistics indicate the current exercise type and the current style.

[0126] The style statistics predictor 1 comprises a feed-forward neural network (FFNN), sometimes referred to as a first FFNN (not shown in FIG. 1). The first FFNN may be a probabilistic neural network (PNN).

[0127] The first FFNN is configured to receive input motion information describing a current motion of a current motion type and a current style, and to receive a target style for a target motion of the current motion type and a target style. According to Figure 1, the first FFNN of the style statistics predictor 1 receives the input motion information in the form of input statistics.

[0128] The first FFNN of the style statistics predictor 1 is configured to generate target statistics based on the input statistics and the target style, where the target statistics indicate the current motion type and the target style, so that the target statistics enable generating output motion information describing the target motion of the current motion type and the target style.

[0129] Therefore, the first FFNN of the style statistics predictor 1 allows the current movement of the robot or virtual avatar (current movement is the current movement type and the current style) to be styled to the target style without changing the current movement type.

[0130] With reference to Fig. 5, an embodiment of a method for training the first FFNN of the style statistics predictor 1 is illustratively described. With reference to Fig. 6, an embodiment of the functionality of the first FFNN of the style statistics predictor 1 after training is illustratively described.

[0131] FIG. 2 illustrates an exemplary embodiment of a portion of a control device that controls the movement of an object, such as a robot or virtual avatar.

[0132] The part of the control device according to Fig. 2 corresponds to the part of the control device shown in Fig. 1. Therefore, the description of Fig. 1 above is also valid for the part of the control device of Fig. 2, and only the differences will be explained below. As shown in Fig. 2, the control device comprises a movement primitive (MP) generator 3, in addition to a movement statistics module 2 and a style statistics predictor 1 comprising a first FFNN (not shown).

[0133] The above description of the MP generator in the first aspect and the method according to its implementation, and the above description of the MP generator in the second aspect and the control device according to its implementation, are correspondingly valid for the MP generator 3 of FIG. 2.

[0134] The MP generator 3 includes a feed-forward neural network (FFNN), which may be referred to as a second FFNN (not shown in FIG. 2). The second FFNN is configured to generate a movement primitive (MP) according to a target style based on the target statistics. Preferably, the second FFNN may be configured to generate an MP according to a target style based on the target statistics and input statistics, as shown in FIG. 2.

[0135] With reference to Fig. 7, an embodiment of a method for training the second FFNN of the MP generator 3 is exemplarily described. With reference to Fig. 8, an embodiment of the functionality of the second FFNN of the MP generator 3 after training is exemplarily described.

[0136] FIG. 3 illustrates an exemplary embodiment of a portion of a control device that controls the movement of an object, such as a robot or virtual avatar.

[0137] As shown in FIG. 3, a controller for controlling the motion of an object such as a robot or virtual avatar may include a motion generator 4 and an inverse and / or forward kinematics module 8 .

[0138] The above description of the method according to the first aspect and its implementation forms relating to the motion generator and the inverse and / or forward kinematics module, and the above description of the control device according to the second aspect and its implementation forms relating to the motion generator and the inverse and / or forward kinematics module, are correspondingly valid for the motion generator 4 and the inverse and / or forward kinematics module 8.

[0139] The motion generator 4 is configured to generate output motion information based on motion primitives (MPs) and MP activations. Specifically, the motion generator 4 is configured to generate desired motion information based on the MPs and MP activations. The inverse and / or forward kinematics module 8 is configured to generate output motion information based on the desired motion information and physical constraints of the object, such as one or more joint angles and one or more bone lengths of the object.

[0140] The desired motion information may describe a desired motion of a current motion type and a target style, and the output motion information generated based on the desired motion information may describe the desired motion of a current motion type and a target style while taking into account physical constraints of the object.

[0141] The MP input to the motion generator 4 to generate the output motion information may correspond to the MP (modified MP) generated by the MP generator 3 shown in FIG. 2. The style of the motion described by the output motion information may be set by the MP generated by an MP generator such as the MP generator 3 shown in FIG. 2 based on target statistics indicating the target style. The motion type of the motion described by the output motion information may be set by MP validation. The MP validation may be generated by an optimization process so that the output motion information describes the target motion, i.e., a motion of the current motion type and target style (of the input motion information). Elements required for such an optimization process are shown in FIG. 4.

[0142] One embodiment of an optimization process that may be performed to generate an MP validation that provides a target motion for the current motion type and target style is illustratively described with respect to FIG.

[0143] FIG. 4 illustrates an exemplary embodiment of a portion of a control device for controlling the movement of an object, such as a robot or virtual avatar.

[0144] The portion of the control device shown in Figure 4 includes the portions of Figures 2 and 3, with the MP generator 3 of Figure 2 providing an MP to the motion generator 4 of Figure 3. Therefore, the above description of Figures 1-3 is also valid for the portion of the control device of Figure 4, and only the differences will be explained below. As shown in Figure 4, in addition to the elements of Figures 2 and 3, the control device includes a motion statistics module 7, sometimes called a second motion statistics module, a loss function module 5, and an optimizer 6.

[0145] The above description of the method according to the first aspect and its implementation form relating to the second motion statistics module, the loss function module and the optimizer, and the above description of the control device according to the second aspect and its implementation form relating to the second motion statistics module, the loss function module and the optimizer, are correspondingly valid for the second motion statistics module 7, the loss function module 5 and the optimizer 6.

[0146] The second motion statistics module 7 is configured to generate output statistics based on the output motion information generated by the inverse and / or forward kinematics module 8. The inverse and / or forward kinematics module 8 is an optional element. If the inverse and / or forward kinematics module 8 is not present, the target motion information generated by the motion generator 4 is input to the second statistics module 7 as the output motion information.

[0147] In either case, according to FIG. 4, the output motion information, output statistics, input motion information, and target statistics are input to a loss function module 5.

[0148] The loss function module 5 may be configured to generate a loss value based on the input motion information and the output motion information, where the loss value indicates a difference between the motion type of the output motion information and the motion type of the input motion information (current motion type). The loss function module 5 may be configured to generate a loss value based on the output motion information, output statistics, input motion information, and target statistics, where the loss value indicates a difference between the motion type of the output motion information and the motion type of the input motion information (current motion type), and the loss value indicates a difference between the style of the output motion information and the target style.

[0149] The optimizer 6 is configured to optimize the MP activation based on the loss value. The optimization process performed by the motion generator 4, the optional inverse and / or forward kinematics module 8, the motion statistics module 7, the loss function module 5, and the optimizer 6 is repeated each iteration until the loss value generated by the loss function module 5 becomes smaller than a threshold. In the first iteration, output motion information is generated based on the MP and any MP activation according to the goal style. The motion generator 4 is configured to generate the output motion information based on the MP and MP activation, with or without the inverse and / or forward kinematics module 8, where the MP is generated by the MP generator 3 based on the goal statistics and thus corresponds to the MP according to the goal style.

[0150] FIG. 5 exemplarily shows a flow diagram of one embodiment of a method for training a first feed-forward neural network (FFNN) of a style statistics predictor used by the method according to the first aspect or any of its implementations.

[0151] 5, in step S51, training data is generated by recording the movements of M people (M≧1, preferably M≧10), where each of the M people performs F different movement types (F≧1, walking, dancing, running, jumping, making facial expressions, making gestures, etc.) in E different styles (E≧2, neutral style, young, old, happy, angry, sad, etc.) using motion capture and motion tracking. Preferably, the motion capture and motion tracking is performed using reflective markers on the body of each of the M people.

[0152] In step S52 following step S51, training data is input to the exercise statistics module in the form of input exercise information, which describes the exercise of each person when they move according to their respective exercise type and their respective style. The exercise statistics module corresponds to the first exercise statistics module used in the method according to the implementation form of the first aspect. In particular, the exercise statistics module may be exercise statistics module 2 shown in FIGS.

[0153] The input motion information is a time-series matrix X of length N. 1:N The data are xyz positions corresponding to the degrees of freedom of the object (robot or virtual avatar) being controlled, and the matrix X has dimension (k×3), where k is the number of degrees of freedom of the object. That is, for each time point t=1 to N, the matrix X 1:N comprises xyz positions corresponding to the degrees of freedom of the object.

[0154] Step S52 is followed by step S53, in which the exercise statistics module generates input statistics for each style of each exercise type for each person based on the respective input exercise information describing each person's exercise according to the respective exercise type and each style. 1:N The mean and / or variance of the xyz positions corresponding to the degrees of freedom of each person may be used. In step S53, input statistics are generated for each style of each movement type for each person based on the respective training data. Thus, M input statistics are generated for each style of each movement type.

[0155] In step S54 following step S53, for each style of each motion type, M input statistics resulting from the motions of M different people according to the respective motion types and styles are input to a first feedforward neural network (FFNN) of the style statistics predictor. The first FFNN may be a probabilistic neural network (PNN). In particular, the first FFNN may be the first FFNN of the style statistics predictor 1 shown in FIGS. 1-4.

[0156] Step S54 is followed by step S55, in which the first FFNN is taught the associations of M input statistics corresponding to each style of each movement type with each movement type and style. This may be done by one or more algorithms calculating the associations and / or by storing the associations in a data storage, preferably in the form of one or more look-up tables, allowing the associations to be retrieved when needed. As a result, the first FFNN network learns to distinguish between different input statistics of different movement types and different styles.

[0157] 6 exemplarily illustrates a flow diagram of one embodiment of the functionality of a first feedforward neural network (FFNN) of a style statistics predictor after training, the style statistics predictor being used by the method according to the first aspect or any of its implementations. In particular, the first FFNN may be the first FFNN of the style statistics predictor 1 shown in FIGS.

[0158] After training the first FFNN of the style statistics predictor, for example by the method shown in FIG. 5, the style statistics predictor can generate target statistics for the target motion of the target style by the following steps.

[0159] 6, in step S61, input motion information describing a current motion, which is a current motion type such as "walking" and a current style such as "fun", is input to a motion statistics module (e.g., a first motion statistics module). In step S62 following step S61, the motion statistics module generates corresponding input statistics indicating the current motion (e.g., a fun walking motion, i.e., a current motion type of "walking" and a current style of "fun") based on the input motion information. The input statistics indicate the current motion type and the current style.

[0160] Step S62 is followed by step S63, in which the generated input statistics and a target style such as "sad" are input into a first FFNN of a style statistics predictor.

[0161] Step S63 is followed by step S64, in which the first FFNN adapts or modifies the input statistics representing the current movement, i.e., the current movement type (e.g., "walking") and the current style (e.g., "happy"), to match the target statistics by using the learned associations between different input statistics and each combination of movement type and style. The target statistics represent a target movement of the same movement type as the current movement (i.e., the current movement type, e.g., "walking"), but represent a target style (e.g., "sad") instead of the current style (e.g., "happy").

[0162] The target statistic is, for example, Data X 1:N may be the mean and / or variance of the xyz positions corresponding to the degrees of freedom, which describes a target motion having the same motion type as the input motion information (current motion type) and a style corresponding to the target style.

[0163] If the first FFNN is a probabilistic neural network (PNN), the target statistics may be a range or distribution, e.g., 1:N , a distribution of means and / or variances of xyz positions corresponding to degrees of freedom, where the distribution describes the probability that the movement indicated by the target statistics corresponds to the target movement according to the current movement type and target style.

[0164] In step S65 following step S64, the first FFNN outputs target statistics, i.e., statistics relating to the target movement that is the current movement type and the target style (e.g., the sad walking movement, i.e., the current movement type "walking" and the target style "sad").

[0165] FIG. 7 exemplarily shows a flow chart of one embodiment of a method for training a second feed-forward neural network (FFNN) of a motion primitive (MP) generator used by an embodiment of the method according to the first aspect.

[0166] As shown in FIG. 7 , in step S71, the style statistics predictor used by the method according to the first aspect or any of its implementation forms generates target statistics indicating each movement type and target style based on G different target styles (G≧2, neutral style, young, old, happy, angry, sad, etc.) and E different current styles (E≧1, neutral style, young, old, happy, angry, sad, etc.), F different current movement types (F≧1, walking, dancing, running, jumping, making a facial expression, making a gesture, etc.).

[0167] Step S71 is followed by step S72, in which for each current style and each target style of each current motion type, the corresponding target statistics are input to a second feedforward neural network (FFNN) of the motion primitive (MP) generator.

[0168] Step S72 is followed by step S73, in which the second FFNN is taught the association of target statistics with each movement primitive (MP). This may be done by one or more algorithms calculating the associations and / or by storing the associations in a data storage, preferably in the form of one or more look-up tables, allowing the associations to be retrieved when needed.

[0169] As a result, the second FFNN learns to distinguish between different target statistics for different movement types and different styles, and learns how to modify the MP accordingly.

[0170] FIG. 8 exemplarily illustrates a flow diagram of one embodiment of the functionality of a second feedforward neural network (FFNN) of a trained motion primitive (MP) generator, which is used by an implementation of the method according to the first aspect.

[0171] After training the second FFNN of the MP generator, the MP generator may generate MPs according to the goal style so that the MPs correspond to basic element motion shapes that represent all possible individual motions that can be performed by the object to generate a robot motion in the goal style. The motion type is controlled by the activation of the motion primitives (MPs). The MP activation determines which MPs, and therefore which of the object's possible individual motions, should be performed to generate a motion of the current motion type. Since the MPs are generated according to the goal style, the generated motion of the current motion type becomes the goal style. The following steps are performed:

[0172] As shown in FIG. 8, in step S81, a target statistic indicating a current movement type, such as "walking," and a target style, such as "sad," is input into a second FFNN.

[0173] Step S81 is followed by step S82, in which input statistics indicating the current exercise type and the current style, such as "fun", are input into the second FFNN.

[0174] In step S83 following step S82, the second FFNN determines the current exercise type (exercise content) based on the input statistics and target statistics, determines the target style based on the target statistics, and generates an MP for exercise of the current exercise type in the target style.

[0175] In step S84 following step S83, the second FFNN changes the MP according to the current style to an MP according to the target style based on the input statistics and the target statistics, specifically based on the determined current movement type and the determined target style.

[0176] Therefore, in steps S83 and S84, the second FFNN uses the learned associations between different target statistics and respective MPs to generate MPs according to the target style based on the input statistics and target statistics. The MPs according to the target style are basic element motion shapes that represent all possible individual motions that can be performed by the object to move according to the target style.

[0177] Step S82 is an optional step. Therefore, step S82 may be omitted, and step S81 may be followed by step S83. In this case, in step S83, the second FFNN determines the current exercise type (exercise content) based on the target statistics, determines a target style based on the target statistics, and generates an MP for exercise of the current exercise type in the target style. Also, in step S84 following step S83, the second FFNN changes the MP according to the current style to an MP according to the target style based on the target statistics, specifically based on the determined current exercise type and the determined target style.

[0178] Thus, if step S81 is followed by step S83, in steps S83 and S84, the second FFNN uses the learned associations between different target statistics and respective MPs to generate MPs according to the target style based on the target statistics.

[0179] Step S84 is followed by step S85, in which the second FFNN of the MP generator outputs MPs according to the target style. These MPs may be called modified MPs.

[0180] FIG. 9 exemplarily shows a flow diagram of method steps of an optimization process according to one embodiment of an implementation of the method according to the first aspect.

[0181] 9, in step S91, the optimization process begins with any motion primitive (MP) activation after MPs for the target style are generated. These MPs may be referred to as modified MPs or MPs according to the target style.

[0182] Step S91 is followed by step S92, in which a movement generator used in an implementation of the method according to the first aspect generates output movement information describing a movement according to a certain movement type (set by MP activation) and a certain style based on the MP and MP activation according to the target style, and the style must correspond to the target style (set by MP and MP activation).

[0183] In step S93 following step S92, a motion statistics module (second motion statistics module) used in an implementation of the method according to the first aspect generates output statistics, such as the mean and / or variance of the data of the output motion information, based on the generated output motion information.

[0184] Step S93 is followed by step S94, in which the output motion information, output statistics, input motion information and target statistics are input to a loss function module used in an implementation of the method according to the first aspect.

[0185] In step S95 following step S94, the loss function module determines, based on the output motion information and the input motion information, how well the motion type of the output motion information matches with the current motion type of the input motion information, i.e., the loss function module determines the degree of difference between the motion type of the output motion information and the current motion type of the input motion information.

[0186] Also, in step S96 following step S95, the loss function module determines how well the style of the output motion information matches the target style based on the output statistics and the target statistics, i.e., the loss function module determines the degree of difference between the style of the output motion information and the target style.

[0187] Step S96 is followed by step S97, in which a loss function module generates a loss value that is used by the optimizer to generate an optimized MP activation based on the two determined differences in motion type and style. The loss value indicates how well the motion described by the output motion information, specifically the motion type and style, matches the target motion, specifically the current motion type and target style.

[0188] Steps S93 and S96 are optional steps. They may be omitted. In such a case, step S92 is followed by step S94, in which the output motion information and the input motion information are input to a loss function module. Step S94 may also be followed by step S95, in which step S97, in which the loss function module generates a loss value that is used by the optimizer to generate an optimized MP activation based on the determined difference in motion type. The loss value indicates how well the motion described by the output motion information, specifically the motion type, matches the target motion, specifically the current motion type.

[0189] Step S97 is followed by step S98, in which an optimizer generates optimized MP activations based on the loss values. The optimizer may use one or more optimization algorithms, such as a gradient descent algorithm, to generate optimized MP activations based on the loss values.

[0190] If the loss value is smaller than the threshold, the difference between the motion described by the output motion information and the target motion is sufficiently low, and therefore the optimization process is terminated after step S98. That is, if the loss value is smaller than the threshold, the difference between the motion type of the output motion information and the current motion type of the input motion information is sufficiently low. Therefore, the motion type of the output motion information is sufficiently consistent with the current motion type of the input motion information. Furthermore, if the loss value is smaller than the threshold, the difference between the style of the output motion information and the target style is sufficiently low. That is, the style of the output motion information is sufficiently consistent with the target style.

[0191] If the loss value is equal to or greater than the threshold, the method returns to step S92 after step S98, ie, steps S92 to S98 are repeated until the loss value becomes smaller than the threshold.

Claims

1. 1. A method for controlling the movement of an object, such as a robot or virtual avatar, comprising: receiving input motion information describing a current motion of a current motion type and a current style, and a target style for a target motion of said current motion type and a target style; - transforming said input motion information into output motion information based on said target style using a first feedforward neural network (FFNN) of a style statistics predictor (1), said output motion information describing said target motion; - generating input statistics based on said input exercise information using a first exercise statistics module (2), said input statistics indicating said current exercise type and current exercise style; - using the first FFNN to generate goal statistics based on the input statistics and the goal style, the goal statistics being indicative of the current motion type and the goal style; - transforming said input motion information into said output motion information based on said input statistics and said target statistics; - generating MPs according to said target style based on said target statistics and said input statistics using a second feedforward neural network (FFNN) of a movement primitive (MP) generator (3); - controlling the motion of the object based on the output motion information including the MP according to the target style so that a motion type of the motion of the object corresponds to the current motion type and a style of the motion of the object corresponds to the target style; A method comprising:

2. The method of claim 1 , wherein the first FFNN is a probabilistic neural network (PNN).

3. The method comprises: - using said first FFNN to predict, based on said input statistics and said target style, how one or more statistics, such as the mean and / or variance of said input statistics, must be changed to transform said input statistics into said target statistics; - using said first FFNN to transform said input statistics into said target statistics based on the results of said prediction; The method of claim 1 , comprising:

4. The method comprises: - using said second FFNN to generate said MP according to said target style based on said input statistics and said target statistics; The method of claim 1 , comprising:

5. The method comprises: - changing the MP according to the current style to the MP according to the target style based on the target statistics, in particular based on the target statistics and the input statistics, using the second FFNN. The method of claim 1 , comprising:

6. The method comprises: - generating said output movement information based on said MP and MP activation according to said target style using a movement generator (4); 6. The method of claim 1, comprising:

7. The method comprises: - generating a loss value based on the input motion information and the output motion information using a loss function module (5), wherein the loss value indicates a difference between the motion type of the output motion information and the motion type of the input motion information; - optimizing said MP activation based on said loss value using an optimizer (6); The method of claim 6, comprising:

8. The method comprises: - generating a loss value based on the input motion information, the output motion information, and the target style using a loss function module (5), wherein the loss value indicates a difference between the motion type of the output motion information and the motion type of the input motion information, and a difference between the style of the output motion information and the target style; - optimizing said MP activation based on said loss value using an optimizer (6); The method of claim 6, comprising:

9. The method comprises: - generating output statistics based on said output athletic information using a second athletic statistics module (7); - generating a loss value based on the input motion information, the output motion information, the target statistics, and the output statistics using a loss function module (5), wherein the loss value indicates a difference between the motion type of the output motion information and the motion type of the input motion information, and a difference between the style of the output motion information and the target style; - optimizing said MP activation based on said loss value using an optimizer (6); The method of claim 6, comprising:

10. 10. The method of claim 7, wherein steps 6 and 7, or 6 and 8, or 6 and 9 are repeated until the loss value is less than a threshold, and in a first iteration, the output motion information is generated based on the MP and any MP activation according to the target style.

11. The method comprises: - generating, using said motion generator (4), target motion information based on said MP and MP activation according to said target style; - generating said output kinematic information using an inverse and / or forward kinematics module (8) based on physical constraints of said object, such as one or more joint angles and one or more bone lengths of said object, and on said target kinematic information; 11. The method of any one of claims 6 to 10, comprising:

12. 12. The method of claim 1, wherein the motion of the object is controlled by controlling one or more control variables of the object, such as one or more Euler angles, quaternions, joint angles and / or xyz positions.

13. - when one or more xyz positions are controlled as control variables, said inverse kinematics module (8) is used to generate output kinematic information; The method of claim 12 when dependent on claim 11, wherein the forward kinematics module (8) is used to generate the output motion information when one or more Euler angles, quaternions, and / or joint angles are controlled as control variables.

14. A control device for controlling the movement of an object such as a robot or a virtual avatar, comprising: A control device, the control device being configured to perform the method of any one of claims 1 to 13.

15. A robot comprising the control device of claim 14 configured to control the movement of the robot.

16. A computer program comprising a program code for performing the method according to any one of claims 1 to 13 when the computer program is executed on a processor.

Images