Method and apparatus for operating a machine

Abstract

An apparatus and method for operating a machine (104), the method comprising providing a skill sequence (132) of the machine (104) for performing a task, selecting a state sequence from a plurality of state sequences depending on a likelihood, wherein the likelihood is determined depending on a transition probability from a final state of a first sub-sequence of state sequences of a first skill in the skill sequence (132) to an initial state of a second sub-sequence of state sequences of a second skill in the skill sequence (132).

Description

BACKGROUND

[0001] It is desirable to solve complex assembly tasks using robotic automation with manually shown robot skills to the robot via kinesthetic teaching. SUMMARY

[0002] This is achieved by a method and an apparatus for operating a machine according to the independent claims.

[0003] The method of operating a machine comprises providing a sequence of machine skills for performing a task, selecting a sequence of states from a plurality of sequences of states depending on a likelihood, wherein the likelihood is determined depending on a transition probability from a final state of a first sub-sequence of the sequence of states of a first skill in the sequence of skills to an initial state of a second sub-sequence of the sequence of states of a second skill in the sequence of skills. The optimal sequence of states is found in this way based on a desired goal of the task and an initial state of the machine or the environment.

[0004] The method can comprise providing a first model for the first skill, wherein parameters of the first model define a trajectory in the states of the first sub-sequence of states, or by providing a first model for the second skill, wherein parameters of the first model define a trajectory in the states of the second sub-sequence of states, the method further comprising determining parameters for the states of the sequence of states and determining a control signal for operating the machine according to the trajectory depending on the parameters. The trajectory is a reference for the optimized control of the machine.

[0005] The method can comprise providing a second model comprising a plurality of sequences of states, determining a likelihood of at least one of the sequences depending on transition probabilities between sub-sequences of states of skills in the at least one sequence, selecting the sequence of states from the second model having a higher likelihood compared to at least one other sequence of states of the plurality of sequences of states. The second model comprises a concatenation of the first models, the concatenation of the first models being composed for determining the optimal sequence of states using a Viterbi algorithm.

[0006] The method can comprise mapping a first observation of the machine to a first state, wherein the sequence of states starts at the first state, and / or mapping a second observation to a second state, wherein the sequence of states ends at the second state. This allows determining the entire optimal sequence of states from the two observations.

[0007] The first observation can characterize the machine or the environment of the machine before performing a first skill in the sequence of skills, and / or wherein the second observation can characterize the machine or the environment of the machine after performing a last skill in the sequence of skills. This enables a simplified Viterbi algorithm based on only the initial and final observations in the task.

[0008] Providing the second model can comprise determining a transition probability from a final state of a first sub-sequence according to the first skill to a plurality of initial states of different second sub-sequences according to different instances of the second skill. This concatenates the different sub-sequences of the second skill with one sub-sequence of the first skill.

[0009] The final state of the first sub-sequence can be determined depending on a final component of the instance of the first skill. The instance of the first skill can be a TP-HSMM having a plurality of final components, i.e. different first sub-sequences. The final component defines the final state of one first sub-sequence and can thus be linked to an initial component of an initial state of a second sub-sequence.

[0010] At least one initial state of the second sub-sequence can be determined depending on an initial component of the instance of the second skill.

[0011] The transition probability can be determined depending on a divergence between a first Gaussian mixture model of the final component and a second Gaussian mixture model of the initial component.

[0012] The instance of the first skill and / or the instance of the second skill can be defined by a task parameterized hidden semi-Markov model.

[0013] An apparatus for operating a machine, adapted to perform the steps of the method. BRIEF DESCRIPTION OF DRAWINGS

[0014] Further advantageous embodiments can be derived from the following description and the attached drawings. In the drawings:

[0015] Figure 1 A part of a control system is depicted,

[0016] Figure 2 A part of a control system is depicted,

[0017] Figure 3 Aspects of a first model for operation of a machine are depicted,

[0018] Figure 4 Aspects of a second model for operation of a machine are depicted,

[0019] Figure 5 Steps of a method are depicted. DETAILED DESCRIPTION

[0020] Figure 1 A part of a control system 100 is depicted, which comprises a controller 102 for controlling a machine 104. The controller 102 is adapted to output a control signal 106 for controlling the machine 104.

[0021] The machine 104 can be a robot. The controller 102 can be adapted to control the machine 104 depending on information about the state of the machine 104 or its environment. The environment can comprise at least one object manipulable by the machine 104. The machine 104 and / or its environment are referred to herein as a system.

[0022] One approach to solve complex assembly tasks is robot automation with robot skills manually demonstrated to the robot via kinesthetic teaching. Kinesthetic teaching is easy to perform by any worker familiar with the assembly process. Several demonstrations are collected for each unique robot skill (e.g. inserting a nail in a hole).

[0023] To capture the randomness of human demonstrations, the demonstrated skills are encoded in a statistical model, e.g. a task-parameterized hidden semi-Markov model TP-HSMM.

[0024] According to the following description, a particular skill is encoded in a first model, and different first models for different skills are executed in order to determine a second model.

[0025] In the described context, a model state refers to a Gaussian distribution of a TP-HSMM. A system state refers to a robot state, e.g. a position-orientation of its end effector, or a position and velocity of its end effector. An observation refers to data used to train a skill (i.e. to train a TP-HSMM) or to reproduce a skill using a Viterbi algorithm with an initial observation of the start of the skill and a target or goal observation of the end of the skill. These observations can correspond to system states, or comprise system states. Observations can include some additional variables specific to the problem.

[0026] This approach uses a concatenation of multiple first models - e.g. model states encoded in a TP-HSMM - to build together a larger second model corresponding to a multi-stage assembly process. For example, for the case of a first model for a pick skill and a first model for a place skill, a second model for the pick and place skills is created that corresponds to a single TP-HSMM. A first model for a particular skill can encode a multimodal trajectory distribution for that particular skill to ensure generality and flexibility. For example, a pick skill can perform an object pick method from the side, top, bottom of an object.

[0027] According to this approach, a large number of possible trajectory distribution patterns are encoded in the concatenated (i.e. ordered) second model. The number of possible trajectory distributions grows exponentially with the number of skills in the concatenation.

[0028] According to this approach, it is determined which trajectory distribution pattern the machine 104 should follow to achieve the goal of the task. A framework will be described below that is able to find the most likely sequence instance of skills that achieves the goal.

[0029] According to one example, a cascaded model for pick and place is provided. Given a goal as a target location for a placed object, forward and backward passes of a second model find a sequence of states that most likely achieves the goal compared to other sequences of states that can be found according to the second model.

[0030] In one example, the orientation of the end effector state of the robot is encoded and represented as a unit quaternion. Quaternions belong to the group of 3 hyperspheres, which can be described as Riemannian manifolds.

[0031] In this example, states are defined in a Riemannian manifold The modeling technique TP-HSMM is adapted to deal with variables in a Riemannian manifold.

[0032] The controller 102 can be used for linear quadratic tracking to solve a control problem, e.g., to retrieve an optimal trajectory of a sequence of states depending on information 108 about the sequence of states. The controller 102 can be adapted according to an extendable finite horizon linear quadratic tracking control algorithm for Riemannian manifolds.

[0033] The controller 102 can be an optimal Riemannian controller: in optimal linear quadratic tracking, the dynamics are defined to be linear in states and actions. The cost function to be minimized is quadratic in states and actions. Given an initial state of the system, a complete sequence of states is defined as a function of the control actions at each time step. Further, the states and control actions can be concatenated into two large column vectors at each time step, and their dynamics relation is concatenated into a large quadratic matrix. The quadratic cost is defined as a linear function of the control actions. Finally, the optimal control actions can be obtained by matrix inversion.

[0034] However, the matrix inversion scales cubically with the size of the matrix. In an exemplary case, the size is the number of time steps times the dimension of the states. This solution is derived purely from linear algebra steps and allows to compute the optimal control and state sequence in one batch.

[0035] Instead of solving the control problem in a batch, the controller 102 can be implemented as an optimal feedforward and feedback controller that recursively solves the control problem. The Bellman equation can be used as described below to solve the control problem using a cost-to-go function. Thus, the optimal behavior at one state decreases its cost in the steepest direction. By exploiting this property of the cost function, the optimal control action is found recursively. In this example, the recursive formula described below is used for Riemannian manifolds. The computational cost of this solution is lower.

[0036] In one aspect, the controller 102 is adapted to control the machine 104 in accordance with a reference trajectory, which corresponds to a sequence of states provided in dependence of a cost function. The sequence of states can be determined using a Viterbi algorithm.

[0037] The optimal control scheme is defined in dependence of a description of the system A, e.g.:

[0038]

[0039] where

[0040]

[0041] and

[0042]

[0043] where

[0044]

[0045]

[0046] where

[0047]

[0048] where

[0049]

[0050] and where is defined and a parallel transmission operation between and where R defines a parameter of the control.

[0051] For the optimal control solution, in general, the properties of A and B are substantially determined by the system dynamics model. The R matrix is usually designed manually.

[0052] The information 108 can be provided by the device 110 to the controller 102 for planning a state for an operation of the machine 104.

[0053] Figure 2 A part of the device 110 is depicted. The device 110 comprises at least one processor 112 and at least one memory 114. The device 110 in this example comprises at least one input 116 for at least one observation 118. The observation 118 can define a state. The device 110 in this example comprises at least one output 120 for the information 108 about the sequence of states. The device 110 can comprise a user interface (not depicted). The user interface can be adapted to detect a task to be performed by the machine 104 from a user input.

[0054] The device 110 and the controller 102 are adapted to perform the steps in the method described below for controlling the machine 104. The at least one processor 112 can be adapted to execute instructions stored in the at least one memory 114 to perform the steps of the method to determine the information 108 for the at least one output 120 from at least one observation 118 received at the at least one input 116.

[0055] The observation 118 can be defined by data points that define a system state. In training, a plurality of observations 118 for performing a skill can be captured from a human demonstration by at least one sensor. Preferably, a spatio-temporal sequence of observations is captured while a human demonstrates the skill.

[0056] In this example, the machine 104 is adapted to perform a plurality of different skills.

[0057] A set of demonstrations 122 can be recorded for the plurality of different skills. Figure 3 Three demonstrations 122-1, 122-2, 122-3 of three different skills are exemplarily depicted. Figure 3 Three spatio-temporal sequences 122-11, 122-12, 122-13 of observations for the same demonstration 122-1 are exemplarily depicted. In one example, the machine 104 is a robot adapted to pick up, place or move objects in a workspace of the robot. The machine 104 can be adapted to perform the same skill on a plurality of different objects or on the same object at a plurality of different locations in the workspace. The processes described herein can be applied to any other skill of the robot or machine 104 and to any other object.

[0058] In this example, the spatial coordinates of one spatio-temporal sequence of observations are defined relative to a coordinate system having an origin assigned to one object. A spatio-temporal sequence of a particular object exercising a particular skill can be recorded from the perspective of a particular coordinate system and transformed into frames. In this example, the set of demonstrations 122 comprises a plurality of spatio-temporal sequences of observations recorded from different perspectives from a plurality of human demonstrations, the spatio-temporal sequences being transformed in frames assigned to different objects. The demonstrations in the example are human demonstrations. The demonstrations can also be demonstrations made by a training device.

[0059] The device 110 is adapted to determine a plurality of first models 124 depending on the set of demonstrations 122. In this example, the first models 124 are determined for each of the different skills. Figure 3In this example, a model 124-1 defined by the first demonstration 122-1 of the first skill, a model 124-2 defined by the second demonstration 122-2 of the second skill, and a model 124-3 defined by the third demonstration 122-3 of the third skill are depicted. In this example, the model 124-1 is a task parameterized hidden semi-Markov model TP-HSMM, the model 124-2 is a TP-HSMM, and the model 124-3 is a TP-HSMM.

[0060] The device 110 can be adapted to determine the HSMM frequency shift dependent on a plurality of Gaussian components of a GMM Gaussian mixture model.

[0061] The device 110 is adapted to determine at least one task parameter 126-1 of the model 124-1, at least one task parameter 126-2 of the model 124-2, and at least one task parameter 126-3 of the model 124-3. The at least one task parameter 126-1 can comprise a model parameter 126-11 for a state before execution of the first skill (precondition model) and a model parameter 126-12 for a prediction of a state after execution of the first skill (predicted final state model). The device 110 is adapted to determine these parameters dependent on at least one frame determined from at least one observation for a system first state before execution of the first skill. The at least one state parameter 126-1 can comprise a further model parameter 126-13 for a state after execution of the first skill (final condition model). These can be implemented as a GMM. The device 110 can be adapted to determine the further model parameter 126-13 dependent on at least one frame determined from at least one observation for a system state after execution of the first skill. In this example, the device 110 is also adapted to determine these model parameters for the models 124-2 and 124-3.

[0062] Figure 4 Aspects of a second model 128 for operation of the machine 104 are depicted. The second model is in the example used to determine a sequence of states of operation of the machine 104.

[0063] The device 110 is adapted to determine the second model 128 dependent on a skill sequence 130 for performing a task 132. The task 132 can comprise manipulating an object with the machine 104. In this example, the task 132 is provided by a user of the machine 104 by user input via a user interface.

[0064] For the exemplary task 132, the first skill 122-1, the second skill 122-2, and the third skill 122-3 can be executed in this order for performing the task 132. In this case, the device 110 is adapted to map the task 132 to this order of the three skills.

[0065] In this example, the device 110 is adapted to determine a concatenation of the instance 128-1 of the model 124-1 with the first instance 128-2 of the model 124-2 and the second instance 128-3 of the model 124-2.

[0066] The device 110 is adapted to update the second model 128 from the instance 128-1 of the model 126-1 and the first instance 128-2 of the model 126-2. The device 110 is adapted to update the second model 128 from the instance 128-1 of the model 126-1 and the second instance 128-3 of the model 124-3. More specifically, the second model 128 can be updated from a first concatenation comprising the instance 128-1 and the first instance 128-2 and from a second concatenation comprising the instance 128-1 and the second instance 128-2.

[0067] In an example, the second model 128 comprises at least one sequence from the first concatenation and at least one sequence from the second concatenation. In an example, the first concatenation comprises different instances of the model 124-3 for a plurality of components of the model 124-2 and in an example, the second concatenation comprises different instances of the model 124-3 for a plurality of components of the model 124-2.

[0068] This means that for different components, the same skill sequence can result in a plurality of different concatenations. The second model 128 comprises a plurality of state sequences.

[0069] The device 110 is adapted to determine an optimal state sequence for controlling the machine 104 in dependence on the model state sequences in the second model 128 . In one aspect, the device 110 is adapted to apply a Viterbi algorithm to determine which model state sequence in the second model 128 is selected as the optimal state sequence .

[0070] A method of determining an optimal state sequence is described below. A model state sequence from the model 128 is considered optimal, e.g. when it has been assigned the highest likelihood out of all the likelihoods of the available model state sequences in its second model 128. It is sufficient for a model state sequence to have a higher likelihood than another model state sequence for it to be useful. Thus, this idea is not limited to the optimal state sequence .

[0071] The input to the method is a skill sequence , an initial state , a target or goal state , a plurality of first models for the skills in the skill sequence and their respective model parameters .

[0072] The method comprises a sequence of skills dependent on the sequence of skills , the second model 128, hereinafter called , is computed by concatenating the TP_HSMMs of the skills in the sequence in their successive order starting with the first skill in the sequence and ending with the last skill of the sequence.

[0073] The following example describes the concatenation of the TP-HSMM of a first skill with the TP-HSMM of a second skill . As described below, the synthetic model parameters can be determined from the GMMs associated with the first skill and the second skill . Other pairs of skills can be similarly processed, for example or .

[0074] The second model may comprise all possible concatenations.

[0075] The TP-HSMM of the first skill and the TP-HSMM of the second skill comprise:

[0076] a) for all final components do:

[0077] create a copy:

[0078]

[0079] compute all initial components of the TP-HSMM for the second skill from the transition probabilities of one final component of the first skill TP-HSMM of the first skill

[0080]

[0081] where KL ​​is the Kullback-Leibler divergence described by J. R. Hershey and P. A. Olsen in "Approximating the Kullback-Leibler divergence between Gaussian mixture models" (Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP), vol. 4. IEEE, 2007, pp. IV-317); where is the GMM associated with component of frame p, where is the GMM associated with component of frame p, where is the set of common frames shared by the two models, which can be forced non-empty by always adding global frames.

[0082] The TP-HSMM of the second skill is updated by and the GMM of the final component of the second skill :

[0083]

[0084] where is defined as

[0085]

[0086] where the parameters of the updated Gaussian at each frame p are computed as:

[0087]

[0088] and

[0089]

[0090] adding the concatenation of and to the second model

[0091] (b) optionally, setting the additional parameters of the second model

[0092] (c) optionally, the synthetic model parameters of the model may be determined by:

[0093] . ​​

[0094] The method further includes calculating the optimal state sequence as follows. :

[0095] For the second model All cascades depend on a given initial observation of the tasks that should be performed for the optimal sequence. and given the final observation For each skill 'a' in the cascade, determine the likelihood.

[0096]

[0097]

[0098] in It is the likelihood that the model is in state j at step or time t but not at step or time t+1, and

[0099]

[0100] in It is the model of skill A. The global Gaussian component of model state j in the second model Gaussian component, where model state j is the state of the model state sequence in the second model Gaussian component.

[0101] In this example, two independent variables, d and i, are recorded for the model state j in the cascade to maximize the likelihood.

[0102] The Viterbi algorithm is used to compute the most probable state sequence using a transition matrix and emission probabilities. The transition matrix is ​​used as input to the forward derivation of the Viterbi algorithm. Given the skill level, initial observations, and final observations represented by an HSMM, the two formulas above are used to compute the most probable HSMM state sequence. This HSMM state sequence corresponds to a TP-GMM Gaussian. This state sequence is given for each time step of the time range. The time range can be specified by, for example, an expert. That is, at each time step of the time range, there exists a single TP-HSMM that is assigned as the most probable one.

[0103] b) In the second model The skill with the highest likelihood is identified in the middle. sequence The final model state j of the last skill in the process.

[0104] c) Determine the model state sequence as the first state for skill selection. sequence The previous model state j is the final model state j of the last skill with the highest likelihood. In this example, the Viterbi algorithm is used to evaluate the likelihood of multiple model state sequences in the second model 128. In one aspect, the likelihood is evaluated as the probability of observation from the initial state of the target state through the model states in the cascaded model. The model state sequence with the highest likelihood is then selected as the optimal sequence.

[0105] The method further includes performing skills by controlling machine 104 as follows. sequence Each skill :

[0106] a) Observe the current system status

[0107] b) Update the parameters of the corresponding first model, i.e., from the given current system state. The optimal sequence at the beginning subsequence of Global Gaussian components of model state j This means subsequence It is a skill The optimal sequence of model states.

[0108] c) Tracking the sequence Global Gaussian components of model state j The provided trajectory. This implies skill. optimal subsequence The trajectory was tracked until the skill Until the final model state.

[0109] From the first skill Initially, steps a) through c) can be repeated until the task's objective is achieved, i.e., until the task has been completed. This continues until the last model state of the last skill in the sequence.

[0110] The following reference Figure 5 An exemplary method for operating machine 104 is described. The model state sequence can be given a first system state. Starting at, and in the given second system state End of section.

[0111] The method includes step 502.

[0112] In step 502, the sequence of skill 132 of machine 104 It is provided for performing tasks.

[0113] Next, proceed to step 504.

[0114] Step 504 may include providing TP-HSMM for the first skill. Step 504 may include providing TP-HSMM for the second skill.

[0115] Parameters of TP-HSMM Define the trajectory in model state j of the corresponding subsequence of model state for the corresponding skill.

[0116] Next, proceed to step 506.

[0117] In step 506, a second model 128 is provided. The second model 128 includes multiple model state sequences.

[0118] Determining the second model 128 may include determining the transition probabilities from a final model state based on a first subsequence of the first skill to multiple initial model states based on different second subsequences of different instances of the second skill. .

[0119] The final model state of the first subsequence can depend on the final component of the instance of the first skill. To determine.

[0120] At least one initial model state of the second subsequence can depend on the initial components of the instance of the second skill. To determine.

[0121] Transition probability It can depend on the final component First Gaussian Mixture Model and Initial Components The divergence between the second Gaussian mixture models is determined, for example, based on the Kullback-Leibler divergence described above.

[0122] Instances of the first skill and / or the second skill can be defined by their respective TP-HSMMs.

[0123] The method may include, in step 508, the first observation of machine 104. Mapped to the first system state Optimal state sequence It can be observed in the first observation First matching system state Start here.

[0124] The method may include providing a second system state in step 510. Second system state It could be a pre-planned second observation. To the second state The mapping. Optimal state sequence. Second system state End of section.

[0125] First Observation This can be characterized in the sequence of skill 132. The first skill in the game refers to the environment of Machine 104 or Machine 104 before the first skill. The second observation... This can be characterized in the sequence of skill 132. The last skill in the game is in Machine 104 or the environment of Machine 104.

[0126] Next, proceed to step 512.

[0127] Step 512 includes selecting the optimal state sequence from multiple model state sequences of the second model 128 based on the likelihood of the corresponding sequence depending on the model states. .

[0128] Likelihood depends on the transition probability as described above.

[0129] This means determining the likelihood of at least one of the plurality of sequences.

[0130] The model state sequence selected from the second model 128 has a higher likelihood than at least one other model state sequence among the plurality of state sequences.

[0131] Next, proceed to step 514.

[0132] In step 514, the optimal state sequence is determined. The parameters of the current model state j .

[0133] Parameters can be Parameters determined for TP-HSMM It defines the trajectory of model state j of the corresponding state subsequence of the corresponding skill.

[0134] Next, proceed to step 516.

[0135] Step 516 includes based on parameters The trajectory is determined for the control signals used to operate machine 104. In this example, the optimal control solution is determined recursively as described above.

[0136] Then, execute step 514 for the next state in the state sequence, until the skill sequence is reached. The final state j of the last skill in the game.

Claims

1. A method for operating a machine (104), characterized in that, A skill sequence (132) for a machine (104) to perform a task is provided (502), and a state sequence is selected (512) from a plurality of state sequences based on likelihood, wherein the likelihood is determined based on the transition probability from the final state of a first state subsequence of a state sequence (110) of a first skill in the skill sequence (132) to the initial state of a second state subsequence of a state sequence of a second skill in the skill sequence (132), wherein the final state of the first state subsequence is determined based on the final component of an instance of the first skill, wherein at least one initial state of the second state subsequence is determined based on the initial component of an instance of the second skill, and wherein the transition probability is determined based on the divergence between a first Gaussian mixture model of the final component and a second Gaussian mixture model of the initial component.

2. The method according to claim 1, characterized in that, The method provides (504) a first model (124-1, 124-2, 124-3) for the first skill, wherein the parameters of the first model (124-1, 124-2, 124-3) define the trajectory in the state of the first state subsequence, or provides (504) a first model (124-1, 124-2, 124-3) for the second skill, wherein the parameters of the first model (124-1) define the trajectory in the state of the second state subsequence, the method further comprising determining (514) parameters for the state of the state sequence, and determining (516) control signals for operating the machine (104) based on the trajectory depending on the parameters.

3. The method according to claim 1 or 2, characterized in that, Provide (506) a second model (128) comprising the plurality of state sequences, and determine (512) the likelihood of the at least one sequence based on the transition probabilities between skill state subsequences in at least one of the plurality of state sequences, and select from the second model (128) a state sequence having a higher likelihood compared to at least one other state sequence among the plurality of state sequences.

4. The method according to claim 1 or 2, characterized in that, Map (508) the first observation for the machine (104) to a first state, where the state sequence begins at the first state, and / or map (510) the second observation to a second state, where the state sequence ends at the second state.

5. The method according to claim 4, characterized in that, The first observation represents the environment of the machine (104) or machine (104) before the first skill in the skill sequence (132), and / or the second observation represents the environment of the machine (104) or machine (104) after the last skill in the skill sequence (132).

6. The method according to claim 3, characterized in that, The second model (128) provided (506) includes determining the transition probabilities from the final state of a first state subsequence based on the first skill to multiple initial states of different second state subsequences based on different instances of the second skill.

7. The method of claim 6, wherein the instance of the first skill and / or the instance of the second skill is defined by a task-parameterized hidden semi-Markov model.

8. An apparatus for operating a machine (104), characterized in that, The apparatus is adapted to perform the steps of the method according to any one of claims 1 to 7.

9. A computer program, characterized in that, The computer program includes computer-readable instructions that, when executed by a computer, cause the computer to perform the steps of the method according to any one of claims 1 to 7.

Images