Method and system for operating a robot

Abstract

The invention relates to a method for operating a robot, comprising the steps of: providing image data of surroundings of the robot, said image data being assigned to robot poses; forecasting or updating a robot target pose or determining new robot target poses by means of data processing which is at least partially based on machine learning; determining or updating an intended robot movement or determining a new intended robot movement on the basis of the target pose and controlling drives of the robot to carry out the intended robot movement. The invention also relates to a method for training the data processing, and to a system and computer program (product).

Description

[0001]2022P00048WO 1 / 41 Kuka Deutschland GmbH Description Method and system for operating a robot The present invention relates to a method for operating a robot, which comprises multiple predictions of a robot target pose by means of data processing based at least partially on machine learning, in particular by means of a regression method and / or at least one artificial neural network, on the basis of provided image data, a method for training the data processing, and a system and a computer program or computer program product for carrying out a method described here. One object of the present invention is to improve the operation of a robot and / or data processing therefor based at least partially on machine learning, in one embodiment the training thereof. This object is achieved by a method having the features of claim 1 or 6.Claims 10 and 11 protect a system, computer program, or computer program product for carrying out a method described here. The subclaims relate to advantageous developments. According to one embodiment of the present invention, a robot has at least one robot arm. In one embodiment, the robot, preferably the or one or more of the robot arms, has at least three, preferably at least six, and in one embodiment at least seven, joints or (movement) axes, preferably rotary joints or axes, which are adjusted by, preferably motor-driven, or in one embodiment electric-driven, drives of the robot or are configured to do so. According to one embodiment of the present invention, a method for operating the robot comprises the steps of: a1) providing start image data of an environment of the robot, which are assigned to a robot start pose, in one embodiment in which orfor the robot start pose are generated with the aid of at least one camera and / or image processing; 2022P00048WO 2 / 41 Kuka Deutschland GmbH c1) predicting a first robot target pose by means of data processing based at least partially on machine learning, in one embodiment by means of a regression method and / or at least one, preferably deep, artificial neural network (“(deep) artificial neural network”), on the basis of the provided start image data, in particular using the provided start image data as input for the data processing; d1) determining a robot target movement on the basis of the first target pose; and e1) controlling drives of the robot to execute this robot target movement. According to one embodiment of the present invention, the method comprises the following, multiple times, in one embodiment until the target is reached orOccurrence of a termination condition, preferably a (sufficiently accurate) reaching of a current target pose or a detection of an error, for example a detection of non-reachability or (insurmountable) blockage, reaching of a predetermined maximum number of attempts or the like, repeated steps: (in each case) a2) Providing new image data of a, preferably the same, environment of the robot, which are assigned to a current robot pose, in one embodiment in which or for the current robot start pose(s) are generated with the aid of at least one, preferably the at least one, camera and / or image processing, wherein this providing takes place during the execution of the (previous ormost recently determined, possibly most recently updated) robot target movement is carried out; c2) predicting an updated robot target pose by means of data processing on the basis of this new image data provided; d2) updating the robot target movement based on this updated target pose; and e2) controlling, preferably the robot drives to execute the updated robot target movement, in particular instead of the previous or most recently determined robot target movement. 2022P00048WO 3 / 41 Kuka Deutschland GmbH Thus, in one embodiment, approaching the target pose is broken down into two sub-problems or sub-tasks and, in one embodiment, continuously during the movement of the robot - in parallel and / or with different clock rates - by an (appropriately trained) artificial intelligence orData processing based at least partially on machine learning, preferably a regression method and / or at least one, preferably deep, artificial neural network, takes current image data (and, as explained below, possibly previous image data and / or further data, in particular kinematic and / or environmental data) as input, which outputs a, preferably most probable, target pose; and - by means of a further function, the (updated) target pose predicted by this artificial intelligence or data processing (and, as explained below, possibly kinematic data) is taken as input, which in turn outputs an (updated) robot target movement. In one embodiment, this allows the robot to be moved reactively, preferably over large parts of a free workspace, for example half a meter or a whole meter.Additionally or alternatively, in some embodiments, explicit (prior) knowledge of variable environmental conditions can be taken into account by the method, preferably by artificial intelligence or data processing, and / or (expensive) force-torque sensors can be dispensed with and / or the robot can be used or operated without expert knowledge. The first or updated robot target pose can, in particular, be a final or robot end pose that the robot should have at the end of a task or application to be performed by it, for example, at the end of a start-up process for picking up a load, at the end of a load transfer, at the end of a processing path, or the like.According to an embodiment of the present invention, which can be combined with steps a2) - e2) or implemented without them, the method comprises, preferably in addition to or particularly preferably instead of steps a2) - e2), the steps following steps a1) - e1), which are repeated several times in one embodiment until a termination condition is reached or occurs, preferably a sufficiently small change in the respectively predicted new target pose compared to a previously 2022P00048WO 4 / 41 Kuka Deutschland GmbH predicted new target pose or a detection of an error, for example a detection of non-accessibility or (insurmountable) blockage, a reaching of a predetermined maximum number of attempts or the like: (in each case) a3) providing new image data of a, preferably the (same), environment of the robot, which are assigned to a current robot pose, in one embodiment in the orfor the current robot start pose(s) are generated using at least one, preferably the at least one, camera and / or image processing, wherein this provision takes place in one embodiment before or after or particularly preferably during the execution of the (previous or most recently determined, possibly most recently updated) robot target movement; c3) predicting a new robot target pose by means of the data processing or artificial intelligence on the basis of this new image data provided; d2) determining a new robot target movement, preferably of a next path section, on the basis of this new target pose; and e2) controlling, preferably the drives of the robot to execute the new robot target movement. In this embodiment of the present invention, the first robot target pose or one or more of the new robot target poses can in particular (each) be a robot target pose of a next path section ortime step, in particular control cycle or the like, wherein the robot moves to the first robot target pose and one or more of the new robot target poses successively one after the other in order to carry out a task or application, for example a move-up process for picking up a load, a transfer of a load, a machining path or the like. Thus, the present invention comprises in particular two different aspects, which are described here together and can advantageously be implemented in combination with one another or, particularly advantageously, independently: A) a (suitably trained) artificial intelligence or data processing based at least partially on machine learning, preferably a 2022P00048WO 5 / 41 Kuka Deutschland GmbH regression method and / or at least one, preferably deep, artificial neural network, outputs a, preferably final or end orTarget pose; or B) a (suitably trained) artificial intelligence or data processing based at least partially on machine learning, preferably a regression method and / or at least one, preferably deep, artificial neural network, outputs, preferably during a movement of the robot, a target pose for one or more next time step(s) or path section(s). In aspect A), in one embodiment, various, preferably arbitrary, movements can advantageously be performed during data acquisition for training the data processing, for example zigzag-like, spiral-like, or the like, whereby a more diverse database can advantageously be obtained, since here only the final pose is important, so to speak, and the data processing or artificial intelligence is trained on its prediction.If, in aspect A), the provision of new image data fails during operation, the robot can advantageously continue moving based on the last forecast. The forecast of the target pose can be used in particular to move the robot directly to the target pose. Alternatively, the forecast can also be used to deliberately avoid a target pose, for example, if the target pose is a variable obstacle or the like. In aspect A), the data processing in one embodiment does not learn how the robot should move, but only where the target pose is. In aspect B), in one embodiment, during data acquisition to train the data processing, the movements are carried out in the way the robot is later supposed to move. If, in aspect B), the provision of new image data fails during operation, the robot comes to a standstill in one embodiment.Advantageously, in aspect B), the data processing learns how the robot should move. As already mentioned, both aspects are discussed together here, wherein according to one embodiment of the present invention or aspect A), the method comprises steps a1), optionally b1), c1), d1) and e1) as well as the subsequent, repeatedly repeated steps a2), optionally b2), c2), d2) and e2), and 2022P00048WO 6 / 41 Kuka Deutschland GmbH according to another embodiment of the present invention or aspect B), the method comprises steps a1), optionally b1), c1), d1) and e1) as well as the subsequent, repeatedly repeated steps a3), optionally b2), c3), d3) and e3) (the step of providing updated kinematic data explained below is referred to uniformly as step b2) for the sake of more compact representation in both aspects A), B).In one embodiment, in step c1), a sequence with the first robot target pose and one or more (respectively) subsequent robot target poses is predicted by means of data processing on the basis of the provided start image data, and in step d1), the robot target movement is determined based on this sequence and / or in step c3), a sequence with the new robot target pose and one or more (respectively) subsequent robot target poses is predicted by means of data processing on the basis of the provided new image data, and in step d3), the new robot target movement is determined based on this sequence. If yi denotes a robot target pose, then in one embodiment, based on the start or new image data, not only the robot target pose y is determined. i , but also one or more subsequent robot target poses y i+1 , y i+2, ... and determine the robot target movement (respectively) based on this sequence {yi, yi+1, yi+2, ...}, preferably in such a way that the robot approaches these robot target poses one after the other. Here, y can in particular comprise, in particular be, the end effector pose x explained below and / or the joint coordinates q explained below. By such a prediction of a sequence of two or more consecutive robot target poses by means of data processing based on image data, in one embodiment, trained robot movements can be reactively implemented particularly advantageously. In one embodiment, in step c2), the updated robot target pose is predicted by means of data processing based on a group or history of at least some of the previously provided image data, in one embodiment chronologically. In one embodiment, in step c3), the new robot target pose, in one embodiment the sequence,by means of data processing based on a chronological group or history of at least some of the previously provided image data. This can improve the accuracy and / or reliability of the forecast in one embodiment. Additionally or alternatively, in one embodiment, in step e2), new motion commands for controlling the robot's drives are determined several times (and these drives are controlled based on these motion commands) before step c2) is performed again. In other words, in one embodiment, the robot's drives are controlled at a higher clock rate or higher frequency compared to an update of the robot's target pose based on new image data.preferably also controlled at a higher clock rate or higher frequency compared to the provision of new image data. Control within the meaning of the present invention can generally comprise, in particular be, regulation or control based on target and fed-back actual values. As a result, in one embodiment, an advantageous, for example, smoother, robot target movement can be executed and / or the execution or control of the robot target movement, in particular its precision, speed and / or reliability, can be improved. Additionally or alternatively, in one embodiment - in a step b1) start kinematics data indicating the robot start pose and / or at least one time derivative thereof, in particular a robot start speed and / or acceleration, are provided; and / or - in a step b2) updated kinematics data indicating a current robot pose and / or at least one time derivative thereof,in particular a current robot speed and / or acceleration. In a further development of this embodiment, - in step c1) the first robot target pose is predicted by means of data processing on the basis of the provided start kinematics data. Additionally or alternatively, in a further development 2022P00048WO 8 / 41 Kuka Deutschland GmbH - in step c2) the updated robot target pose is predicted by means of data processing on the basis of the provided updated kinematics data, in a further development on the basis of a, in particular chronological, group or history of at least some of the previously provided kinematics data; or - in step c3) the new robot target pose, in a further development the sequence, by means of data processing on the basis of the provided updated kinematics data, in a further development on the basis of a, in particular chronological,Group or history of at least some of the previously provided kinematic data is predicted. This can improve the accuracy and / or reliability of the prediction in one embodiment. Additionally or alternatively, in a further development of the embodiment in which start or updated kinematic data is provided, - in step d1) the robot target movement is determined on the basis of the provided start kinematic data. Additionally or alternatively, in a further development - in step d2) the robot target movement is updated on the basis of the provided updated kinematic data, in particular on the basis of a group or history, in particular chronological, of at least some of the previously provided kinematic data; or - in step d3) the new robot target movement is determined on the basis of the provided updated kinematic data, in particular on the basis of a group or history, in particular chronological,Group or history of at least some of the previously provided kinematic data is determined. In one embodiment of at least one of the aforementioned developments, steps b2), d2), and e2) are repeated several times before re-executing step c2; in one embodiment of steps a2) and c2), - updated kinematic data indicating the current robot pose and / or at least a time derivative thereof are provided (step b2)); 2022P00048WO 9 / 41 Kuka Deutschland GmbH - the robot target movement is updated based on the most recently updated target pose and also the kinematic data provided (in particular compared to this most recently updated target pose), in particular also based on a, in particular chronological, group of the previously provided kinematic data (step d2)); and - (the) drives of the robot are controlled to execute this updated robot target movement (step e2)).before again - a (new) updated robot target pose is predicted by means of data processing based on provided new image data (step c2)); - and if necessary, these new image data of an environment of the robot, assigned to a (now) current robot pose, are provided for this purpose during the execution of the last updated robot target movement (step a2)). Analogously, in an embodiment of at least one of the aforementioned developments, steps b2), d3) and e3) are repeated several times before step c3) is executed again in an embodiment of steps a3) and c3). In other words, in one embodiment, the robot target movement is compared based on new kinematic data of the robot with an update of the robot target pose based on new image data, if necessary also with a provision of the new image data,updated at a higher clock rate or higher frequency and used to control the robot's drives, or the new robot target movement is determined at a higher clock rate or higher frequency based on new kinematic data of the robot compared to a determination of the new robot target pose based on new image data, possibly also compared to a provision of the new image data, and used to control the robot's drives. Of course, new image data can also be provided during the higher-frequency update of the robot target movement or determination of the new robot target movement, but preferably the robot target pose is updated or the new robot target pose is determined, in a further development, the sequence is determined,preferably at a lower frequency compared to the updating of the robot target movement or the determination of the new robot target movement. 2022P00048WO 10 / 41 Kuka Deutschland GmbH In one embodiment, this allows an advantageous, for example smoother, robot target movement to be executed and / or the execution or control of the robot target movement, in particular its precision, speed and / or reliability, to be improved. In one embodiment, - in step c1) the first robot target pose is predicted by means of data processing on the basis of, in particular, previously specified environmental data; and / or - in step c2) the updated robot target pose is predicted by means of data processing on the basis of, in particular, previously specified environmental data, preferably the environmental data used in step c1), or in step c3) the new robot target pose, in a further development the sequence, by means of data processing on the basis of,in particular, predefined environmental data, preferably the environmental data used in step c1). Such predefined environmental data can in particular comprise geometries of the environment, in particular geometries and / or poses of workpieces to be approached or transported, obstacles to be bypassed or avoided, or the like. In one embodiment, this can improve the accuracy and / or reliability of the forecast. Additionally or alternatively, in one embodiment - in step c1), the first robot target pose is predicted by means of data processing on the basis of environmental data, preferably in or for the start pose, recorded by sensors; and / or - in step c2), the updated robot target pose is predicted by means of data processing on the basis of environmental data, preferably in or for the current robot pose, recorded by sensors, in particular on the basis of a, in particular chronological,Group or history of at least some of the previously sensor-recorded environmental data, predicts or, in step c3), the new robot target pose, in a further development, the sequence, predicts by means of the 2022P00048WO 11 / 41 Kuka Deutschland GmbH data processing on the basis of environmental data recorded by sensors, preferably in or for the current robot pose, in particular on the basis of a group or history, in particular chronological, of at least some of the previously sensor-recorded environmental data. Such environmental data recorded by sensors can in particular include poses of workpieces to be approached or transported and / or obstacles to be avoided or avoided and / or joint and / or drive loads of the robot and / or audio data or the like. Accordingly, the sensors for (sensor-based) recording of the environmental data can in particular be force and / or torque sensors, distance sensors, radar sensors,Microphones and the like, whereby a combination of two or more sensors or a combination of environmental data acquired by different sensors (sensor-based) can be particularly advantageous. In one embodiment, this can improve the accuracy and / or reliability of the forecast. In one embodiment, one or more of the robot target movements (each) are determined taking into account constraints that can be parameterized or parameterized in a further development, in one embodiment one or more, preferably parameterized or parameterized, safety areas and / or collision avoidance, in particular self- and / or external collision avoidance,determined. This can improve the operation of the robot in one embodiment. Additionally or alternatively, in one embodiment, one or more of the robot target movements are determined (each) based on a numerical kinematic model of the robot. This can allow advantageous path planning algorithms to be used in one embodiment. 2022P00048WO 12 / 41 Kuka Deutschland GmbH Additionally or alternatively, in one embodiment, one or more of the robot target movements are determined (each) using data processing that is also at least partially based on machine learning, in a further development using Gaussian processes, Gaussian mixture models, Bayesian learning and / or at least one additional artificial neural network,determined. This allows advantageous movements to be executed in one embodiment. Additionally or alternatively, in one embodiment, one or more of the robot target movements (each) comprise movement commands in an axis or joint coordinate space, in particular axis or joint angle space, of the robot, wherein in an advantageous further development, hardware limits and / or (self-)collisions and / or other restrictions such as safety areas are taken into account. This allows movements to be advantageously determined or controlled in one embodiment and / or the determination and / or control can be improved, in particular simplified and / or accelerated. Additionally or alternatively, in one embodiment, the robot target movements determined in step d1) and / or one or more of the robot target movements updated in step d2) (each) comprise a displacement of an end effector of the robot by at least 10 cm, preferably by at least 25 cm,in particular by at least 50 cm. As explained elsewhere, in one embodiment, the robot can be moved reactively or by updating the target pose over large parts of a free workspace, for example, half a meter or a whole meter. This is based in particular on the realization that although the image data often vary significantly more for large distances or movement sections than for small(er) movements near the (final) target pose, the reactive control ensures convergence of the target pose prediction or target pose-predicting data processing. In one embodiment, the provided start kinematics data and / or the updated kinematics data (each) comprise a position, preferably one-, two-, or three-dimensional, and / or an orientation, preferably one-, two-, or three-dimensional, of an end effector of the robot, in particular corresponding, preferably one-,two- or three-dimensional, position and / or, preferably one-, two- or three-dimensional, orientation data, hereinafter referred to as x without restriction of generality, t . Additionally or alternatively, in one embodiment, the provided initial kinematic data and / or the updated kinematic data (each) comprise joint positions of the robot, in particular corresponding joint position data or coordinates, the dimension of which preferably corresponds to the number of degrees of freedom or joints or (movement) axes of the robot, hereinafter referred to as q without restriction of generality tAdditionally or alternatively, in one embodiment, the provided initial kinematics data and / or the updated kinematics data (each) comprise first and / or higher time derivatives of the aforementioned variables, in particular translational and / or rotational velocities and / or accelerations of the end effector and / or the joints. In one embodiment, the first robot target pose and / or the updated or new robot target pose(s), in a further development the robot target poses of the sequence, (each) comprise a, preferably one-, two-, or three-dimensional, position and / or a, preferably one-, two-, or three-dimensional, orientation of one or the end effector of the robot, in particular corresponding, preferably one-, two-, or three-dimensional, position and / or, preferably one-, two-, or three-dimensional, orientation data, hereinafter referred to as x* or x without limitation of generality. i or x i+1, …. Additionally or alternatively, in one embodiment, the first robot target pose and / or the updated or new robot target pose(s), in a further development the robot target poses of the sequence, (each) comprise joint positions of the robot, in particular corresponding joint position data or coordinates, the dimension of which preferably corresponds to the number of degrees of freedom or joints or (movement) axes of the robot, hereinafter referred to as q* or qi or qi+1 without loss of generality, …. 2022P00048WO 14 / 41 Kuka Deutschland GmbH By using position data and / or orientation data of the end effector, particularly preferably at least as robot target pose(s), robot target movements can be determined particularly advantageously, and in particular advantageous algorithms for determining corresponding robot target movements can be used. Additionally or alternatively, these are particularly well suited for forecasting by artificial intelligence orat least partially based on machine learning data processing. By using joint position (sdata / scoordinates) of the robot, preferably at least as kinematic data for predicting robot target poses by an artificial intelligence or at least partially based on machine learning data processing or as (part of) an input of such a target pose-predicting artificial intelligence or data processing, robot target poses can be determined particularly advantageously; they are particularly well suited as (part of) an input of such a target pose-predicting artificial intelligence or data processing. Accordingly, this or a prediction of a first or updated or new robot target pose, which includes position (sdata) and / or orientation (sdata) x* of the end effector, by means of data processing based also on start or updated or new kinematic data, whichJoint position (sdat / scoordinate)s qt of the robot comprise (x* = KI(image data, qt))), each represent particularly preferred embodiments, without the invention being limited thereto. Accordingly, it may also be expedient, for example, to use position (sdat) and / or orientation (sdat) of the end effector as kinematic data or (part of) an input of a target pose-predicting artificial intelligence or data processing and / or to use joint position (sdat / scoordinate)s of the robot as a target pose or input for determining the robot's target movements. In one embodiment, the robot start pose and / or the (respective) current robot pose is a position of the robot, in particular its joints or (motion) axes. Accordingly, in one embodiment, the start image data is assigned to a start position of the robot and / or the current robot pose is (respectively) assigned to a current position of the robot. 2022P00048WO 15 / 41 Kuka Deutschland GmbH In one versionthe robot start pose and / or the (respective) current robot pose a, preferably one-, two-, or three-dimensional, position and / or, preferably one-, two-, or three-dimensional, orientation of its end effector. Accordingly, in one embodiment, the start image data are assigned to a start position and / or orientation of the end effector and / or the current robot pose is (respectively) assigned to a current position and / or orientation of the robot's end effector. As also explained elsewhere, in several steps b2), updated kinematic data can be provided, which indicate a (respective) current robot pose and / or at least a time derivative thereof. In one embodiment, in several steps d2), the robot target movement is determined on the basis of the target pose updated last or in the last step c2) and also on the basis of the kinematic data updated last or in the last step b2).updated, before in a new step c2) on the basis of kinematic data provided in a preceding step b2), which are preferably also used to update the robot's target movement, the target pose is updated again and thus at a lower frequency than the robot's target movement. Analogously, in one embodiment, in several steps d3), the new robot's target movement is updated on the basis of the new target pose predicted most recently or in the most recently performed step c3) and also on the basis of the kinematic data updated most recently or in the most recently performed step b2), before in a new step c3) on the basis of kinematic data provided in a preceding step b2), which are preferably also used to determine the new robot's target movement, the new target pose is predicted and thus determined at a lower frequency than the new robot's target movement. In one embodiment, the twoThe sub-problems "prediction / updating the target pose" and "determining the target movement" are solved at different frequencies and thus particularly advantageously, preferably in parallel and / or synchronized. In one embodiment, by means of data processing based on the image data and, if applicable, kinematic and / or environmental data 2022P00048WO 16 / 41 Kuka Deutschland GmbH - in step c1) at least one, preferably first, time derivative of the first robot target pose is additionally predicted and, in step d1), the robot target movement is determined based on this time derivative(s); and / or - in step c2) at least one, preferably first, (updated) time derivative of the updated robot target pose is additionally predicted and, in step d2), the robot target movement is updated based on this (updated) time derivative(s), or, in step c3), at least one, preferably first, (new) time derivative of the new robot target pose is additionally determined.In a further development, at least one, preferably first, time derivative of the robot target poses of the sequence is predicted, and in step d3) the robot target movement is determined on the basis of this (new) time derivative(s). By predicting (also) preferably first, time derivatives of the target pose(s), a particularly advantageous robot target movement can be realized in one embodiment. A time derivative of a robot target pose can in particular include, in particular be, a translational and / or rotational speed of the end effector and / or joints or axes of the robot. In one embodiment, in a further development, the data processing is trained according to a method described here. In particular, the method for operating a robot according to an embodiment of the present invention can therefore use a method described here for training one or the at least partially machine learning-basedData processing for predicting robot target poses based on image data or the steps thereof. According to one embodiment of the present invention, in a method for training a data processing system or the data processing system based at least partially on machine learning for predicting robot target poses based on image data for operating the robot as described here, the training comprises the steps of: - successively performing movements of the robot or a demonstrator, in particular a loose end effector, end effector dummy, pointing device or the like, in each case - from a start pose to reach a, preferably into a, target pose; or 2022P00048WO 17 / 41 Kuka Deutschland GmbH - from a target pose to reach a, preferably into a, start pose with - different movement trajectories; and / or - under different environmental conditions; and / or - at least once from a start pose to reach a target poseand at least once from a target pose to reach a start pose, preferably several times from a start pose to reach a target pose, preferably from different start poses to reach the same or different target pose(s), and / or several times from a target pose to reach a start pose, preferably from the same or different target pose(s) to reach different start poses, wherein for one or more of these movements, for several poses of the robot or demonstrator assumed during the (respective) movement, - image data of a, preferably the, environment of the robot associated with this pose; and - the respective target pose are collected for machine learning of the prediction; and - training the data processing on the basis of these collected, mutually associated image data and target poses. In one embodiment, the demonstrator has a, preferably integrated, sensor system, with which, in a further development, poses and, in an embodiment(assigned) time stamps are recorded. Additionally or alternatively, the pose of the demonstrator and, in one embodiment, (assigned) time stamps can be recorded using external sensors, in particular a tracking system or the like. One embodiment of the present invention is based on the idea of ​​continuously collecting image data during (the execution of) such movements and using it to train the (target pose) artificial intelligence or target pose-predicting data processing. In one embodiment, the training can be carried out at least partially during the execution of (further) movements to collect (further) image data and target poses, which can improve and, in particular, accelerate training. 2022P00048WO 18 / 41 Kuka Deutschland GmbH One or more movements from a target pose to a start pose or opposite to a path direction running during normal operation can, in particular, enable an approach to a target pose inimprove the inventive operation of the robot in the regular path direction and / or control of the training movements, in particular simplify these and / or increase precision. In one embodiment, the start pose and / or the target pose are specified for one or more of the movements, preferably entered by means of an operator input (separately or in addition to assuming the pose). Additionally or alternatively, in one embodiment, for one or more of these movements, for several of the poses assumed by the robot or demonstrator during this movement, kinematic data which indicate this robot pose and / or at least a time derivative thereof are collected and assigned to the corresponding image data, preferably chronologically, in one embodiment by corresponding time stamps or the like, and the data processing is trained on the basis of this collected kinematic data, which is preferably chronologically assigned to the corresponding poses or image data.Additionally or alternatively, in one embodiment, environmental data is specified for one or more of these movements; in a further development as already mentioned elsewhere, geometries of the environment, in particular geometries and / or poses of workpieces to be approached or transported, obstacles to be avoided or avoided, or the like, and the data processing is also trained on the basis of this specified environmental data. Additionally or alternatively, in one embodiment, during one or more of these movements, preferably in or for one or more poses assumed during this movement, environmental data is recorded by sensors; in a further development as already mentioned elsewhere, poses of workpieces to be approached or transported and / or poses of obstacles to be avoided or avoided and / or joint and / or drive loads of the robot and / or audio data or the like, and the data processing is also trained on the basis of this(Sensor-) acquired environmental data, preferably temporally associated with the corresponding poses or image data. 2022P00048WO 19 / 41 Kuka Deutschland GmbH This can improve the accuracy and / or reliability of the prediction in one embodiment. In one embodiment, the starting poses are identical for two or more of the movements used or utilized to train the data processing. Additionally or alternatively, in one embodiment, the target poses are identical for two or more of the movements used or utilized to train the data processing. In one embodiment, the accuracy and / or reliability of the prediction can be improved by the different movement trajectories and / or different environmental conditions. Additionally or alternatively, in one embodiment, the starting poses (from each other) are different for two or more of the movements used or utilized to train the data processing.or alternatively, in one embodiment, the target poses are different (from each other) for two or more of the movements used or utilized to train the data processing. This can improve the accuracy and / or reliability of the prediction in one embodiment. In one embodiment, the starting poses and the target poses are identical and identical, but the movements in between are different (from each other). In one embodiment, the accuracy and / or reliability of the prediction can be improved by the different movement trajectories and / or different environmental conditions. In one embodiment, the robot or demonstrator reaches the respective target pose in one or more of the movements performed. In other words, positive examples are used to train the data processing in one embodiment. 2022P00048WO 20 / 41 KukaGermany GmbH Additionally or alternatively, in one embodiment, the robot or demonstrator does not reach the respective specified target pose during one or more of the movements performed. In other words, in one embodiment, negative examples are (also) used to train the data processing. This can improve the accuracy and / or reliability of the prediction in one embodiment. In one embodiment, the robot or demonstrator is hand-guided during one or more of the movements performed and used to train the data processing by manually exerted forces; in a further development, the robot is controlled accordingly. It can be advantageous to limit the hand guidance by control technology, preferably to specified joint angle limits of the robot and / or orientation and / or position limits of the end effector. This can make these movements veryintuitively, precisely and / or even without expert knowledge. Additionally or alternatively, in one embodiment, the robot is gesture-controlled or teleoperates for one or more of the movements performed. As a result, in one embodiment, these movements can be performed quickly(er). Additionally or alternatively, the robot performs one or more of the movements used or utilized for training the data processing automatically, in a further training - with the aid of automated path planning based on environmental recognition; and / or - with the aid of a, in particular automated, variation - of a movement previously guided or controlled by a user; or - an automatically planned path; and / or 2022P00048WO 21 / 41 Kuka Deutschland GmbH in a work assignment, in a further training in a productive or production operation. Through the, in particular additional, use (also) of automated movements, theTraining is improved, in particular by using many movements, thereby improving the accuracy and / or reliability of the forecast. If movements performed automatically during a work assignment are used to train the data processing, the data processing is advantageously (also) (further) trained during the assignment. In one embodiment, a duplicate of the forecast is further trained on the basis of the movements performed automatically during a work assignment, in a further training in a productive or production company, or the data collected during this. It is checked whether this further trained duplicate achieves a better result than the forecast used in the work assignment. In one embodiment, a user decides whether the further trained duplicate will be used as the new forecast for further work assignments instead of the previously used forecast. In one embodiment, in a further training, the start and / or theNew image data is generated using at least one camera guided by the robot and / or using at least one environment-side camera and / or using image processing. In one embodiment, providing image data in step a1) and / or in step a2) or a3) comprises such generation. In one embodiment, one or more robots are (each) equipped with at least one camera, and at least one further robot moves to the corresponding pose. Robot-guided cameras can, in particular, make it possible to use more significant perspectives, while environment-side cameras can advantageously make it possible to use calmer and / or more global perspectives, whereby an environment-side camera can itself be fixed in the environment or can be moved and / or rotated by a movement device separate from the robot. Image processing can generate image data that is particularly advantageous for data processing. In one embodiment, aPredicting a target pose means predicting one or more target poses that have the highest probability. In other words, in one embodiment, the data processing determines for each of the various target poses a probability that it is an actual (desired) target pose and selects the one with the highest probability as the predicted target pose. Additionally or alternatively, in one embodiment, predicting a target pose comprises outputting a target pose based on an input, wherein this input comprises image data, and in a further development, additionally kinematic data and / or environmental data. According to one embodiment of the present invention, a system, in particular hardware and / or software, in particular program technology, is configured to carry out a method described here. According to an embodiment of theThe present invention comprises a system or the system: - means for providing start image data of an environment of the robot, associated with a robot start pose, in particular comprising at least one camera and / or image processing; - means for predicting a first robot target pose by means of data processing based at least partially on machine learning, in particular by means of a regression method and / or at least one artificial neural network, on the basis of the provided start image data, in particular comprising the data processing; - means for determining a robot target movement based on the first target pose; and - means for controlling drives of the robot to execute the robot target movement; and - means for subsequently providing, repeatedly, new image data of an environment of the robot, associated with a respective current robot pose, in a further development during the execution of theRobot target movement, in particular comprising the at least one camera and / or image processing; - means for predicting an updated or new robot target pose, in a further development of the sequence, by means of data processing based on the new image data provided, in particular comprising the data processing; - means for updating the robot target movement based on the updated target pose or for determining the new robot target movement based on the new target pose or sequence; and - means for controlling drives of the robot to execute the updated or new robot target movement. According to one embodiment of the present invention, a or the system additionally or alternatively comprises: - means for, when sequentially performing movements of the robot or a demonstrator, in each case - from a start pose to reaching a target pose; or - from a target pose to reachinga starting pose with different movement trajectories and / or under different environmental conditions and / or at least once from a starting pose to reach a target pose and at least once from a target pose to reach a starting pose, for at least one of these movements for several poses of the robot or demonstrator assumed during this movement, collecting: - image data of an environment associated with this pose; and - the respective target pose, in a further development of the sequence, for machine learning of the forecast, in particular a correspondingly (filled) data memory or a corresponding data volume; and - means for training the data processing based on these collected, mutually associated image data and target poses. A system and / or means within the meaning of the present invention can be implemented in hardware and / or software, in particular at least one, preferably with a memory and / or bus systemdata- or signal-connected, in particular digital, processing, in particular a microprocessor unit (CPU), graphics card (GPU), or the like, and / or one or more programs or program modules. The processing unit can be designed to execute commands implemented as a program stored in a 2022P00048WO 24 / 41 Kuka Deutschland GmbH storage system, to acquire input signals from a data bus, and / or to output output signals to a data bus. A storage system can have one or more, in particular different, storage media, in particular optical, magnetic, solid-state, and / or other non-volatile media. The program can be designed such that it embodies or is capable of executing the methods described here, so that the processing unit can execute the steps of such methods and thus, in particular, operate the robot or train the data processing. AIn one embodiment, a computer program product can comprise, in particular be, a storage medium, in particular a computer-readable and / or non-volatile one, for storing a program or instructions or with a program or instructions stored thereon. In one embodiment, execution of this program or these instructions by a system or a controller, in particular a computer or an arrangement of multiple computers, causes the system or the controller, in particular the computer(s), to execute a method described here or one or more of its steps, or the program or the instructions are configured to do so. In one embodiment, one or more, in particular all, steps of a method described here are fully or partially computer-implemented or one or more, in particular all, steps of the method are carried out fully or partially automatically.in particular by the system or its means. In one embodiment, the system comprises the robot and / or the data processing and / or the at least one camera and / or image processing. In one embodiment, the robot preferably moves continuously during several successively performed steps e2) or e3). In other words, in one embodiment, drives of the robot are continuously controlled to execute the robot's target movement, while this is continuously updated on the basis of a target pose that is continuously updated by means of artificial intelligence or data processing, or at least one new target pose is predicted and a path section to this is planned while approaching a last determined target pose. 2022P00048WO 25 / 41 Kuka Deutschland GmbH. Further advantages and features emerge from the subclaims and the exemplary embodiments. In this regard, the following shows, partly schematically: Fig. 1: a system according to an embodiment of thepresent invention; Fig. 2: a method according to an embodiment of the present invention; and Fig. 3: an information flow in the system or method according to an embodiment of the present invention. Fig. 1 shows a system according to an embodiment of the present invention, which comprises a robot with a controller 1 and a robot arm 10 with an end effector 11, as well as at least one camera 20 guided by the robot and / or at least one environment-side camera 21. In modifications not shown, the camera 20 or 21 can be omitted and / or additional cameras guided by the robot and / or environment-side cameras can be provided and / or the robot arm can have a different configuration, for example, more or fewer than the six joints or (movement) axes shown. By way of example, in Fig. 1, a robot start pose is shown in solid lines and a robot target pose is shown in dashed lines, while a dashed-dotted line represents a movement of the robot.based on a "LIN" movement command, which causes a straight line of a TCP of the robot indicated in Fig. 1 by coordinate systems in Cartesian space, a dash-double-dotted line illustrates a movement of the robot based on a "CIRC" movement command, which causes a circular path of the TCP in Cartesian space, and a double-dash-dotted line illustrates a movement of the robot based on a "PTP" movement command. Fig. 2 shows a method according to an embodiment of the present invention. To train a data processing system based at least partially on machine learning for predicting robot target poses based on 2022P00048WO 26 / 41 Kuka Deutschland GmbH image data, environmental data, for example known geometries of workpieces to be handled by the robot 10 and / or obstacles to be avoided or the like, are specified in a step S10. Then, in step S20, a movement of the robot 10 or a demonstrator, for example only theloose end effector 11, from a starting pose to reach a target pose. Additionally or alternatively, movements from a target pose to reach a starting pose can also be performed. The movement is carried out with different movement trajectories (cf. Fig. 1) and / or under different environmental conditions. For these movements, in step S20, for several poses assumed by the robot or demonstrator during the respective movement, image data of an environment 30 associated with this pose and the respective target pose are collected for machine learning of the forecasting. In a preferred development, kinematic data indicating the respective robot pose and / or sensor-detected environmental data, for example sensor data from force-torque sensors, joint torque sensors, radar sensors,Audio microphones, tracking systems 22 for detecting obstacles 31 or the like. If sufficient data has been collected (S25: "Y"), in a step S30 the data processing is trained on the basis of these collected, mutually associated image data and target poses and, if applicable, kinematic and / or environmental data. If appropriate, the training can also take place at least partially in parallel with the collection of further data based on further movements performed. To operate the robot 10, in a step S100 environmental data, for example known geometries of workpieces or the like to be handled by the robot 10, are specified; in a step S110 start 2022P00048WO 27 / 41 Kuka Deutschland GmbH kinematic data, which indicate the robot start pose, are provided; in a step S120 start image data of an environment 30 of the robot 10, associated with this robot start pose, as well as sensor-detected environmental data, for example sensor data fromForce-torque sensors, joint torque sensors, radar sensors, audio microphones, tracking systems 22 for detecting obstacles 31 or the like (not shown) are provided, in a step S130 a first robot target pose is predicted by means of the trained data processing on the basis of the provided environmental data, start image data and start kinematic data, in a step S140 a robot target movement is determined on the basis of this first target pose, and in a step S150 drives of the robot 10 are controlled to execute the robot target movement, of which three drives are provided with the reference numeral 12 as an example. During this control in step S150, analogous to step S110, multiple current kinematic data are provided, analogous to step S140 the robot target movement is updated on the basis of the first target pose and this current kinematic data, and then the drives 12 of the robot 10 are controlled to execute this updatedThe robot target movement is controlled instead of the previously executed robot target movement. While the robot target movement continues to be executed, in a step S210, as previously in step S110, updated kinematic data indicating the current robot pose are provided. In a step S220, as previously in step S120, new image data of the environment of the robot 10 associated with this current robot pose, as well as sensor-detected environmental data, are provided. In a step S230, as previously in step S130, an updated robot target pose is predicted using the trained data processing based on the provided environmental data, new image data, and updated kinematic data. In a step S240, the robot target movement is updated based on this updated target pose. And in a step S250, as previously in step S150, the drives 12 of the robot 10 are now controlled to execute this updated robot target movement. Even duringDuring this control in step S250, current kinematic data are provided multiple times, analogous to step S250 or S210, analogous to step S240, the robot target movement is updated based on the target pose updated in step S230 and 2022P00048WO 28 / 41 Kuka Deutschland GmbH this current kinematic data, and then the drives 12 of the robot 10 are controlled to execute this updated robot target movement. As long as no termination condition is met (S255: "N"), for example, the robot 10 has reached a workpiece or the like, the system or method returns to step S210; otherwise (S255: "Y"), the method is terminated (step S260). The information flow in the system or method shown in Fig.3 illustrates that by means of the data processing based at least partly on machine learning on the basis of the image data (“image”) and kinematic data (“qt”) a target pose (“x*”) is predicted and on the basis of this predicted target poseand updated kinematic data (“x t = F^(q t)”) a robot target movement is determined or updated, or corresponding movement commands (“q*”) for controlling the drives 12 of the robot 10 are determined. In a modification, which can preferably be implemented in addition to or particularly preferably as an alternative to the above-explained aspect of predicting an updated robot target pose, updating the robot target movement on the basis of the updated target pose, and controlling drives of the robot to execute the updated robot target movement, not only or not the final target or end pose is predicted, but additionally or alternatively (in each case) one or more new robot target pose(s) are predicted on the basis of the new image data assigned to a current robot pose. As already described above, in steps S10,S20 Environmental data is specified and movement of the robot 10 or a demonstrator is carried out, and image data of the environment 30 and the respective target pose are collected for machine learning of the prediction, wherein the target pose is not the final target or end pose, but rather the target pose approached in a subsequent time step, or a sequence with several consecutive such target poses is collected. Here, too, kinematic data indicating the respective robot pose, which is assigned to these poses or image data, and / or environmental data acquired by sensors, for example, sensor data from force-torque sensors, joint torque sensors, radar sensors, audio microphones, 2022P00048WO 29 / 41 Kuka Deutschland GmbH tracking systems 22 for detecting obstacles 31, or the like, can be temporally assigned to these poses or image data, preferably via time stamps or the like.are collected. In step S30, the or, if applicable, further data processing is trained on the basis of these collected, associated image data and target poses, and, if applicable, kinematic and / or environmental data, however, not to predict the final target or end pose, but rather the target pose(s) to be approached in the next time step(s). To operate the robot 10, environmental data, for example, known geometries of workpieces or the like to be handled by the robot 10, are specified in step S100; starting kinematic data indicating the robot starting pose are provided in a step S110; starting image data of an environment 30 of the robot 10 associated with this robot starting pose are provided in a step S120; as well as sensor-detected environmental data, for example, sensor data from force-torque sensors, joint torque sensors, radar sensors, audio microphones,Tracking systems 22 for detecting obstacles 31 or the like (not shown) are provided; in a step S130, a first robot target pose is predicted by means of the trained data processing based on the provided environmental data, start image data, and start kinematic data; in a step S140, a robot target movement is determined based on this first target pose; and in a step S150, drives of the robot 10 are controlled to execute the robot target movement, of which three drives are provided with the reference numeral 12 by way of example. After or preferably already during this control in step S150, in a step S210, as previously in step S110, updated kinematic data indicating the current robot pose are provided; in a step S220, as previously in step S120, new image data of the environment of the robot 10 associated with this current robot pose, as well as sensor-detected environmental data, are provided; in a step S230,As previously in step S130, a new robot target pose to be approached in the next time step or several consecutive such robot target poses (to be approached in consecutive time steps) are predicted using the trained data processing based on the provided environmental data, new image data, and updated kinematics data; in a step S240, as previously in step S140, a new robot target movement is determined based on these new target pose(s); and in a step S250, as previously in step S150, the drives 12 of the robot 10 are controlled to execute this new robot target movement. Also during this control in step S250, analogous to step 150 or S210, current kinematics data are provided multiple times.Analogous to step S240, a new robot target movement is determined based on the new target pose determined in step S230 and this current kinematic data, and then the drives 12 of the robot 10 are controlled to execute this new robot target movement. As long as no termination condition is met (S255: "N"), for example, the new target pose deviates only (sufficiently) slightly from the last determined target pose or the final or end pose is reached, or the like, the system or method returns to step 210; otherwise (S255: "Y"), the method is terminated (step S260). Advantageously, upon reaching the final or end pose, the artificial intelligence or data processing predicts the current pose as the new target pose, so that the robot stops automatically. The information flow shown in Fig. 3 also illustrates,that by means of the data processing based at least partially on machine learning on the basis of the image data ("image") and kinematic data ("qt"), a new target pose ("x*") is determined in each case, and on the basis of this predicted target pose and updated kinematic data ("xt = F^(qt)"), a new robot target movement is determined or corresponding movement commands ("q*") for controlling the drives 12 of the robot 10 are determined. In the present disclosure, "has an X" generally does not imply an exhaustive list, but is a shortened form of "has at least one X" and also includes "has two or more X" and "has Y in addition to X." Although exemplary embodiments have been explained in the preceding description, it should be noted that a multitude of modifications are possible. In particular, in Fig. 3, the function x* = f(image, q, t) can also be replaced by one of the following variants with other inputs: 2022P00048WO 31 / 41 Kuka Deutschland GmbH x* = f(image, xt); or x* = f(image, qt, xt); or x* = f(image); or q* = f(image, x t ); or q* = f(image, q t , x t ); or q* = f(image). In addition, time derivatives of the target poses can also be predicted and taken into account when determining the new robot target movements, purely exemplarily in the form dx* / dt = f'(image, q t ), dq* / dt = f'(image, q t ), dx* / dt = f'(image, x t ), dq* / dt = f'(image, x t ), dx* / dt = f'(image, q t , x t ), dq* / dt = f'(image, q t , x t ), dx* / dt = f'(image), dq* / dt = f'(image) or the like. Additionally or alternatively, the kinematic data used can also have time derivatives (dqt / dt, dxt / dt, d 2 qt / dt 2 , d 2 xt / dt 2,,,,). In a corresponding modification, the respective data with a time stamp is then also collected and used in the training process. Furthermore, it should be noted that the exemplary embodiments are merely examples that are not intended to limit the scope of protection, the applications, and the structure in any way. Rather, the preceding description provides the person skilled in the art with a guide for the implementation of at least one exemplary embodiment, whereby various changes, in particular with regard to the function and arrangement of the described components, can be made without departing from the scope of protection as it results from the claims and equivalent combinations of features. 2022P00048WO 32 / 41 Kuka Deutschland GmbH List of reference symbols 1 Controller 10 Robot 11 End effector 12 Drive 20, 21 Camera 22 Tracking system 30 Environment 31 Obstacle

Claims

2022P00048WO 33 / 41 Kuka Deutschland GmbH Patent claims 1. Method for operating a robot (10), comprising the steps of: a1) providing start image data of an environment of the robot assigned to a robot start pose; c1) predicting a first robot target pose by means of data processing based at least partially on machine learning, in particular by means of a regression method and / or at least one artificial neural network, on the basis of the provided start image data; d1) determining a robot target movement based on the first target pose; and e1) controlling drives of the robot to execute the robot target movement; wherein the method comprises the following, multiple repeated steps: a2) providing new image data of an environment of the robot assigned to a current robot pose during the execution of the robot target movement;c2) predicting an updated robot target pose by means of data processing based on the provided new image data; d2) updating the robot target movement based on the updated target pose; and e2) controlling drives of the robot to execute the updated robot target movement; and / or the method comprises the steps following steps a1) - e1), repeated several times: a3) providing new image data of an environment of the robot associated with a current robot pose, in particular during the execution of the robot target movement; c3) predicting a new robot target pose by means of data processing based on the provided new image data; d3) determining a new robot target movement based on the new target pose; and; 2022P00048WO 34 / 41 Kuka Deutschland GmbH e3) controlling drives of the robot to execute the new target robot movement.

2. Method according to claim 1, characterized in that: - in step c1) a sequence with the first robot target pose and at least one subsequent robot target pose is predicted by means of the data processing and in step d1) the target robot movement is determined on the basis of this sequence and / or in step c3) a sequence with the new robot target pose and at least one subsequent robot target pose is predicted by means of the data processing and in step d3) the new target robot movement is determined on the basis of this sequence; and / or that - in step c2) the updated robot target pose orin step c3) the new robot target pose, in particular the sequence, is predicted by means of the data processing on the basis of a, in particular chronological, group of the previously provided image data; and / or that - in step e2) new movement commands for controlling drives of the robot are determined several times before step c2) is carried out again; and / or that - in a step b1) start kinematics data which indicate the robot start pose and / or at least one time derivative thereof, and / or in a step b2) updated kinematics data which indicate a current robot pose and / or at least one time derivative thereof are provided and - in step c1) the first robot target pose is predicted by means of the data processing on the basis of the provided start kinematics data and / or in step c2) the updated robot target pose orin step c3) the new robot target pose, in particular the sequence, is predicted by means of data processing on the basis of the provided updated kinematic data, in particular on the basis of a, in particular chronological, group of the previously provided kinematic data; and / or. 2022P00048WO 35 / 41 Kuka Deutschland GmbH - in step d1), the robot target movement is determined on the basis of the provided start kinematics data and / or in step d2), the robot target movement is updated on the basis of the provided updated kinematics data, in particular on the basis of a group of previously provided kinematics data, in particular a chronological group, or in step d3), the new robot target movement is determined on the basis of the provided updated kinematics data, in particular on the basis of a group of previously provided kinematics data, in particular a chronological group, in particular steps b2), d2), and e2) are repeated several times before step c2) is executed again. 3.Method according to one of the preceding claims, characterized in that - in step c1), the first robot target pose is predicted by means of data processing based on predetermined environmental data; and / or - in step c2), the updated robot target pose or, in step c3), the new robot target pose, in particular the sequence, is predicted by means of data processing based on predetermined environmental data; and / or - in step c1), the first robot target pose is predicted by means of data processing based on sensor-recorded environmental data; and / or - in step c2), the updated robot target pose or, in step c3), the new robot target pose, in particular the sequence, is predicted by means of data processing based on sensor-recorded environmental data, in particular based on a group, in particular a chronological group, of the previously sensor-recorded environmental data.Method according to one of the preceding claims, characterized in that - at least one of the robot target movements is determined taking into account, in particular parameterizable, constraints, in particular at least one safety area and / or collision avoidance; and / or. 2022P00048WO 36 / 41 Kuka Deutschland GmbH - at least one of the robot target movements is determined on the basis of a numerical kinematics model of the robot; and / or - at least one of the robot target movements is determined by means of data processing based at least partially on machine learning; and / or - at least one of the robot target movements comprises movement commands in an axis coordinate space of the robot; and / or - at least one of the robot target movements comprises a displacement of an end effector of the robot by at least 10 cm; and / or - the provided start kinematics data and / or updated kinematics data comprise a position and / or orientation of an end effector of the robot and / or at least a time derivative thereof and / or joint positions of the robot and / or at least a time derivative thereof; and / or - the first robot target pose and / or the updated robot target pose orthe new robot target pose, in particular the robot target poses of the sequence, each comprise a position and / or orientation of an end effector of the robot and / or joint positions of the robot; and / or - in step c1), at least one time derivative of the first robot target pose is additionally predicted by means of the data processing, and in step d1), the robot target movement is also determined based on this time derivative, and / or in step c2), at least one time derivative of the updated robot target pose is additionally predicted by means of the data processing, and in step d2), the robot target movement is also updated based on this time derivative, or in step c3), at least one time derivative of the new robot target pose, in particular time derivatives of the robot target poses of the sequence, is additionally predicted by means of the data processing, and in step d3), the robot target movement is also determined based on this time derivative(s). 5.Method according to one of the preceding claims, characterized in that the data processing is trained, in particular is carried out, according to a method according to one of the following claims. 2022P00048WO 37 / 41 Kuka Deutschland GmbH 6. Method for training a data processing system based at least partially on machine learning for predicting robot target poses on the basis of image data for operating the robot according to a method according to one of the preceding claims, wherein the training comprises the steps of: - successively carrying out movements of the robot or of a demonstrator in each case - from a start pose to reach a target pose; or - from a target pose to reach a start pose with different movement trajectories and / or under different environmental conditions and / or at least once from a start pose to reach a target pose and at least once from a target pose to reach a start pose, wherein for at least one of these movements for a plurality of poses of the robot or of the demonstrator assumed during this movement.Demonstrators each - image data of an environment assigned to this pose; and - the respective target pose, in particular the sequence of the respective target poses, are collected for machine learning of the forecasting; and - training the data processing on the basis of these collected, mutually assigned image data and target poses.

7. Method according to the preceding claim, characterized in that - for at least one of these movements - the start pose; and / or - the target pose; are specified; and / or - for at least one of these movements, for several of the poses of the robot orDemonstrator's kinematic data, which indicate this robot pose and / or at least a time derivative thereof, are collected and assigned to the corresponding image data or poses, and the data processing is trained on the basis of this collected kinematic data; and / or - environmental data is specified for at least one of these movements and / or environmental data is recorded by sensors for at least one of these movements. 2022P00048WO 38 / 41 Kuka Deutschland GmbH and the data processing is trained on the basis of this specified and / or recorded environmental data; and / or - the start poses are identical for at least two of these movements; and / or - the target poses are identical for at least two of these movements; and / or - the start poses are different for at least two of these movements; and / or - the target poses are different for at least two of these movements; and / or - the start poses are identical and the target poses are identical for at least two of these movements and the movements in between are different; and / or - the robot or demonstrator reaches the respective target pose for at least one of the movements carried out; and / or - the robot or demonstrator does not reach the respective specified target pose for at least one of the movements carried out; and / or - the robot orDemonstrator is hand-guided by forces exerted on it during at least one of the movements performed; and / or - the robot is gesture-controlled or teleoperated during at least one of the movements performed; and / or - the robot performs at least one of the movements performed automatically, in particular with the aid of automated path planning based on environmental recognition and / or with the aid of varying a movement previously guided or controlled by a user or automatically planned path and / or in a work application.

8. Method according to one of the preceding claims, characterized in that image data are generated with the aid of at least one camera guided by the robot and / or with the aid of at least one camera on the environment side and / or with the aid of image processing, in particular providing image data comprises such generation. 2022P00048WO 39 / 41 Kuka Deutschland GmbH 9. Method according to one of the preceding claims, characterized in that predicting a target pose comprises predicting one of several target poses that has the highest probability, and / or outputting a target pose based on an input that includes image data, in particular additionally kinematic data and / or environmental data.

10. System for operating a robot and / or training a data processing system based at least partially on machine learning, which is configured to carry out a method according to one of the preceding claims and / or comprises: - means for providing start image data of an environment of the robot that are assigned to a robot start pose, in particular comprising at least one camera and / or image processing;- Means for predicting a first robot target pose by means of data processing based at least partially on machine learning, in particular by means of a regression method and / or at least one artificial neural network, based on the provided initial image data, in particular comprising the data processing; - Means for determining a robot target movement based on the first target pose; and - Means for controlling drives of the robot to execute the robot target movement; and - Means for subsequently providing, repeatedly, new image data of an environment of the robot associated with a respective current robot pose, in particular during the execution of the robot target movement, in particular comprising the at least one camera and / or image processing;- means for predicting an updated or new robot target pose by means of data processing on the basis of the respective new image data provided, in particular comprising data processing; 2022P00048WO 40 / 41 Kuka Deutschland GmbH - means for updating the robot's target movement on the basis of the updated target pose or for determining the new robot's target movement on the basis of the new target pose; and - means for controlling drives of the robot to execute the updated or new robot's target movement; and / or comprises: - means for, when movements of the robot or of a demonstrator are carried out one after the other - from a start pose to reach a target pose; or - from a target pose to reach a start pose with different movement trajectories and / or under different environmental conditions and / or at least once from a start pose to reach a target pose and at least once from a target pose to reach a start pose, for at least one of these movements for a plurality of poses of the robot or of the demonstrator assumed during this movement.Demonstrators each collect: - image data of an environment associated with this pose; and - the respective target pose, in particular the sequence, for machine learning of the prediction; and - means for training the data processing based on these collected, associated image data and target poses.

11. A computer program or computer program product, wherein the computer program or computer program product contains instructions, in particular stored on a computer-readable and / or non-volatile storage medium, which, when executed by one or more computers or a system according to claim 10, cause the computer(s) or the system to carry out a method according to one of claims 1 to 9.

Images