Creating a robot application

Abstract

A method for creating a robot application comprises the steps of: S10: providing a robot application; S20: determining a modification of the robot application by means of at least partially machine-learning-based first data processing on the basis of a user specification; S30: carrying out a simulation of the robot application modified with the modification, by means of an integrated development environment; S40: evaluating the modification on the basis of the simulation carried out; S50: carrying out steps S20-S40 again if the evaluation satisfies an iteration criterion, wherein a new modification of the robot application is then determined in step S20 by means of the first data processing on the basis of the simulation carried out; S60: outputting the modification to a user interface if the evaluation satisfies a proposal criterion; S70: checking a user response; S80: carrying out steps S20-S70 again if the user response satisfies a rejection criterion, wherein a new modification of the robot application is then determined in step S20 by means of the first data processing on the basis of a new user specification; S90: implementing the modification if the user response satisfies an acceptance criterion; S100: carrying out steps S20-S90 again if a user input requests a further modification, wherein a new modification of the robot application is then determined in step S20 by means of the first data processing on the basis of a new user specification; and S110: outputting the robot application if the user input requests an output. The invention also relates to a method for operating a robot application, and to a system and a computer program (product).

Description

[0001] Description

[0002] Creating a robot application

[0003] The present invention relates to a method for creating a robot application, in particular for operating a robot arrangement, as well as a system or computer program or computer program product for carrying out the method.

[0004] According to internal practice, robot applications are created by users creating program code, configuring robots and other elements, and generating a scene to test the robot application using simulation. This is very time-consuming.

[0005] One idea is to use artificial intelligence (AI) to create the program code.

[0006] Furthermore, a user must review modifications to the program code suggested by the AI, which also makes the AI-supported creation of robot applications very time-consuming.

[0007] The object of the present invention is to improve the creation of robot applications, in particular the operation of robot arrangements.

[0008] This object is achieved by a method having the features of claim 1 and 8. Claims 9 and 10 represent a system or computer program or

[0009] A computer program product for carrying out a method described herein is protected. The subclaims relate to advantageous developments.

[0010] According to one embodiment of the present invention, a method for creating an application of a robot arrangement (“robot application”) comprises the following steps, preferably carried out in the order mentioned, possibly several times:

[0011] S10: Providing a robot application;

[0012] S20: Determining a modification of the robot application using a first data processing based at least partially on machine learning on the basis of a user specification;

[0013] S30: Performing a simulation of the robot application modified with the modification using an integrated development environment;

[0014] S40: Evaluate the modification based on the simulation performed;

[0015] S50: performing steps S20 - S40 again if the evaluation meets an iteration criterion, wherein in step S20 a new modification of the robot application is then determined using the first data processing on the basis of the simulation carried out;

[0016] S60: Outputting the modification to a user interface, in particular a user, if the evaluation meets a suggestion criterion;

[0017] S70: Checking a user response;

[0018] S80: performing steps S20-S70 again if the user response meets a rejection criterion, wherein in step S20 a new modification of the robot application is then determined using the first data processing on the basis of a new user specification;

[0019] S90: Implement the modification if the user response meets an acceptance criterion;

[0020] S100: performing steps S20-S90 again if a user input requests a further modification, wherein in step S20 a new modification of the robot application is determined using the first data processing based on a new user specification; and

[0021] S110: Output the robot application if the user input requests an output.

[0022] One idea of ​​the present invention is to use data processing which is at least partially based on machine learning and which, without loss of generality, is referred to as a first data processing method, in order to modify a provided robot application, preferably step by step or successively, and to provide a type of inner loop in which a modification determined or suggested by the data processing method is first tested by means of simulation and only after successful testing (in a type of outer loop) is it proposed to a user or presented for testing. The user then checks the modification - which has already been automatically tested by means of simulation in the inner loop - possibly also using a simulation of the robot application modified with the modification, and only after release orAcceptance (also) by the user of implementing the modification into the robot application (to be created). This can advantageously reduce the time required to create the robot application and thus to commission or operate the robot assembly, and / or make the creation or operation more reliable and / or efficient.

[0023] In one embodiment, the robot arrangement for which the robot application is created or which is operated according to an embodiment of the present invention has one or more robots, wherein the or one or more of the robots in one embodiment (each) have a robot arm and / or a mobile base and / or at least three, preferably at least six, (movement) axes or joints which are actuated or adjusted by corresponding drives, preferably electric motors, of the (respective) robot, in particular thus (each) have a robot arm with at least three, preferably at least six, (movement) axes or joints and drives therefor.

[0024] In one embodiment, the robot application comprises a movement of the robot(s) of the robot arrangement and / or an operation of one or more, preferably robot-guided, tools of the robot arrangement and / or further components of the robot arrangement, in particular conveyors, sensors, or the like. In one embodiment, the robot application comprises program code for operating the robot arrangement to carry out the robot application, preferably KRL code or the like. The present invention is particularly suitable for such robot arrangements or robot applications, in particular due to their boundary conditions, in particular kinematics, safety requirements, complexity, and the like.

[0025] In one embodiment, providing a robot application in step S10 comprises loading or calling an initial robot application, which may also be an empty robot application, in particular providing a corresponding file for storing program code or the like. Creating a robot application may in particular comprise generating and / or modifying program code, preferably stepwise or incrementally or successively, in particular iteratively.

[0026] In one embodiment, data processing based at least partially on machine learning comprises one or more artificial neural networks or the like and / or maps inputs to outputs or is trained accordingly for this purpose. The first data processing can in particular have a basic or fundamental computer model in the field of artificial intelligence or foundation model, in particular be, preferably a machine learning model that is or is trained on a large amount of data or broad data in such a way, in one embodiment by self-supervised learning or semi-supervised learning, that it can be adapted to a large number of downstream tasks. In a preferred development, the first data processing can have, in particular be, a foundation model that has already been adapted, in particular by fine-tuning and / or to simulation results.Such data processing is particularly suitable for the present invention, in particular due to its learning, input and output characteristics.

[0027] In one embodiment, a simulation performed in step S30 is a virtual, numerical, or computer simulation. An integrated development environment (IDE) in one embodiment comprises a development environment for setting up and / or programming, preferably offline, robot applications and / or preferably a simulation environment.

[0028] The iteration and proposal criteria can be complementary, meaning the proposal criterion is met if the iteration criterion is not met, and vice versa. Likewise, a termination criterion can also be provided that terminates the process, for example, when a predetermined number of failed attempts is reached, or something similar, such that the iteration and proposal criteria each indicate that the predetermined number of failed attempts has not yet been reached, or something similar.

[0029] Analogously, rejection and acceptance criteria can be complementary, meaning that the acceptance criterion is met if the rejection criterion is not met, and vice versa. Likewise, a termination criterion can also be provided that terminates the process, for example, when a predetermined number of failed attempts is reached, or something similar, such that the rejection and acceptance criteria each indicate that the predetermined number of failed attempts has not yet been reached, or something similar.

[0030] Analogously, a user input requesting further modification and a user input requesting an output can be complementary.

[0031] In one embodiment, an input of the first data processing unit comprises a configuration, in particular a state or status, of the development environment; in a further development, a state or status of the robot application and / or simulation environment and / or robot arrangement and / or an environment of the robot arrangement and / or a program code and / or a user-specified configuration. Based on such inputs, the data processing unit can operate particularly advantageously, in particular efficiently and / or robustly.

[0032] In one embodiment, an input for the first data processing, in a further development at least the user specification, is made at least partially in the form of natural language, for example in the form "Suggest a KRL code so that the robot moves from A to B", "Suggest the next program line", "Suggest the next program block", "I need a robot that can move a weight of ... kg" or the like. This allows the creation of the robot application to be improved, in particular faster and / or less error-prone. Additionally or alternatively, an input for the first data processing can also contain image data. This allows the creation of the robot application to be (further) improved, in particular more variable and / or less complex.

[0033] In one embodiment, the configuration of the development environment includes a configuration of the robot application and / or is or will be transformed into text format. Preferably, at least one image and / or at least one numerical model, for example, a preferably two- or three-dimensional CAD model or the like, is or will be transformed into text format. Transformation into text form is preferably performed by parsing or a parser. This allows for improved, in particular faster and / or less error-prone, creation of the robot application.

[0034] In one embodiment, an input of the first data processing is or will be at least partially augmented, preferably using prompt engineering. This allows the creation of the robot application to be improved, in particular faster and / or less error-prone.

[0035] In one embodiment, a simulation of a robot application modified with a modification is carried out in step S30 using another data processing method that is at least partially based on machine learning. In one development, this other data processing maps an output, in one embodiment textual, of the first data processing method to one or more actions of the simulation (environment). As a result, the simulation of the robot application can be better used to check the modification suggested or determined by the first data processing method, in particular can be better integrated and / or evaluated. An output or a result of the simulation can in particular comprise error messages, images and / or in particular videos or films, and / or can be at least partially in text form. As a result, the simulation of the robot application can be better used to check the modification suggested or determined by the first data processing method.identified modification can be used, in particular better integrated and / or evaluated.

[0036] In one embodiment, a modification suggested or determined by the first data processing is or will be limited by a predefined restriction, which can preferably be part of an input of the first data processing. In other words, in one embodiment, an input of the first data processing has a predefined restriction on the modification. For example, such a restriction can be "output one of the following modifications [a1], [a2], [a3]..." or the like. This allows the creation of the robot application to be improved, in particular faster and / or less error-prone.

[0037] In one embodiment, the modification is evaluated in step S40 using the first data processing unit; in particular, the first data processing unit can be instructed or "requested" to do so. This allows the method to be implemented very compactly.

[0038] In another embodiment, the modification is evaluated in step S40 using further data processing based at least partially on machine learning. This can, in particular, use a result of the simulation as input and map it to a corresponding evaluation, preferably already deciding whether the evaluation meets the iteration or proposal criteria. This allows the first data processing to work more efficiently and / or improve the evaluation, in particular, to make it more precise and / or more meaningful.

[0039] In one embodiment, outputting the modification to a user interface or a user in step S60 includes performing a simulation of the robot application modified with the modification using the development environment. This can improve the user's evaluation, in particular make it more reliable and / or easier.

[0040] In one embodiment, the first data processing unit is or will be trained, preferably in advance, based on simulations of modified robot applications and / or on existing robot applications. In a further development, random modifications and / or modifications determined by the first data processing unit are or will be implemented in a simulation, and the result is used as label information to train the first data processing unit.

[0041] According to one embodiment of the present invention, a method for operating a robot assembly comprises the steps of:

[0042] - Creating a robot application according to a method described here; and

[0043] - Operating the robot assembly to execute the created robot application.

[0044] According to one embodiment of the present invention, a system, in particular hardware and / or software, in particular program technology, is set up to carry out a method described here and / or comprises:

[0045] - means for providing a robot application;

[0046] - means for determining a modification of the robot application using a first data processing based at least partially on machine learning on the basis of a user specification, in particular the first data processing;

[0047] - means for performing a simulation of the robot application modified with the modification using an integrated development environment, in particular the development environment;

[0048] - means for evaluating the modification based on the simulation performed; these means are configured to repeat the determining, performing, and evaluating if the evaluation meets an iteration criterion, whereby a new modification of the robot application is then determined using the first data processing based on the simulation performed;

[0049] - means for outputting the modification to a user interface, in particular a user, if the evaluation meets a suggestion criterion;

[0050] - means for verifying a user response;

[0051] - means for repeating the determining, performing, evaluating, outputting and checking if the user response satisfies a rejection criterion, wherein a new modification of the robot application is then determined using the first data processing on the basis of a new user specification;

[0052] - Means for implementing the modification if the user response meets an acceptance criterion;

[0053] - means for repeating the determining, performing, evaluating, outputting and checking if a user input requests a further modification, wherein a new modification of the robot application is then determined using the first data processing on the basis of a new user specification; and

[0054] - Means for outputting the robot application if the user input requests an output.

[0055] In one embodiment, the system or its means comprises:

[0056] - means for at least partially augmenting an input of the first data processing; and / or

[0057] - means for carrying out a simulation of a robot application modified with a modification using another data processing method based at least partly on machine learning, in particular the other data processing method; and / or

[0058] - Means for evaluating the modification in step S40 using the first or a further data processing method based at least partially on machine learning, in particular the further data processing method; and / or - Means for training the first data processing method based on simulations of modified robot applications and / or on the basis of existing robot applications; and / or

[0059] - Means for operating the robot arrangement to carry out the created robot application, in particular a controller for controlling the robot arrangement and / or the robot arrangement.

[0060] A means within the meaning of the present invention can be designed in hardware and / or software, in particular at least one, in particular digital, processing unit, in particular a microprocessor unit (CPU), graphics card (GPU) or the like, preferably connected to a memory and / or bus system for data or signals, and / or one or more programs or program modules. The processing unit can be designed to execute instructions implemented as a program stored in a memory system, to detect input signals from a data bus, and / or to output output signals to a data bus. A memory system can have one or more, in particular different, storage media, in particular optical, magnetic, solid-state, and / or other non-volatile media. The program can be designed in such a way that it embodies the methods described here oris capable of carrying out, so that the processing unit can carry out the steps of such methods and thus in particular can create the robot application or operate the robot arrangement.

[0061] In one embodiment, a computer program product can comprise, in particular be, a storage medium, in particular a computer-readable and / or non-volatile one, for storing a program or instructions or with a program or instructions stored thereon. In one embodiment, execution of this program or these instructions by a system or a controller, in particular a computer or an arrangement of multiple computers, causes the system or the controller, in particular the computer(s), to carry out a method described here or one or more of its steps, or the program or the instructions are configured to do so.

[0062] In one embodiment, one or more, in particular all, steps of the method are fully or partially computer-implemented or one or more, in particular all, steps of the method are fully or partially automated, in particular by the system or its means.

[0063] In one embodiment, the system comprises the robot assembly and / or a controller for controlling the robot assembly.

[0064] Further advantages and features emerge from the subclaims and the exemplary embodiments. The following shows, partly schematically:

[0065] Fig. 1 : a system for carrying out a method according to an embodiment of the present invention;

[0066] Fig. 2: a method according to an embodiment of the present invention; and

[0067] Fig. 3: a signal flow in a method or in a system according to an embodiment of the present invention.

[0068] Fig. 1 shows a system for carrying out a method according to an embodiment of the present invention with a robot arrangement, in the exemplary embodiment a robot 1, a robot controller 2 and a computer (system) 3 with a first data processing based at least partially on machine learning, in the exemplary embodiment a foundation model, indicated by way of example by the reference symbol 3.1, and an integrated development environment 3.2 (cf. Fig. 3).

[0069] Fig. 2 shows a method according to an embodiment of the present invention, which is carried out using the system of Fig. 1, in particular its computer (system) 3. In a step S10, a robot application is provided, for example, by creating or loading a new or still empty KRL program in the development environment 3.2.

[0070] In a step S20, the first data processing or the Foundation Model 3.1 determines a modification a of the robot application, for example a further line of program code, a further program block, a change in the setup or the like, on the basis of a user specification in the form of a prompt p from a user interface or a user 4 (cf. Fig. 3), for example, "Create a program code so that the robot moves from A to B", and a configuration of the development environment 3.2 in the form of a state or status s of the development environment 3.2 (cf. Fig. 3), for example, a state and / or numerical kinematic model of the robot 1, its environment, a user-specified configuration and / or a, in particular, user-specified, program code. The configuration of the development environment can advantageously be parsed in text format and integrated into the user input, which in turn can be augmented orIn addition or alternatively to the textual input, the input of the first data processing may also comprise one or more images, for example one or more perspectives of the scene with the robot arrangement 1 and its environment (modeled in the simulation environment) or the like.

[0071] In step S30, a simulation of the robot application modified with modification a is performed using the development environment 3.2. A result of this simulation or feedback f provided by this simulation (see Fig. 3) can include, in particular, error messages and / or images, especially videos or films.

[0072] Based on this, in step S40 the modification a is evaluated, in one embodiment by the first data processing or the Foundation Model 3.1, in another embodiment by a further data processing based at least partially on machine learning. If the evaluation fulfills an iteration criterion (S40: "I"), the steps S20 - S40 are carried out again (indicated in Fig. 2 by a step S50), wherein in step S20 a new modification of the robot application is then determined with the help of the first data processing on the basis of the simulation performed. It can also be provided that the first data processing or the Foundation Model 3.1 takes additional inputs into account for the evaluation or decision.

[0073] If, however, the evaluation fulfills a suggestion criterion (S40: “V”), the modification a (thus automatically tested using the simulation) is output to the user or the user interface 4 in a step S60.

[0074] In step S70, a user response entered by the user via the user interface 4 is checked. Accordingly, a method for creating a robot application in one embodiment may generally include a corresponding (input of) a user response. The modification can be tested in the simulation environment.

[0075] If the user response fulfills a rejection criterion (S70. "S"), steps S20-S70 are performed again (indicated in Fig. 2 by a step S80), wherein in step S20 a new modification of the robot application is determined using the first data processing on the basis of a new user specification, ie the user provides a new prompt p to the first data processing or the Foundation Model 3.1 via the user interface 4 (cf. Fig. 3).

[0076] If, however, the user response meets an acceptance criterion (S70: "K"), modification a is implemented in a step S90, for example, the additional line or program block of the program code suggested or determined by the initial data processing or the Foundation Model 3.1, or a change to the setup, or the like. In a step S95, the user or a corresponding user input decides whether the creation of the robot application is complete or should be terminated, or whether further modifications are necessary or should be made.

[0077] If the user or his user input, preferably via the user interface, requests a further modification (S95: "W"), steps S20-S90 are carried out again (indicated in Fig. 2 by a step S100), wherein in step S20 a new modification of the robot application is then determined using the first data processing on the basis of a new user specification.

[0078] If, however, the user or his user input, preferably via the user interface, requests an output (S95: “E”), the robot application is output (Fig.2: step S110) and, in one embodiment, in a step S120 the robot arrangement is operated to carry out the robot application created in this way, in particular the robot controller 2 controls the robot 1 accordingly.

[0079] In a very simplified example, the user could first specify: “Create a KRL program to move the robot from A to B”.

[0080] Based on this prompt and an initial configuration, for example a pose of the robot, the (appropriately trained) Foundation Model determines a first part of a KRL program, for example the program line “LINE A”, which instructs a linear movement in the robot’s workspace to A.

[0081] This is tested using simulation. If a collision is detected and an error message is provided as feedback, the Foundation Model instead tries the program line "PTP A," which instructs a P(oint)-T(o)-P(oint) movement to A. This is again tested using simulation. If no collision is detected, this program line is output to the user.

[0082] The program accepts the program line, so it is implemented. The process then continues until the goal of a (collision-free) movement from A to B is reached.

[0083] Even from this highly simplified example, it is clear that the iteration loop of first suggesting the program line "LINE A" to the user so that the user has to check and reject the linear movement can be eliminated, thus reducing the time and effort required to create the robot application.

[0084] In the present disclosure, "has an X" generally does not imply an exhaustive list, but is a shortened form of "has at least one X" and also includes "has two or more Xs" and "has Y in addition to X." Although exemplary embodiments have been explained in the foregoing description, it should be noted that numerous modifications are possible. Furthermore, it should be noted that the exemplary embodiments are merely examples and are not intended to limit the scope, applications, or construction in any way.Rather, the foregoing description provides the skilled person with a guide for implementing at least one exemplary embodiment, whereby various modifications, particularly with regard to the function and arrangement of the described components, can be made without departing from the scope of protection as defined by the claims and equivalent combinations of features. List of reference symbols.

[0085] 1 robot (arrangement)

[0086] 2 Robot control 3 Computer (system)

[0087] 3.1 Foundation Model (initial data processing)

[0088] 3.2 integrated development environment

[0089] 4 User (interface) a Modification f Feedback p Prompt (user default) s State / status (configuration) of the development environment

Claims

Patent claims 1. A method for creating a robot application, the method comprising the steps of: S10: Providing a robot application; S20: Determining a modification of the robot application using a first data processing based at least partially on machine learning on the basis of a user specification; S30: Performing a simulation of the robot application modified with the modification using an integrated development environment; S40: Evaluate the modification based on the simulation performed; S50: performing steps S20 - S40 again if the evaluation meets an iteration criterion, wherein in step S20 a new modification of the robot application is determined using the first data processing on the basis of the simulation carried out; S60: Outputting the modification to a user interface if the evaluation meets a suggestion criterion; S70: Checking a user response; S80: performing steps S20-S70 again if the user response meets a rejection criterion, wherein in step S20 a new modification of the robot application is then determined using the first data processing on the basis of a new user specification; S90: Implement the modification if the user response meets an acceptance criterion; S100: performing steps S20-S90 again if a user input requests a further modification, wherein in step S20 a new modification of the robot application is determined using the first data processing based on a new user specification; and S110: Output the robot application if the user input requests an output.

2. Method according to claim 1, characterized in that an input of the first data processing is at least partially in the form of natural language and / or has a configuration of the development environment and / or a predetermined restriction of the modification and / or is at least partially augmented.

3. Method according to claim 2, characterized in that the configuration of the development environment comprises a configuration of the robot application and / or is transformed into text format.

4. Method according to one of the preceding claims, characterized in that a simulation of a robot application modified with a modification is carried out with the aid of another data processing method based at least partially on machine learning.

5. Method according to one of the preceding claims, characterized in that the modification is evaluated in step S40 using the first or a further data processing based at least partially on machine learning.

6. Method according to one of the preceding claims, characterized in that outputting the modification to a user in step S60 comprises performing a simulation of the robot application modified with the modification using the development environment.

7. Method according to one of the preceding claims, characterized in that the first data processing is trained on the basis of simulations carried out by modified robot applications and / or on the basis of existing robot applications.

8. A method for operating a robot assembly, comprising the steps of: - Creating a robot application according to a method according to one of the preceding claims; and - Operating (S120) the robot arrangement to carry out the created robot application.

9. System for creating a robot application, in particular for operating a robot arrangement (1), wherein the system is set up to carry out a method according to one of the preceding claims and / or comprises: - means for providing a robot application; - means for determining a modification of the robot application using a first data processing based at least partially on machine learning on the basis of a user specification; - means for performing a simulation of the robot application modified with the modification using an integrated development environment; - means for evaluating the modification based on the simulation performed; these means being configured to repeat the determining, performing, and evaluating if the evaluation meets an iteration criterion, whereby a new modification of the robot application is then determined using the first data processing based on the simulation performed; - means for issuing the modification to a user if the evaluation meets a suggestion criterion; - means for verifying a user response; - means for repeating the determining, performing, evaluating, outputting and checking if the user response satisfies a rejection criterion, wherein a new modification of the robot application is then determined using the first data processing on the basis of a new user specification; - Means for implementing the modification if the user response meets an acceptance criterion; - means for repeating the determining, performing, evaluating, outputting and checking if a user input requests a further modification, in which case a new modification of the Robot application is determined using the first data processing based on a new user specification; and - means for outputting the robot application if the user input requests an output.

10. A computer program or computer program product, wherein the Computer program or computer program product, in particular containing instructions stored on a computer-readable and / or non-volatile storage medium, which, when executed by one or more computers or a system according to claim 9, cause the computer(s) or the system to carry out a method according to one of claims 1 to 8.

Images