Creating a robot application

Abstract

A method for creating a robot application includes the following steps: S10: providing a robot application; S20: determining modifications to the robot application based on user settings using a first data processing method at least partially based on machine learning; S30: simulating the modified robot application using an integrated development environment; S40: evaluating the modifications based on the performed simulation; S50: if the evaluation meets iteration criteria, repeating steps S20–S40, wherein in step S20, a new modification to the robot application was determined based on the performed simulation using the first data processing method; S60: if the evaluation meets recommended criteria, modifying the robot application... The output is then sent to the user interface; S70: Check the user response; S80: If the user response meets the rejection criteria, repeat steps S20-S70, wherein in step S20, a new modification to the robot application is determined based on the new user settings using first data processing; S90: If the user response meets the acceptance criteria, implement the modification; S100: If the user inputs a request for further modification, repeat steps S20-S90, wherein in step S20, a new modification to the robot application is determined based on the new user settings using first data processing; and S110: If the user inputs a request for output, output the robot application. This invention also relates to a method for operating a robot device and a system or computer program (product).

Description

Technical Field

[0001] The present invention relates to a method for creating robotic applications, particularly for operating robotic devices, and a system or computer program or computer program product for performing the method. Background Technology

[0002] Based on internal practice, robot applications are created by having users write code, configure the robot and other elements, and generate scenarios for testing the application through simulation. This is very time-consuming.

[0003] One idea is to use artificial intelligence (KI or AI) to generate program code.

[0004] Here, users still need to review the program code modifications suggested by KI, which makes creating robot applications based on KI also very time-consuming. Summary of the Invention

[0005] The purpose of this invention is to better create robotic applications, and in particular to better operate robotic devices.

[0006] The object of the invention is achieved by a method having the features described in claim 1 or 8. Claim 9 or 10 protects a system, computer program, or computer program product for performing the method described herein. Dependent claims relate to advantageous extensions.

[0007] According to one embodiment of the present invention, a method for creating an application (“robot application”) for a robotic device includes the following steps, which are preferably performed in the listed order and may be partially repeated if necessary: S10: Provides robotic applications; S20: Using first data processing that is at least partially based on machine learning, determine a modification to the robot application based on user settings; S30: Using an integrated development environment, simulate the robot application modified with this modification; S40: Evaluate the modification based on the simulation performed; S50: If the evaluation meets the iteration criteria, then repeat steps S20–S40; wherein in step S20, new modifications to the robot application are determined based on the performed simulation using the first data processing. S60: If the evaluation meets the recommended criteria, then output the modification to the user interface, in particular to the user; S70: Check user response; S80: If the user's response meets the rejection criteria, then steps S20-S70 are re-executed; wherein in step S20, new modifications to the robot application are determined based on the new user settings using the first data processing. S90: If the user's response meets the acceptance criteria, then implement this modification; S100: If the user inputs a request for further modifications, steps S20-S90 are re-executed; wherein in step S20, new modifications to the robot application are determined based on the new user settings using the first data processing; and S110: If the user inputs a request for output, then output the robot application.

[0008] The concept of this invention lies in using at least part of machine learning-based data processing (generally referred to as first data processing) to preferably modify the provided robot application incrementally or continuously, and in this process, setting up an internal loop in which the modifications determined or suggested by the data processing are first tested by simulation; and only after the test is successful (in the manner of an outer loop) are they suggested to the user or submitted to the user for review; then the user reviews the modifications that have been tested by simulation in the internal loop, and if necessary, a simulation of the robot application modified with the modifications can also be used; only after the user also approves or accepts are the modifications implemented into the (to be created) robot application. Thus, the time required to create robot applications and therefore debug or run robot devices can be advantageously reduced, and / or the creation or operation can be made (more) reliable and / or (more) efficient.

[0009] In one embodiment, the robotic device (for which a robotic application is created or operated according to an embodiment of the invention) has one or more robots, wherein in one embodiment, the one or more robots have (each) a robotic arm and / or a mobile base and / or at least three, preferably at least six (motion) axes or joints, which are actuated or adjusted by corresponding drives, preferably electric motors, of the (corresponding) robots, particularly having (each) a robotic arm with at least three, preferably at least six (motion) axes or joints and their drives.

[0010] In one embodiment, the robotic application includes: the movement of one or more robots within a robotic device; and / or the operation of one or more preferably robot-guided tools and / or other components of the robotic device, particularly conveying devices, sensors, etc. In one embodiment, the robotic application includes program code, preferably KRL code, for operating the robotic device to perform the robotic application.

[0011] This invention is particularly suitable for such robotic devices or applications due to its boundary conditions, especially kinematics, safety requirements, and complexity.

[0012] In one implementation, providing the robot application in step S10 includes loading or invoking an initial robot application, which may also be an empty robot application, particularly providing corresponding files for storing program code, etc. Creating the robot application may particularly include, and is particularly preferably, gradually or in stages or continuously, especially iteratively, generating and / or modifying the program code.

[0013] In one embodiment, the data processing, at least partially based on machine learning, includes one or more artificial neural networks, and / or mapping inputs to outputs or being trained accordingly. The first data processing may, in particular, include, a foundational computer model or large-scale model in the field of artificial intelligence, preferably a machine learning model trained on a large or extensive dataset, in one embodiment, through self-supervised or semi-supervised learning, to adapt it to a wide range of downstream tasks. In a preferred extension, the first data processing may include, in particular, a tuned foundational model, especially one tuned through fine-tuning and / or based on simulation results. Such data processing is particularly suitable for the present invention due to its learning characteristics, input characteristics, and output characteristics.

[0014] In one implementation, the simulation performed in step S30 is a virtual, numerical, or computer simulation. In one implementation, the integrated development environment (IDE) includes a development environment for preferably offline designing and / or programming robot applications, and / or preferably includes a simulation (environment).

[0015] The iteration criteria and the recommended criteria can be complementary; that is, the recommended criteria are met when the iteration criteria are not met, and vice versa. Similarly, in particular, a termination criterion can be set, such as terminating the process when a preset number of error attempts is reached, or similarly, such as making both the iteration criteria and the recommended criteria include conditions such as not having reached a preset number of error attempts.

[0016] Similarly, rejection and acceptance criteria can be complementary; that is, acceptance criteria are met when rejection criteria are not met, and vice versa. Likewise, termination criteria can be set, such as terminating the process when a preset number of erroneous attempts is reached, or similar conditions, such as including a preset number of erroneous attempts or similar conditions before both rejection and acceptance criteria are reached.

[0017] Similarly, user input that requests further modification and user input that requests output can be complementary.

[0018] In one implementation, the input to the first data processing includes the configuration of the development environment, particularly its state or condition; in an extended embodiment, it includes the state of the robot application and / or simulation environment and / or the robot device and / or the environment of the robot device, and / or the program code and / or the user-defined configuration. Based on such input, the data processing can operate particularly advantageously, especially efficiently and / or stably.

[0019] In one implementation, the input to the first data processing, in an extended embodiment at least, is user-defined and at least partially in the form of natural language, such as "suggest KRL code to make the robot move from A to B," "suggest the next line of code," "suggest the next block of code," "I need a robot capable of moving ... kg of weight," etc. This can improve the creation of robot applications, particularly making them faster and / or less error-prone. Additionally or alternatively, the input to the first data processing may also include image data. This can (further) improve the creation of robot applications, particularly making them more flexible and / or less resource-intensive.

[0020] In one implementation, the configuration of the development environment includes configuring the robotic application and / or converting it to or to be converted to text format. Preferably, at least one image and / or at least one digital model (e.g., preferably a two-dimensional or three-dimensional CAD model) is converted to or to be converted to text format. Preferably, the conversion to text format is performed by parsing or a parser. This can improve the creation of the robotic application, particularly making it faster and / or less error-prone.

[0021] In one implementation, the input to the first data processing is at least partially enhanced, preferably by means of prompt engineering. This can improve the creation of robotic applications, particularly making them faster and / or less error-prone.

[0022] In one implementation, step S30 involves performing a simulation of the modified robot application using the aforementioned modifications via another data processing method at least partially based on machine learning. In an extended embodiment, this other data processing maps the output of the first data processing, in one embodiment, to one or more actions of the simulation (environment). This allows for better use of the simulation of the robot application to examine modifications suggested or determined by the first data processing, particularly for better integration and / or evaluation. The output or results of the simulation may include, in particular, error reports, images and / or especially videos or films, and / or at least partially presented in text form. This allows for better use of the simulation of the robot application to examine modifications suggested or determined by the first data processing, particularly for better integration and / or evaluation.

[0023] In one implementation, the modifications suggested or determined by the first data processing are subject to preset constraints, which are preferably part of the input to the first data processing. In other words, in one implementation, the input to the first data processing includes preset constraints on the modifications. For example, such constraints could be "output one of the following modifications [a1], [a2], [a3]". This can improve the creation of robotic applications, particularly making them faster and / or less error-prone.

[0024] In one implementation, the modification is evaluated in step S40 using a first data processing method; specifically, the first data processing method can be delegated or "queried" for this purpose. Thus, the method can be implemented very compactly.

[0025] In another implementation, the modification is evaluated in step S40 using another data processing method at least partially based on machine learning. This modification may, in particular, use simulation results as input and map them to the corresponding evaluation, preferably after determining whether the evaluation meets iterative or recommended criteria. Thus, the first data processing can work (more) efficiently and / or can improve the evaluation, particularly making it more accurate and / or more persuasive.

[0026] In one implementation, outputting the modifications to the user interface or user in step S60 includes performing a simulation of the modified robot application using the modifications via a development environment. This can improve the user's evaluation, particularly making it more reliable and / or simpler.

[0027] In one implementation, the first data processing, preferably in advance, is based on simulations performed on the modified robot application and / or on training based on an existing robot application. In an extended embodiment, random and / or modifications determined by the first data processing are implemented in the simulation, and the results are used as label information to train the first data processing.

[0028] According to one embodiment of the present invention, a method for operating a robotic device includes the following steps: - Create robot applications based on the methods described in this article; and - Run the robotic device to execute the created robotic application.

[0029] According to one embodiment of the invention, a system is specifically designed, using hardware and / or software techniques, particularly programming techniques, to perform the methods described herein, and / or include: - Devices used to provide robotic applications; - A device for determining modifications to a robot application based on user settings using first data processing, at least in part based on machine learning; particularly the first data processing. - A device for simulating a robot application modified with the aforementioned modifications using an integrated development environment, particularly a development environment; - A means for evaluating the modifications based on the performed simulation; wherein the means is designed to re-execute the determination, simulation and evaluation if the evaluation meets the iteration criteria, wherein the new modifications to the robot application are determined by means of a first data processing and based on the performed simulation; - A device for outputting modifications to the user interface, and in particular the user, when evaluating whether the recommended criteria are met; - A device for checking user responses; - A device for re-executing determination, simulation, evaluation, output and inspection when the user response meets the rejection criteria, wherein new modifications to the robot application are determined by means of first data processing and based on new user settings; - A means for implementing modifications when the user response meets the acceptance criteria; - A device for re-executing determination, simulation, evaluation, output, and inspection when user input requests further modification, wherein the new modification to the robot application is determined by means of first data processing and based on the new user settings; and - A device for outputting robot applications when a user inputs a request for output.

[0030] In one embodiment, the system or its apparatus has: - A means for at least partially enhancing the input of the first data processing; and / or - A device for simulating a modified robotic application using the modification with the aid of another data processing at least partially based on machine learning; and / or - A means for evaluating the modification in step S40 using a first or other data processing at least partially based on machine learning, particularly the other data processing; and / or - A means for training a first data processing based on simulations performed on a modified robotic application and / or based on an existing robotic application; and / or - Devices used to operate robotic devices to perform created robotic applications, particularly controllers for controlling one or more robotic devices.

[0031] The apparatus under the present invention can be implemented using hardware and / or software techniques, and in particular includes: at least one processing unit, preferably connected to a storage system and / or a bus system for data or signal communication, particularly a digital processing unit, such as a microprocessor unit (CPU), graphics card (GPU), etc.; and / or one or more programs or program modules. The processing unit can be designed to: execute instructions implemented as a program stored in the storage system; acquire input signals from the data bus; and / or send output signals to the data bus. The storage system can have one or more, particularly different, storage media, particularly optical, magnetic, solid-state, and / or other non-volatile media. The program can be designed to embody or perform the methods described herein, enabling the processing unit to perform the steps of these methods, thereby enabling, in particular, the creation of robotic applications or the operation of robotic devices.

[0032] A computer program product, in one embodiment, may include, and may in particular be, a computer-readable, non-volatile storage medium for storing a program or instructions, or a storage medium having a program or instructions stored thereon. In one embodiment, the program or instructions are executed by a system or controller, in particular a computer or an array of computers, such that the system or controller, in particular one or more computers, performs the methods described herein or one or more steps thereof, or the program or instructions are designed for this purpose.

[0033] In one implementation, one or more, particularly all, steps of the method are implemented entirely or partially by computer, or one or more, particularly all, steps of the method are executed entirely or partially automatically, particularly by the system or its apparatus.

[0034] In one implementation, the system includes a robotic device and / or a controller for controlling the robotic device. Attached Figure Description

[0035] Further advantages and features are given in the dependent claims and embodiments. For this purpose, some schematic representations are provided: Figure 1A system for performing a method according to an embodiment of the present invention is shown; Figure 2 A method according to an embodiment of the present invention is shown; and Figure 3 The signal flow in a method or system according to one embodiment of the present invention is illustrated. Detailed Implementation

[0036] Figure 1 A system for performing a method according to an embodiment of the present invention is shown, the system comprising a robotic device (robot 1 in this embodiment), a robot controller 2, and a computer (system) 3; the computer having a first data processing (based on a machine learning model in this embodiment), exemplarily shown by reference numeral 3.1, and an integrated development environment 3.2 (see Figure 3.2). Figure 3 ).

[0037] Figure 2 A method according to an embodiment of the present invention is shown, the method by means of Figure 1 The system, especially its computer (system) 3, is used to execute.

[0038] In step S10, a robot application is provided, for example by creating a new or still empty KRL program in development environment 3.2 or loading the program.

[0039] In step S20, the first data processing or basic model 3.1 is based on data from the user interface or user 4 (see...). Figure 3 The prompts in the form of p (e.g., "Create program code to make the robot move from A to B") and the status or condition of the development environment in the form of s (see Development Environment 3.2 configuration) are also provided. Figure 3 (e.g., the state and / or numerical kinematic model of robot 1, its environment, user-defined configuration and / or particularly user-defined program code), to determine modifications to the robot application, such as another line of program code, another program block, settings changes, etc. Here, the configuration of the development environment can advantageously be parsed into text format and integrated or will be integrated into the user input, while the user input itself can be enhanced or will be enhanced. In addition to or instead of text input, the input for the first data processing may also include one or more images, such as one or more perspectives of a scene having robot device 1 and its environment (modeled in a simulation (environment)).

[0040] In step S30, the robot application modified using the modified method a is simulated using development environment 3.2. The simulation results or the feedback f provided by the simulation (see...) Figure 3 This may include error reports and / or images, especially videos or movies.

[0041] Based on this, the modification a is evaluated in step S40, which in one embodiment is performed by the first data processing or the base model 3.1, and in another embodiment is performed by another data processing that is at least partially based on machine learning.

[0042] If the evaluation meets the iteration criterion (S40: "I"), then repeat steps S20–S40 (in Figure 2 (This is illustrated in step S50), where in step S20, new modifications to the robot application are determined based on the performed simulation using the aid of first data processing. It can also be specified here that the first data processing or base model 3.1 considers additional inputs for evaluation or decision-making.

[0043] Conversely, if the evaluation meets the recommended criteria (S40: "V"), then in step S60, the modification a (which automatically passes the simulation test) is output to the user or user interface 4.

[0044] In step S70, the user response input by the user through user interface 4 is examined. Accordingly, in one embodiment, the method for creating a robot application typically includes the corresponding user response (input). Here, this modification can be tested in a simulation (environment).

[0045] If the user's response meets the rejection criterion (S70: "S"), then repeat steps S20-S70 (in Figure 2 (As shown in step S80), in step S20, new modifications to the robot application are determined based on new user settings using the first data processing, i.e., the user provides new prompts p to the first data processing or the base model 3.1 through the user interface 4 (see...). Figure 3 ).

[0046] Conversely, if the user response meets the acceptance criteria (S70: "K"), then modification a is implemented in step S90, such as another line or block of program code or setting change suggested or determined by the first data processing or basic model 3.1.

[0047] In step S95, the user or the corresponding user input determines whether the creation of the robot application is complete or should end, or whether further modifications are needed or should be made.

[0048] If the user or their user input preferably requests further modification via the user interface (S95: "W"), then steps S20-S90 are re-executed (in... Figure 2 (This is represented by step S100), wherein in step S20, new modifications to the robot application are determined based on new user settings using the first data processing.

[0049] Conversely, if the user or their user input preferably requests output via a user interface (S95: "E"), then the robot application outputs ( Figure 2 (Step S110), and in one embodiment, in step S120, the robot device is operated to execute the robot application thus created, in particular the robot controller 2 controls the robot 1 accordingly.

[0050] In a highly simplified example, for instance, a user could first set: "Create a KRL program to make the robot move from A to B."

[0051] Based on this prompt and initial configuration, such as the robot's pose, the first part of the KRL program is determined by the (correspondingly trained) base model, such as the program line "LINE A", which indicates the linear motion to A in the robot's workspace.

[0052] This is tested through simulation. If a collision is detected during this process and an error report is provided as feedback accordingly, the base model attempts to replace it with the line "PTP A", which indicates the PTP (point-to-point) motion to A.

[0053] This is tested again through simulation. If no further collisions are detected during this process, the line of code is output to the user.

[0054] The user accepts the procedure line and implements it. Now, the method continues accordingly until the goal of moving from A to B (without collision) is achieved.

[0055] This highly simplified example already demonstrates that the iterative loop first suggests the program line "LINE A" to the user, requiring the user to check and reject linear motion, thereby reducing the time and effort required to create robot applications.

[0056] In this disclosure, "having X" generally does not represent a closed enumeration, but is a shorthand for "having at least one X," and also includes "having two or more X" and "having Y in addition to X." Although exemplary embodiments have been set forth in the foregoing description, it should be noted that many variations are possible. Furthermore, it should be noted that these exemplary embodiments are merely examples and should not be construed as limiting the scope of protection, application, or construction. Rather, the foregoing description provides guidance for those skilled in the art to make modifications to at least one exemplary embodiment, wherein various changes can be made, particularly regarding the function and arrangement of the components, without departing from the scope of the invention, for example, as can be obtained according to the claims and equivalent combinations of features thereof. List of reference numerals 1. Robot (device) 2. Robot Controller 3. Computer (System) 3.1 Basic Model (First Data Processing) 3.2 Integrated Development Environment 4. User (Interface) a Modify f Feedback p Prompt (User Settings) s The status / condition (configuration) of the development environment.

Claims

1. A method for creating a robot application, wherein the method includes the following steps: S10: Provides robotic applications; S20: Using first data processing at least partially based on machine learning, determine modifications to the robot application based on user settings; S30: Using an integrated development environment, simulate the robot application modified with the aforementioned modifications; S40: Evaluate the modifications based on the simulation performed; S50: If the evaluation meets the iteration criteria, then steps S20–S40 are re-executed, wherein in step S20, new modifications to the robot application are determined based on the performed simulation by means of the first data processing; S60: If the evaluation meets the recommended criteria, the modification is output to the user interface; S70: Check user response; S80: If the user response meets the rejection criteria, then steps S20-S70 are re-executed, wherein in step S20, the new modification of the robot application is determined based on the new user settings by means of the first data processing; S90: If the user response meets the acceptance criteria, then the modification is implemented; S100: If the user inputs a request for further modifications, then steps S20-S90 are executed again, wherein in step S20, the new modifications to the robot application are determined based on the new user settings using the first data processing. as well as S110: If the user input requests an output, then output the robot application.

2. The method according to claim 1, characterized in that, The input to the first data processing is at least partially in the form of natural language, and / or includes the configuration of the development environment and / or the preset constraints of the said modifications, and / or is at least partially enhanced.

3. The method according to claim 2, characterized in that, The configuration of the development environment includes the configuration of the robot application and / or its conversion to text format.

4. The method according to any one of the preceding claims, characterized in that, Simulations of the modified robot application utilizing the aforementioned modifications are performed using another data processing method, at least in part based on machine learning.

5. The method according to any one of the preceding claims, characterized in that, In step S40, the modification is evaluated using either the first data processing, which is at least partially based on machine learning, or the other data processing, which is at least partially based on machine learning.

6. The method according to any one of the preceding claims, characterized in that, Step S60, outputting the modification to the user, includes: performing a simulation of the robot application modified using the modification in the development environment.

7. The method according to any one of the preceding claims, characterized in that, The first data processing is based on simulation performed on the modified robot application and / or training based on an existing robot application.

8. A method for operating a robotic device, comprising the following steps: - Create a robot application according to any one of the preceding claims; as well as - Run (S120) the robot device to execute the created robot application.

9. A system for creating robotic applications, particularly for operating robotic devices (1), wherein the system is designed to perform the method according to any one of the preceding claims, and / or includes: - Devices used to provide robotic applications; - A means for determining modifications to the robot application based on user settings using first data processing that is at least partially based on machine learning; - A device for simulating a robot application modified with the aforementioned modifications using an integrated development environment; - A means for evaluating the modifications based on the performed simulation; The device is designed to re-execute the determination, simulation, and evaluation if the evaluation meets the iterative criteria, wherein the new modifications to the robot application are determined by means of the first data processing and based on the performed simulation; - A means for outputting the modification to the user when the evaluation meets the recommended criteria; - A device for checking user responses; - A means for re-executing determination, simulation, evaluation, output and inspection when the user response meets the rejection criteria, wherein the new modification of the robot application is determined by means of the first data processing and based on the new user settings; - A means for implementing the modification when the user response meets the acceptance criteria; - A means for re-executing determination, simulation, evaluation, output, and inspection when user input requests further modifications, wherein the new modifications to the robot application are determined by means of the first data processing and based on the new user settings; and - A means for outputting the robot application when the user inputs a request for output.

10. A computer program or computer program product, wherein, The computer program or computer program product comprises, in particular, instructions stored on a computer-readable and / or non-volatile storage medium, which, when executed by one or more computers or the system according to claim 9, cause the one or more computers or the system according to claim 9 to perform the method according to any one of claims 1 to 8.

Images