Creating and implementing robot applications

Abstract

In a method for creating a robot application, the following steps are repeated: selecting a routine from a predefined set of selectable routines (SR) using data processing based at least partially on machine learning and based on speech input (S); parameterizing at least one parameter of the selected routine using data processing based at least partially on machine learning and based on the speech input and / or a context of the robot application to be created; and adding the selected and parameterized routine to the robot application. The invention also relates to a system or computer program (product).

Description

[0001] The present invention relates to a method for creating or carrying out a robot application, as well as a system and a computer program or computer program product for carrying out a method described herein.

[0002] The object of the present invention is to improve the creation, and in particular the execution, of robot applications.

[0003] This problem is solved by a method with the features of claim 1 and 10, respectively. Claims 11 and 12 protect a system, computer program, and computer program product for carrying out a method described herein. The dependent claims relate to advantageous embodiments.

[0004] According to one embodiment of the present invention, the following steps are taken to create a robot application: - Selecting one routine each - from a predefined set of selectable routines - by means of data processing based at least partially on machine learning - based on (each) one voice input; - Parameterizing one or more parameters of the (respectively) selected routine - by means of data processing based at least partially on machine learning - based on the (respective) speech input and / or a context of the robot application to be created, in particular therefore - Parameterizing one or more parameters of the (respectively) selected routine based on the (respective) speech input and / or - Parameterizing one or more parameters of the (respectively) selected routine based on the context of the robot application to be created; - in a further training course, parameterizing one or more parameters of the (respective) selected routine based on the (respective) speech input and the context of the robot application to be created; and - Adding the (respective) selected and parameterized routine to the robot application This can be repeated several times. According to one embodiment of the present invention, a robot application created according to the invention or according to a method described herein is carried out.

[0005] By creating routines using or with the aid of artificial intelligence - based on voice input - selected from a predefined set of selectable routines and - the selected routines parameterized This allows the robot application to be developed more quickly, securely, and / or by non-experts. In particular, it allows for the use of more general chatbot interfaces that are not fine-tuned to specific routines, and / or helps avoid or reduce chatbot errors, especially so-called hallucinations or unexpected and incorrect routines.

[0006] In one embodiment, a robot application comprises a movement and / or a work process of one or more robots and / or a robot environment, in a further embodiment, tools, conveying devices, and the like that interact with the robot(s), and / or a program for controlling the robot(s) and / or the robot environment, wherein the robot(s) in a further embodiment (each) has a stationary or mobile base and / or a robot arm with at least three, preferably at least six, and in particular at least seven, joints or axes of movement, and / or the robot environment comprises one or more tools, in particular machining and / or holding tools, and / or conveying devices that interact with or are set up or used for this purpose by the robot(s). The robot application can, in particular, be a real robot application or...Movement and / or work process of one or more real robots and / or a real robot environment and / or a (corresponding) program and / or a virtual robot application, or movement and / or work process of one or more virtual robots and / or a virtual robot environment and / or a (corresponding) program, may include, in particular, the execution of a robot application. Accordingly, the execution of a robot application may include, in particular, the real execution of a (real) robot application, or the movement and / or work process of one or more real robots and / or a real robot environment, and / or the virtual, in particular simulated, execution of a (virtual) robot application, or the movement and / or work process of one or more virtual robots and / or a virtual robot environment.

[0007] One or more of the routines, in each execution, comprise actions, in particular movements and / or functions, of the (real and / or virtual) robot(s) and / or robot environment, and / or commands or instructions for the robot(s) (environment), and / or functions of a robot simulation and / or robot programming and / or simulation environment. In one execution, the procedure for creating a robot application comprises providing, or in a further development, specifying, the (predefined) set of selectable routines. In one execution, the selectable routines are specified in textual form; in a further development, they comprise, in each execution, a command and / or a parameter list and / or a description, in particular a functional description, and / or in textual form.Additionally or alternatively, the selectable routines, especially their descriptions, include (program) code and / or images in one execution, and image sequences in a further training.

[0008] In one implementation, selecting one or more of the routines (each) involves the following steps: - Embedding the speech input into a vector called the input vector in a vector space, wherein this vector space contains vectors called function vectors into which the selectable routines are embedded, in an execution based on the description of the respective routine; and - Selecting one of the selectable routines, taking into account a relation, preferably proximity, between the input vector associated with the speech input and the function vectors associated with the selectable routines.

[0009] In the present context, embedding is understood in particular, in a manner customary in the field, as an assignment, preferably bijective, between the data to be embedded, here the speech input or the selectable routines, and the vectors, preferably such that the distance between vectors is smaller the more similar the data assigned to them (to be embedded) are to each other or the better they match or correspond to each other.

[0010] By transforming speech input and selectable routines into a common vector space, these can be handled advantageously, in particular suitable routines can be selected particularly efficiently and / or reliably, and / or speech input can be processed particularly efficiently and / or easily, thereby (further) improving the creation of the robot application.

[0011] In a further education course, selecting a routine, taking into account a relation between the (respective) input vector and the function vectors, involves (at least) the following steps: - Determining the distances of the input vector to the function vectors; and - Selecting a routine based on a determined distance between the input vector and the function vector associated with that (selected or to be selected) routine. A distance between two vectors u, v can encompass a mathematical norm, in particular, a cosine metric or similarity u · v / (||u|| · ||v||) with the scalar product u · v and the magnitudes ||u||, ||v|| has proven particularly suitable. As can be seen from this, a distance between vectors in the vector space is understood to mean, in particular, a similarity of these vectors, whereby other metrics for the distance are also suitable or possible.

[0012] By using distances in the (common) vector space, the most suitable routines can be selected particularly advantageously, especially in a particularly efficient and / or reliable manner, and thereby the creation of the robot application can be (further) improved.

[0013] Preferably, the routine that, according to a machine-learned model, shows the greatest similarity to the speech input is selected.

[0014] In a further training course, selecting a routine taking into account the determined distance of an input vector to the function vectors involves a nearest-neighbor classification ("nearest-neighbor algorithm") of the function vectors, whereby in one execution a predetermined number of nearest neighbors (among or from the function vectors) to the input vector are determined and / or the routine whose assigned function vector is closest to the input vector is selected.

[0015] Additionally or alternatively, in one implementation a ranking or ranking process, or in a further training, a reranking or adjustment of a ranking of the selectable routines is carried out, preferably based on a match or compatibility of the speech input with the respective routine determined by a machine-learned model, preferably based on the description of the respective routine, wherein in one further training the determined nearest neighbors, or in another further training the specified number of nearest neighbors, are ranked (re)ranked and the remaining routines are discarded or can be disregarded (in the (respective) selection).

[0016] Therefore, two particularly advantageous further training courses for selecting a routine are suggested: - The routine whose assigned function vector is closest to the input vector is selected; or - Several routine candidates are identified based on the proximity of their assigned function vectors to the input vector, and from these, based on a match or alignment of the speech input with the respective routine determined by a machine-learned model, preferably its description, the best one is selected. The learned model is selected to best match the speech input routine, although the selection is of course not limited to these two approaches.

[0017] Through a nearest-neighbor algorithm and (optional) ranking, the most suitable routines can be selected particularly advantageously, especially efficiently and / or reliably, thereby (further) improving the creation of the robot application.

[0018] In addition to or as an alternative to a nearest-neighbor algorithm and / or reranking, in one implementation a speech input is classified as an anomaly if the determined distance of the input vector associated with this speech input to the function vectors fulfills a predefined condition; in a further development, a determined distance of the input vector associated with this speech input to the function vectors, in particular a minimum, maximum, or statistically averaged distance, exceeds a predefined limit, or the like. In one implementation, in another further development, this limit is specified based on statistical evaluations and / or statistically and / or probabilistically determined distributions, preferably such that statistically more frequent and / or more probable speech inputs or input vectors are not classified as anomalies, or statistically rarer and / or less probable speech inputs or input vectors are not classified as anomalies.Input vectors can be classified as anomalies. If an anomaly is classified, a corresponding message can be issued.

[0019] This allows unsuitable language inputs to be filtered out particularly advantageously, especially efficiently and / or reliably, thereby (further) improving the creation of the robot application.

[0020] In one implementation, reranking is performed using a (machine-learned) Large Language Model (“LLM”). Additionally or alternatively, in another implementation, one or more parameters of one or more selected routines are parameterized using one or more, preferably different, (machine-learned) Large Language Models. Additionally or alternatively, in another implementation, language input is embedded in an input vector and / or the routines are embedded in the function vectors using at least one (machine-learned) embedding model; in a further development, this is done using one or the same or different (machine-learned) Large Language Models. By using, in particular, these machine-learned models for selecting and / or embedding the routines in the function vectors, the system achieves a more efficient and efficient reranking process.Parameterizing a routine using data processing that is at least partially based on machine learning allows for particularly advantageous handling of speech inputs and routines, especially enabling the selection of suitable routines to be particularly efficient and / or reliable, and / or the processing of speech input to be particularly efficient and / or simple, thereby (further) improving the creation of the robot application.

[0021] In one implementation, the parameterization of one or more parameters of one or more selected routines is performed based on the voice input on which that routine was selected. For example, if a voice input is "The robot's TCP should perform a linear movement to the nearest welding point at a speed of 0.8 m / s," then, due to the small distance between the input vector associated with this voice input and the function vector associated with this command, the command "LinMotion(working point, speed)" can be selected. Then, its parameter "speed" can be set based on the phrase "at a speed of 0.8 m / s," and its parameter "working point" can be set based on the context of the robot application to be created.In this context, "context" is generally understood to mean information for characterizing the situation of an entity in interaction with other entities, in particular a relationship between one or more references of one or more robots and / or a robot environment (the robot application).

[0022] Additionally or alternatively, in one embodiment the parameterization of several parameters of one or more selected routines is carried out step by step, preferably in such a way that one of the parameters of the (respective) routine is parameterized or its value is set successively.

[0023] This allows the routines to be parameterized in a particularly advantageous, especially efficient and / or reliable way, thereby (further) improving the creation of the robot application.

[0024] In one version, one or more of the voice inputs (each) are written voice inputs, or in a training course, voice inputs entered via keyboard, touchscreen, or the like. Additionally or alternatively, one or more of the voice inputs (each) are acoustic voice inputs, or in a training course, voice inputs entered via microphone or the like and / or converted into text form using speech recognition.

[0025] This allows the robot application to be created more quickly and / or safely and / or by non-experts.

[0026] As already mentioned, the selectable routines in one implementation include descriptions, preferably at least partially in textual form. In a further development, the descriptions may also include images, especially image sequences, or the like. The descriptions preferably characterize the respective routine and may, in particular, describe its functionality. Preferably, the routines are selected based on these descriptions or their match with the respective language input. In a further development, the routines are embedded in the vector space based on their descriptions.

[0027] In this way, voice input can be implemented in routines in a particularly advantageous, especially efficient and / or reliable way.

[0028] In one implementation, these descriptions are adapted using data processing based at least partially on machine learning, preferably based on language input.

[0029] This allows the creation of robot applications to adapt to the language inputs used to customize the descriptions, and thus further language inputs can be implemented in routines in a particularly advantageous, especially more efficient and / or reliable way.

[0030] In one implementation, an LLM-based chatbot interface is provided, implemented, or used for creating a robot application, whereby the chatbot can advantageously select from a predefined set of routines and / or parameterize the selected routines using user or voice input and / or context and / or stepwise. This allows the robot application to be created advantageously faster and / or more securely and / or by non-experts.

[0031] According to one embodiment of the present invention, a system for creating a robot application, in a further development for carrying out the created robot application, is set up and / or has, in particular in terms of hardware and / or software, especially programming, for carrying out a method described herein: - Means of selecting a routine from a given set of selectable routines by means of data processing based at least partially on machine learning and based on speech input; - Means of parameterizing at least one parameter of the selected routine by means of data processing based at least partially on machine learning, based on speech input and / or a context of the robot application to be created; and - Means of adding the selected and parameterized routine to the robot application.

[0032] In one version, the system or its means exhibit: - Means for embedding speech input into an input vector in a vector space, wherein the vector space contains function vectors in which the selectable routines are embedded; and / or - Means of selecting one of the selectable routines, taking into account a relation between the input vector associated with the speech input and the function vectors associated with the selectable routines; and / or - Means of determining distances between the input vector and the function vectors; and / or - Means of selecting a routine taking into account a determined distance of the input vector to the function vector assigned to that routine; and / or - Means of performing a nearest-neighbor classification of the function vectors and / or a ranking of the selectable routines for selecting a routine, taking into account a determined distance of an input vector to function vectors; and / or - Means of classifying a speech input as an anomaly if the determined distance of the input vector associated with this speech input to the function vectors satisfies a predefined condition; and / or - at least one Large Language Model to perform the ranking; and / or - at least one Large Language Model for parameterizing one or more parameters of at least one selected routine based on speech input and / or a context of the robot application to be created, preferably stepwise; and / or - Means of adapting descriptions of routines based on speech input using data processing that is at least partially based on machine learning; and / or - the robot(s) and / or robot environment and / or a robot simulation and / or robot programming environment and / or robot simulation environment and / or the data processing(s) that are at least partially based on machine learning.

[0033] A means according to the present invention can be configured as hardware and / or software, in particular comprising at least one processing unit, preferably a microprocessor unit (CPU), graphics processing unit (GPU), or the like, preferably connected to a storage and / or bus system via data or signals, and / or comprising one or more programs or program modules. The processing unit can be configured to execute instructions implemented as a program stored in a storage system, to acquire input signals from a data bus, and / or to output signals to a data bus. A storage system can comprise one or more, in particular different, storage media, in particular optical, magnetic, solid-state, and / or other non-volatile media. The program can be configured to embody the methods described herein.is capable of executing such procedures, enabling the processing unit to perform the steps of such processes and, in particular, to create or execute the robot application. A computer program product may, in one version, include a storage medium, in particular a computer-readable and / or non-volatile medium, for storing a program or instructions, or with a program or instructions stored thereon. In one version, the execution of this program or these instructions by a system or controller, in particular a computer or an arrangement of several computers, causes the system or controller, in particular the computer(s), to execute a procedure described herein or one or more of its steps, or the program or instructions are configured for this purpose.

[0034] In one embodiment, one or more, in particular all, steps of the procedure are fully or partially computer-implemented, or one or more, in particular all, steps of the procedure are fully or partially automated, in particular by the system or its means.

[0035] Further advantages and features will become apparent from the dependent claims and the exemplary embodiments. These are shown, in part schematically: Fig. 1: a system for creating and executing a robot application according to an embodiment of the present invention; and Fig. 2: a part of the system and a method for creating and executing the robot application according to an embodiment of the present invention

[0036] Fig. Figure 1 shows a system for creating and executing a robot application according to an embodiment of the present invention. Fig. 2 a part of this system and a method for creating and executing the robot application according to one embodiment of the present invention. The system comprises (see Fig. 1) A robot 1000 and a computer system 2000, which may in particular include a robot programming and / or simulation environment and / or robot control. The computer system 2000 may in particular include multiple computers, and the robot 1000 may be a virtual or simulated robot or a real robot.

[0037] In step S10, a set of selectable routines are specified, which are then executed. Fig. 2 are schematically indicated by squares and capital letters. For example, A could be a function or command "InsertObject(object_name:str)" for inserting an object parameterized by the string "object_name", B a function or command "DeleteObject(object_name: str)" for deleting an object parameterized by the string "object_name", C a function or command "LinMotion(point_name: str, velocity: float = 0.1, blending: bool = False)" for linear movement of a TCP of robot 1000 to the target point parameterized by the string "point_name" with the velocity parameterized by the floating-point number "velocity" and preset with the default value "0.1" and the blending selectable by the boolean parameter "blending" and preset with the default value "False", D an analogous function or command “PTPMotion(point_name: str, velocity: float = 0.1, blending: bool = False) for a PTP movement of the TCP, , E a function or command “OpenWeldingDialogue(welding_object: str, power_source: str, welding_gun: str)” to configure welding of the environment object or workpiece parameterized by the string “welding_object” with the energy source parameterized by the string “power_source” and the welding tool parameterized by the string “welding_gun”, and F a function or command “TransformObject(transform_object: str,...)” to transform an object parameterized by the string “transform_object” according to the further parameters, which are not listed here.

[0038] The routines (here, examples A-F) each have a functional description, preferably at least partially in textual form, and in further training, also including images or the like. Using routine C ("LinMotion(point_name: str, velocity: float = 0.1, blending: bool = False)") as an example, such a description might read as follows:

[0039] "Generates a linear motion to a target point named point_name. The motion is linear in Cartesian space and is performed at the specified velocity [m / s]. The motion can be overridden at the target point or not."

[0040] Parameters: point_name: String - the name of the target point of the linear movement. velocity: Floating-point number - the speed of the movement. blending: Boolean variable - whether the movements are blended or not.

[0041] The routines are embedded in step S15 using a LLM1 (Language Model) based preferably on their descriptions in function vectors in a vector space, which are in Fig. 2 are schematically indicated by circles and corresponding lowercase letters, i.e., routine A is embedded in the function vector a assigned to it, routine B in the function vector b, routine C in the function vector c, routine D in the function vector d, routine E in the function vector e, and routine F in the function vector f.

[0042] In step S20, a speech input S, also indicated by a square, is provided.

[0043] This language input S is embedded in a vector space in step S25 using the (same) L(arge)L(anguage)M(odel)s LLM1.

[0044] The embedding model “text-embedding-ada” from OpenAI serves as an example.

[0045] In step S30, an anomaly detection method (ADD) classifies a speech input as an anomaly and discards it accordingly if the determined distance of the input vector associated with this speech input to the function vectors meets a predefined condition. Fig. 2. This is schematically indicated by exceeding a distance threshold for a hatched input vector s, into which, for example, the voice input "Let's play football" would be embedded. This distance threshold can advantageously be predefined, and in particular adjusted, based on static evaluations of previously processed voice inputs or the like. In the case of a classified anomaly, an (error) message can be output in step S30.

[0046] In step S35, a Nearest-Neighbour-Search (NNS) is used to determine, for example, the three nearest neighbors from the function vectors a - f to the input vector s that have the smallest distance, again, in an advantageous embodiment, the smallest cosine metrics.

[0047] These are reranked in step S40 by another LLM2 (Language Model), which is configured, for example, by an instruction.

[0048] "You are an AI assistant that selects suitable functions from lists. You only answer in natural numbers (the indices of the list)" and some examples, for instance { 'User': "Move the robot 5 cm in the z-direction and rotate it around the y-axis by 15 degrees.

[0049] Function list: [ 'create_point_at_tcp(robot_name: str)', 'transform_object(object_name: str, x: float = 0, y: float = 0, z: float = 0, a: float = 0, b: float = 0, c: float = 0)', 'create_linear_motion(point_name: str, velocity: float = 0.1, blending: bool = False)' ],'Assistant': '1'}.

[0050] The list for this model is the output of NNS. If the LLM2 model outputs an index j != 0, the element of j is moved to the top of the list. This step can also be used to detect unusual speech inputs and, in this case, output the message: "No routine in the list is suitable for this speech input".

[0051] In this example, the user entered the voice command "Please create a linear motion with a speed of 0.8 m / s".

[0052] Accordingly, routine C (LinMotion(point_name: str, velocity: float = 0.1, blending: bool = False)) is selected ( Fig. 2: Step S40) and in step S50 parameterized by another LLM3 (Language Model). The simulation context is included and taken into account. This context can be determined from the simulation state and transformed into text form for the LLM3, for example, in the following form: “The user has selected point ‘P5’” or “The only welding source in the scene is ‘ABCD1234’”.

[0053] The LLM3 receives as input a description of the selected routine, the user's voice input, and context (information) of the simulation (environment) in text form and can be configured as follows, for example: 'You are an AI assistant that identifies parameters for routines. The parameters are taken either directly from the user's voice input or from the additional context description. Ensure that the parameter names exactly match the routine description. For example, if the routine description contains 'point_name', the parameter should be 'point_name' and not 'point'. Prioritize voice input over context. Always respond in JSON format.'

[0054] Examples: [ {'User': 'Create a linear motion to P5. Routine description: [... routine docstring...] Context: empty, 'Assistant': '{"point": "P5"}'}, {'User': 'Create a linear motion. Routine description: [... routine docstring...] Context: P12 selected', 'Assistant': '{"point": "P12"}'}, {'User': 'Create weld source "SFR10_as". Routine description: [... routine docstring...] Context: empty', 'Assistant': '{"object_name": "SFR10_as"}'}, {'User': Remove object "AB12". Routine description: [... routine docstring...] Context: Object "CD34" selected, 'Assistant': '{"object_name": "AB12"}'}].

[0055] Further speech input may be processed. Otherwise, the robot application created in this way by adding the corresponding routines is completed and executed if necessary. Fig.2: Step S60). The descriptions of the routines can also be adapted to more closely resemble voice input.

[0056] In the present disclosure, "has an X" does not generally imply an exhaustive list, but is a shorthand for "has at least one X" and also includes "has two or more X" as well as "has Y in addition to X". Although exemplary examples were explained in the preceding description, it should be noted that a multitude of variations are possible.

[0057] For example, the reranking step by LLM2 can be omitted and the routine whose associated function vector is closest to the input vector can be selected.

[0058] Furthermore, it should be noted that the exemplary embodiments are merely examples and are not intended to restrict the scope of protection, applications, or structure in any way. Rather, the preceding description provides the skilled person with a guideline for implementing at least one exemplary embodiment, whereby various modifications, particularly with regard to the function and arrangement of the described components, can be made without departing from the scope of protection as defined by the claims and these equivalent combinations of features. Reference symbol list 1000 robots 2000 computers AF Routine af function vector S voice input s input vector LLM 1 / 2 / 3 Large Language Model NNS Nearest Neighbour Search ADD Anomaly Detection Based on Distance APP Application

Claims

[1] Method for creating a robot application, wherein the steps are: - Selecting a routine from a given set of selectable routines (AF) using data processing based at least partially on machine learning and language input (S); - Parameterizing at least one parameter of the selected routine using data processing based at least partially on machine learning, using speech input and / or a context of the robot application to be created; and - Adding the selected and parameterized routine to the robot application be repeated several times. [2] Method according to claim 1, characterized by , that selecting a routine involves the following steps: - Embedding the speech input in an input vector (s) in a vector space, wherein the vector space has function vectors (af) in which the selectable routines are embedded; and - Selecting one of the selectable routines, taking into account a relation between the input vector assigned to the speech input and the function vectors assigned to the selectable routines. [3] Method according to claim 2, characterized by , that selecting a routine, taking into account a relation between input vector and function vectors, involves the following steps: - Determining the distances of the input vector to the function vectors; and - Selecting a routine taking into account a determined distance between the input vector and the function vector assigned to that routine. [4] Method according to claim 3, characterized by , that - selecting a routine taking into account a determined distance of an input vector to function vectors - a nearest-neighbor classification of the function vectors and / or - a ranking of the selectable routines exhibits; and / or that - a speech input is classified as an anomaly if the determined distance of the input vector assigned to this speech input to the function vectors fulfills a predefined condition. [5] Method according to claim 4, characterized by that the ranking is carried out using a Large Language Model (LLM2). [6] Method according to any one of the preceding claims, characterized bythat the parameterization of one or more parameters of at least one selected routine is carried out step by step and / or using at least one Large Language Model (LLM3) based on a language input and / or a context of the robot application to be created. [7] Method according to any one of the preceding claims, characterized by that at least one voice input includes a written or acoustic voice input. [8] Method according to any one of the preceding claims, characterized by that the selectable routines have descriptions. [9] Method according to claim 8, characterized by that these descriptions are adapted by means of data processing based at least partially on machine learning, in particular on the basis of language input. [10] Method for performing a robot application comprising the steps: - Creating a robot application according to a method according to one of the preceding claims; and - Execute (S60) the created robot application. [11] System for creating a robot application, which is set up and / or comprises a method according to one of the preceding claims: - Means of selecting a routine from a given set of selectable routines (AF) by means of data processing based at least partially on machine learning and on the basis of speech input (S); - Means of parameterizing at least one parameter of the selected routine by means of data processing based at least partially on machine learning, based on speech input and / or a context of the robot application to be created; and - Means of adding the selected and parameterized routine to the robot application. [12] Computer program or computer program product, wherein the computer program or computer program product contains instructions, in particular stored on a computer-readable and / or non-volatile storage medium, which, when executed by one or more computers or a system according to claim 11, cause the computer(s) or system to perform a method according to any one of claims 1 to 10.

Images