Apparatus and method for natural language controlled industrial assembly robot

Abstract

Apparatus and methods for a natural language controlled industrial assembly robot are provided. A computer-implemented method of determining an action for controlling a robot, in particular an assembly robot, comprises the steps of receiving first and second inputs, where the first input (101) is a statement describing an action that the robot should perform, where the second input (102) is an image of a current state of an environment of the robot. The first input (100) is fed into a first machine learning model (101) and the second input (102) is fed into a second machine learning model (103), where the first and second machine learning models (101, 103) are configured to determine the lexical elements of their respective inputs. The lexical elements are fed into a third machine learning model (104), where the third machine learning model (104) outputs two outputs, where the first output is a switch for incorporation into the specialized skill network and the second output is an action (106).

Description

Technical Field

[0001] The invention relates to a method for determining actions for controlling a robot, in particular an assembly robot, based on natural language prompts and based on an image of the robot's environment. Background Art

[0002] Large language models (LLMs) and transformer-based vision models have recently enabled rapid progress in the field of vision-language-action models for robot control. The main goal of these methods is to develop a general policy that can control a robot in a variety of environments. Strategies are well known in the field of reinforcement learning.

[0003] However, in industrial robotics applications such as automated assembly and disassembly, some tasks, such as insertion, require higher precision and involve complex factors such as contact engagement, friction handling, and fine motor skills. Implementing these skills using general-purpose strategies is challenging because the strategies may integrate additional sensory data, for example, including force or torque measurements, to improve accuracy.

[0004] An object of the present invention is to provide a locally accurate and globally robust control strategy for a limited skill set that is specially trained to perform high-precision tasks.

[0005] Advantages of the present invention

[0006] In industrial robotic assembly, some tasks, such as insertion, require higher precision and involve complex factors such as contact engagement, friction management, and fine motor skills. The advantage of this approach lies in the introduction of a global control strategy based on LLMs, which is able to transfer control policy to a limited skill set specifically trained to perform high-precision tasks through dynamic context switching. The ability to independently support skill models is also a benefit. Furthermore, the integration of LLMs facilitates not only the translation and processing of language input but also the development of control mechanisms for diverse and complex robotic operations. Summary of the Invention

[0007] In a first aspect, according to claim 1 a computer-implemented method for determining actions for controlling a robot, in particular an assembly robot, is proposed. Advantageously, the method is applied to assembly processes that require a sequence of skills. This is advantageous because the proposed method supports the prediction of complex assembly processes that require a sequence of skills. To this end, after completing a skill, the model can predict a new skill as a continuation of the process, which new skill is similarly derived from the two inputs described (text, image). If the current state does not encapsulate all the required information for the purpose of inferring the next skill (i.e. the system is Markovian), then not only the previous image can be added as input, but also additional images from history, which, for example, have been previously stored.

[0008] The method begins by receiving first and second inputs, where the first input is a statement describing the robot's (assembly) task. The statement can be a natural language statement, also known as a natural language prompt. The task can be described by defining a goal or (general) action that the robot should achieve or perform. The second input is a sensor output, particularly an image, representing the current state of the robot's environment.

[0009] Subsequently, the first and second inputs are fed into the first and second machine learning models, respectively. The first and second machine learning models are configured to determine tokens of their respective inputs. A token is a low-dimensional machine-understandable representation of the input and may also be referred to as an embedding.

[0010] Subsequently, the determined word-grams of the first and second machine learning models are grouped together, also called concatenated.

[0011] Subsequently, the concatenated word units are fed into a third machine learning model. The third machine learning model includes a s ) and the move action (a m ) two strategies.

[0012] Skill Action (a s ) is a classification or categorization of high-level categories of actions. Preferably, the categories of skills are independent of each other. Skill actions (a s ) can be understood as an identifier or pointer to the category of skills or actions that will achieve the best results from the skill set. Note that the skills are not conditional on text instructions. Skill Action (a s ) can be used directly to obtain the appropriate skill or the method behind the skill to be executed. The high-level categories represent general actions that the robot can perform based on the current state of the environment in order to complete its task according to the language prompt.

[0013] The skill action is determined based on the concatenated word-grams. In the case of using spoken word-grams, it is preferred that the skill action is determined based only on the output spoken word-grams.

[0014] Move action (a m ) is the robot's explicit movement proposal. A movement action is movement information indicating the direction the robot should move, and preferably the distance the robot should move.

[0015] Then, based on the skill action (a s ), determines the move action (a m ) is output as the robot's action, or is a moving action (a m ) More precise move proposals. The external source can be a database that includes methods for determining precise move proposals for corresponding high-level categories. The methods can be dedicated neural networks for their corresponding skills.

[0016] Preferably, the method of the first aspect of the present invention is used for a language-driven robotic industrial assembly solution.

[0017] The proposed external sources include a set of expertise, where an expertise is a method configured to provide move proposals for a specific skill, where the expertise is provided with additional sensory input of the robot's current state and the robot's environment.

[0018] In addition, the proposed skill action (a s ) includes a list of different high-level action categories, where these high-level action categories are: according to the movement action (a m ) and termination and movement of different predefined specialized skills. The corresponding strategy for skill actions can determine the logits of each category. The decision to take action can be made by taking the category with the maximum or minimum logits. Preferably, specialized skills provide higher precision and involve complex factors such as contact engagement, friction handling, and fine motor skills.

[0019] Furthermore, it is proposed to add a readout word during word concatenation.The readout word can be determined by an additional transformer model having, for example, at least one conventional transformer layer. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Embodiments of the present invention will be discussed in more detail with reference to the following drawings. These drawings show:

[0021] Figure 1 is a schematic framework of an embodiment of robotic assembly control;

[0022] Figure 2 is a schematic centralized controller architecture;

[0023] Figure 3 It is the control system that controls the manufacturing machines;

[0024] Figure 4 It's a training system. DETAILED DESCRIPTION

[0025] Figure 1 We present a user-friendly industrial assembly framework based on natural language prompts and LLMs that can be easily adapted to new environments and is modular in terms of adding or adjusting components to meet specific needs. The modular nature of the system is evident in both its inputs and outputs. Regarding input flexibility, the model is designed to support various types of specialized neural network models.

[0026] Figure 1 The framework shown presents a method for processing natural language-based goals from an assembly line, such as "carefully grasp the yellow plug and insert it into the appropriate socket" (100), to output control instructions based on image observation. In principle, the framework includes a centralized goal-conditioning model (104) that initiates easily acquired independent specialized skills (105) by handing over control to specialized models based on context switching. The centralized module (104) is responsible for inferring the skill to be performed and moving the robot to an initial pose so that the specialized control model (skill model) can perform precise manipulation. Specifically, the centralized model (104) can output two control signals, namely (1) a 6-degree-of-freedom (6DoF) pose for the default "move" action, and (2) a skill-based context class.

[0027] The "skills" mentioned above can be small pre-trained networks that specialize in fine-grained control tasks such as insertion. Note that other production tasks after insertion are also possible. The fine-grained skill "crawl for insertion" will be used below as an example of a specialized skill. The framework is modular in terms of text and image encoders and the type of strategy used (which can be a simple multi-layer perceptron). In addition, the framework is modular in terms of specialized skills by simply fine-tuning the context classifier and calling the relevant strategy to execute.

[0028] A centralized control model (104), discussed below, employs a transformer architecture that can skillfully switch between specialized control skills from a predefined set, guided by natural language goals (100) and vision-based input (102).

[0029] The centralized controller performs two main functions:

[0030] a. Guide the robot to a specified location based on text prompts.

[0031] b. Based on the textual prompt and the current state of the robot, identify and predict the necessary specialized skills, such as grasping or inserting.

[0032] The first function, which can be called a general "mobility" skill, does not require highly accurate 6-degree-of-freedom (6DoF) pose estimation. The specialized tasks mentioned in the second function require higher precision and involve complex factors such as contact bonding, friction handling, and fine motor skills. In addition, they may integrate additional sensor data, including force or torque measurements, to improve accuracy. Unlike the core model, these specialized skills are developed independently, for example, using data tailored specifically for their needs.

[0033] These special skills can be assumed to work precisely, assuming the robot meets certain constraints, such as being placed in an initial position in the vicinity of the object being manipulated. Goals can be specified using natural language prompts, such as, "Carefully grasp the yellow plug and insert it into the appropriate outlet."

[0034] The transformer model accepts language instruction tokens (101) encoded by a strong language model (101), such as T5 (http: / / arxiv.org / abs / 1910.10683), BLIP (http: / / arxiv.org / abs / 2201.12086.), and CLIP (http: / / arxiv.org / abs / 2103.00020.), which are pre-trained, frozen, and specialized in text encoding, and generates text instruction tokens. In addition, it is proposed to use a pre-trained visual encoder (103), such as ResNet-50 or ViT (http: / / arxiv.org / abs / 2010.11929.), to generate visual tokens embedded with information from the observations. Preferably, the input is padded with learnable "readout tokens" as described in Octo (https: / / octo-models.github.io.). The transformer can implement a Markov strategy where the action depends only on the current observation and is independent of past observations. Consistent with the dual-purpose model, the actions can be divided into two categories: denoted as a s The skill action, which is related to the type of skill being performed, and the m The movement action is related to the movement skill. The problem can be formally defined as follows:

[0035] a s =π s (s),

[0036] a m =π m (s),

[0037] where s is the state vector, which encodes information about the current state (image(t)) and general textual cues.

[0038] Preferably, the two strategies (π s (s),π m (s)) largely share their weights and architecture, but differ in their decoder models. Both mentioned strategies can be deterministic and based on a multilayer perceptron (MLP) architecture. Strategy π s Acts as a high-level controller to predict the required skills by classifying predefined skills as follows:

[0039] 0. Termination

[0040] 1. Mobile (handled by a centralized controller)

[0041] 2. Skill 1 (Specialized)

[0042] 3. Skill 2 (Specialized)

[0043] 4. Skill 3 (Specialized)

[0044] 5. Etc.

[0045] n.Skill n (special)

[0046] "Terminate" indicates that the robot has reached its goal following the provided text cue. s = skill n, control is transferred to the model specializing in skill n. s When predicting the “move” skill (denoted as “1”), the low-level controller’s (π m ) action. Additional specialized skills can be integrated by adding another context class and fine-tuning the classification head with data relevant to the new skill. Figure 2 The strategy (200) of the transformer is schematically shown. The input to the transformer may be given by previously determined tokens. The transformer may output transformed tokens. Preferably, the transformed readout tokens are propagated through an MLP regressor and an MLP classifier to obtain skills and movement actions. In the case where no readout tokens are used, the regressor and classifier are applied to the language and / or observation tokens.

[0047] a m The action space of can be defined as a 7-dimensional vector, which is trained to predict the unit vector in the direction of the delta ground truth of the desired object or task using MSE loss. It is formulated as:

[0048] ΔP=[Δx,Δy,Δz,ΔR x ,ΔR y ,ΔRz ,g]

[0049] In this formulation, Δx, Δy, Δz represent the translation components, and Δr x ,Δr y ,Δr z represents the orientation component expressed in terms of axis angle, and g corresponds to the opening of the gripper. The mean square error (MSE) (L mse ) is trained in a supervised manner. ΔP is output by the centralized controller (104) as the robot's action (106) and can be used directly to control the robot. Typically, the robot may include a gripper.

[0050] The activity domain can be defined as the region where the specialized skill can be successfully executed. The boundary of this activity domain is assumed to be an abstraction threshold ε. This threshold varies for different skills and does not depend solely on distance. For example, when guiding the grasping of a plug to be inserted into a socket, if the position of the plug becomes unfavorable for insertion, the context should revert to the "grasping for insertion" specialized skill. The classifier head is trained to estimate this abstraction threshold and promote context switching accordingly. The classifier is trained using the categorical cross entropy loss (L ce ) to train the multi-class classifier head.

[0051] Figure 3 , an embodiment is shown in which a control system 40 is used to control a manufacturing machine 11, such as a robot or a specialized robot (such as a solder mounter, punch, cutter or gun drill), of a manufacturing system 200, eg as part of an assembly line.

[0052] The control system 40 controls the actuator unit 10 , which in turn controls the manufacturing machine 11 .

[0053] The sensor 30 may be an optical sensor that captures, for example, properties of the manufactured product 12. The actions 106 determined by the control system 40 may be applied to control the actuator unit 10 of the manufacturing machine 11, which may then be controlled depending on the actions 106 to perform a manufacturing step on the manufactured product 12a, 12b.

[0054] Figure 4 Shown is an embodiment of a training system 500. The training device 500 comprises a provider system 51 that provides input from a training dataset. The input is fed to Figure 1 The training is performed by a framework 52 which determines the output variables from the inputs. The output variables and the input image are fed to an evaluator 53 which determines the hyperparameters from them, which are transferred to a parameter memory P where they replace the current parameters.

[0055] The processes performed by training device 500 may be implemented as a computer program stored on machine-readable storage medium 54 and executed by processor 55 .

Claims

1. A computer-implemented method for determining actions (106) for controlling a robot, in particular an assembly robot, comprising the following steps: Receiving first and second inputs, wherein the first input (100) is a statement describing a robot task, and wherein the second input (102) is a sensor output, in particular an image, representing a state of the robot's environment; Feeding the first and second inputs (100, 102) into first and second machine learning models (101, 103), respectively, wherein the first and second machine learning models (101, 103) are configured to determine word-grams of their respective inputs; Concatenating the determined word-grams of the first and second machine learning models (101, 103); The concatenated word-grams are fed into a third machine learning model (104), wherein the third machine learning model (104) includes a model configured to output skill actions (a s ) and the move action (a m ) two strategies, where the skill action (a s ) characterizes the classification of different high-level action categories of the robot, and the mobile action (a m ) is the robot’s explicit move proposal; Based on skill actions (a s ) determines whether to move the action (a m ) is output as an action (106), or is it based on a skill action (a s )'s high-level action category is determined to be higher than the moving action (a m ) A more precise robot movement is proposed as an action (106).

2. A method according to claim 1, wherein the external source includes a set of expertise (105) for different high-level action categories, wherein the expertise (105) is a method configured to provide movement proposals for the corresponding high-level action category based on the state of the robot's current environment, wherein in particular, the expertise (105) is provided with additional sensory input of the robot's current state and the state of the environment.

3. The method according to claim 1 or 2, wherein the first machine learning model (101) is a pre-trained large language model (LLM) and the second machine learning model (103) is a pre-trained visual encoder.

4. The method according to any one of the preceding claims, wherein the third machine learning model (104) is a transformer model, and the two strategies share the transformer model as a basis and differ in the method used to output the movement action (a m )'s return head and for outputting skill actions (a s )’s classification header.

5. A method according to any one of the preceding claims, wherein the skill action (a s ) includes a list of different high-level action categories, where the high-level action categories are: according to the movement action (a m ) and termination, movement of different predefined expertise.

6. A method according to any one of the preceding claims, wherein during the concatenation of word-grams, additional read-out word-grams are added.

7. A method according to any one of the preceding claims, wherein new expertise is added from an external source, wherein the skill action (a s ) is extended by additional categories for new specialized skills, where the third machine learning model (104) targets skill actions (a s )’s strategy is retrained through fine-tuning.

8. The method according to any of the preceding claims, wherein depending on the action (106) a control signal for the robot is determined, wherein preferably the robot is controlled by the control signal to perform the action (106).

9. The method according to any of the preceding claims, wherein the robot is a manufacturing machine (200) or an assembly robot.

10. A computer program configured to cause a computer to perform the method according to any one of claims 1 to 9 and all the steps thereof if the computer program is executed by a processor.

11. A machine-readable storage medium having stored thereon the computer program according to claim 10.

12. A system configured to perform the method according to any one of claims 1 to 9.

Images