Robot teaching model

Abstract

There is a need for more intuitive ways of teaching robots to perform tasks. Provided by the present disclosure is a method for generating robot-instruction code based on multimodal teaching data. The method comprises: obtaining initial multimodal teaching data for teaching a robot to perform a task; passing the initial multimodal teaching data to a multimodal large language model, wherein the multimodal large language model is trained to generate robot-instruction code based on the multimodal teaching data; and generating the robot-instruction code using the multimodal large language model. The method thus enables a higher level of interaction between the user and the robot.

Description

[0001] ABB Schweiz AG 26.08.2024

[0002] A 19254 WO

[0003] ROBOT TEACHING MODEL

[0004] FIELD OF THE INVENTION

[0005] The invention relates to methods and systems for generating robot-instruction code based on multimodal teaching data, and for generating training data for training a multimodal large language model to generate the robot-instruction code.

[0006] BACKGROUND

[0007] Large language models (LLMs) have been used for teaching robots to perform tasks by allowing the user to describe the task in natural language while interacting with the robot physically. Yet there remains a need for more intuitive ways of teaching robots to perform tasks.

[0008] SUMMARY

[0009] Usage of a multimodal large language model for teaching

[0010] There is provided, in a first aspect of invention, a method for generating robotinstruction code based on multimodal teaching data, the method comprising: obtaining initial multimodal teaching data for teaching a robot to perform a task; passing the initial multimodal teaching data to a multimodal large language model, wherein the multimodal large language model is trained to generate robot-instruction code based on the multimodal teaching data; and generating the robot-instruction code using the multimodal large language model.

[0011] Execution and refinement of the code

[0012] The method of the first aspect may further comprise executing the generated robotinstruction code to cause the robot to perform the task. The method of the first aspect may further comprise, following execution of the generated robot-instruction code, P240392W001 - 2 - 26.08.2024 evaluating robot task performance. The method of the first aspect may further comprise, based on evaluation of the robot task performance, obtaining subsequent multimodal teaching data for teaching the robot to perform the task, that is, for refining the robot task performance based on the evaluation.

[0013] Interactive teaching sessions

[0014] Multimodal teaching data may be obtained during teaching sessions. That is, the initial multimodal teaching data may be obtained during an initial multimodal teaching session during which a user teaches the robot to perform the task. Similarly, the subsequent multimodal teaching data may be obtained during a subsequent multimodal teaching session for refining the robot task performance. The method may comprise, during the initial multimodal teaching session or subsequent multimodal teaching session, providing natural language feedback to the user to assist the user in teaching the robot to perform the task. The natural language feedback may for example augment the multimodal teaching data provided by the user. Additionally or alternatively, the natural language feedback may explain what the robot has understood from the multimodal teaching data.

[0015] Model architecture

[0016] The model may use a multimodal architecture to fuse different modalities of input data and generate the code. For example, the multimodal large language model may comprise a plurality of modality-specific encoders, wherein the multimodal teaching data is input to the plurality of modality-specific encoders, and wherein the modalityspecific encoders are configured to produce modality-specific embeddings of the multimodal teaching data. Each encoder of the plurality of encoders may be configured to produce embeddings for a respective modality. The multimodal large language model may comprise a decoder configured to generate the robot-instruction code based on a combination of the embeddings. In an example, the decoder comprises a transformer. The decoder may be configured to generate the robot-instruction code token-by-token. The multimodal large language model may further comprise a multimodal fusion algorithm configured to produce the said combination of the embeddings. The multimodal fusion algorithm may be configured to combine the embeddings by a fusion method such as concatenation.

[0017] Training of the model P240392W001 - 3 - 26.08.2024

[0018] According to a second aspect, there is provided a method for generating training data for training the multimodal large language model used in the method of the first aspect. The method comprises: obtaining preexisting robot instruction code and using the preexisting robot instruction code as target data for the training data; synthesizing multimodal teaching data based on the preexisting robot instruction code; and using the synthesized multimodal teaching data as feature data for the training data.

[0019] Preexisting robot instruction code

[0020] The preexisting robot instruction code may be obtained from a code repository, from a user, or from another source. The robot instruction code may comprise at least one instruction configured to instruct a robot to perform a task or at least part thereof.

[0021] Synthesizing multimodal teaching data

[0022] Synthesizing the multimodal teaching data may comprise deriving at least part of the multimodal teaching data from the robot instruction code and / or capturing at least part of the multimodal teaching data during execution of the robot instruction code. That is, at least part of the multimodal teaching data may be contained in the robot instruction code or may be derivable therefrom using data processing techniques. Synthesizing the multimodal teaching data may comprise executing the robot instruction code and capturing the multimodal teaching data during robot task performance caused by execution of the robot instruction code. Execution of the robot instruction code and capture of the multimodal teaching data during robot task performance may take place in a real or simulated environment.

[0023] Text teaching data

[0024] The multimodal teaching data may comprise text teaching data, which is synthesized by inputting the robot instruction code to a large language model which is trained to derive the text teaching data from the robot instruction code. That is, the large language model may be trained to generate natural language explanations of robot instruction code, wherein the text teaching data is at least partially based on the generated natural language explanation of the robot instruction code. P240392W001 - 4 - 26.08.2024

[0025] In an example in which the multimodal teaching data further comprise video teaching data, the text teaching data may be synthesized by using a visual language model to describe the video teaching data, or at least one frame thereof.

[0026] Audio teaching data

[0027] The multimodal teaching data may comprise audio teaching data. In the case that the multimodal teaching data further comprises text teaching data, the audio teaching data may be synthesized by inputting the text teaching data to a text-to-speech converter.

[0028] Additionally or alternatively, the audio teaching data may be at least partially synthesized by obtaining a verbal description from the user of how to perform the task.

[0029] Video teaching data

[0030] The multimodal teaching data may comprise video teaching data. In an example, the video input data is synthesized by capturing video of robot task performance caused by execution of the robot instruction code.

[0031] Additionally or alternatively, the video input data may be synthesized by capturing video of robot task performance caused by a user, for example in a lead-through programming mode of the robot.

[0032] Trajectory teaching data

[0033] The multimodal teaching data may comprise trajectory teaching data. The trajectory teaching data may comprise at least one pose value or joint value of the robot. The trajectory data may be synthesized by deriving the trajectory teaching data from the robot instruction code and / or by capturing or recording the trajectory teaching data robot task performance caused by execution of the robot instruction code.

[0034] Detected object teaching data

[0035] The multimodal teaching data may comprise detected object teaching data, which is synthesized by capturing video of robot task performance caused by execution of the robot instruction code, inputting the captured video to an object detection algorithm to extract pose data for at least one detected object, and using the extracted pose data as the detected object teaching data. P240392W001 - 5 - 26.08.2024

[0036] Merging and synchronization

[0037] Synthesizing the multimodal teaching data may comprise synchronizing teaching data across modalities. Synchronizing the teaching data may comprise synchronizing capture of teaching data with execution of the robot instruction code. Synchronizing the teaching data may comprise attaching timestamps to teaching data during its capture and using the timestamps for synchronization.

[0038] Iteration

[0039] Steps of the method of the second aspect may be repeated to generate a predetermined amount of training data, that is, an amount which is deemed sufficient for training of the model.

[0040] Training the model

[0041] The method of the second aspect may further comprise using the generated training data to train the multimodal large language model. Training of the model may comprise use of random masking.

[0042] The methods of the first and second aspects may be combined.

[0043] The method of the first and / or second aspect may be computer implemented.

[0044] According to a third aspect, there is provided a computing system configured to perform the method of the first and / or second aspect.

[0045] According to a fourth aspect, there is provided a computer program (product) comprising instructions which, when executed by a computing system, enable or cause the computing system to perform the method of the first and / or second aspect.

[0046] According to a fifth aspect, there is provided a computer-readable (storage) medium comprising instructions which, when executed by a computing system, enable or cause the computing system to perform the method of the first and / or second aspect. The computer-readable medium may be transitory or non-transitory, volatile or non-volatile.

[0047] The computing system can typically comprise a processor, for example a processor that is part of a computer. P240392W001 - 6 - 26.08.2024

[0048] Methods and systems described herein enable a higher level of interaction between the user and the robot, specifically enabling the possibility of having partial demonstrations, merging multiple demonstrations together, showing the robot what needs to be fixed from the resulting motion, and automatic failure recovery through re-prompting of the multimodal large language model. The multimodal large language model not only enables enhanced interaction possibility but further enables the capability to refine robot motion using multimodal robot input with a high level of flexibility and robustness.

[0049] The term “target data” as used herein relates to the expected output or prediction which the model is trained to produce.

[0050] Correspondingly, the term “feature data” as used herein relates the inputs or predictors based on which the model makes predictions.

[0051] By “multimodal teaching data” is meant teaching data comprising two or more of the following: audio (e.g., voice) teaching data; video teaching data; trajectory teaching data; text teaching data; detected object teaching data; any other form of data for teaching a robot to perform a task.

[0052] By “robot instruction code” is meant code which instructs a robot in the performance of a task. The code may comprise final code which can be used directly to control the robot and / or intermediate code which can be used to generate final code. The robot instruction code may comprise RAPID code, for example.

[0053] The term “teaching data” as used herein refers to data which is used to teach the robot to perform a task. The teaching data is used in particular to generate robot instruction code.

[0054] The term “training data” is used herein to refer to data which is used to train a machine learning model.

[0055] The term “obtaining”, as used herein, may encompass receiving from another system, device, or process; receiving via an interaction with a user; loading or retrieving from P240392W001 - 7 - 26.08.2024 storage or memory; measuring or capturing using sensors or other data acquisition circuitry.

[0056] The indefinite article “a” or “an” does not exclude a plurality. In addition, the articles “a” and “an” as used herein should generally be construed to mean “one or more” unless specified otherwise or clear from the context to be directed to a singular form.

[0057] Unless specified otherwise, or clear from the context, the phrases “one or more of A, B and C”, “at least one of A, B, and C”, and “A, B and / or C” as used herein are intended to mean all possible permutations of one or more of the listed items. That is, the phrase “A and / or B” means (A), (B), or (A and B), while the phrase “A, B, and / or C” means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C).

[0058] The term “comprising” does not exclude other elements or steps. Furthermore, the terms “comprising”, “including”, “having” and the like may be used interchangeably herein.

[0059] The invention may include one or more aspects, examples or features in isolation or combination whether specifically disclosed in that combination or in isolation. Any optional feature or sub-aspect of one of the above aspects applies as appropriate to any of the other aspects.

[0060] The above-described aspects will become apparent from, and elucidated with, reference to the detailed description provided hereinafter.

[0061] BRIEF DESCRIPTION OF THE DRAWINGS

[0062] A detailed description will now be given, by way of example only, with reference to the accompanying drawings, in which:-

[0063] FIG. 1 illustrates a method for generating training data for training a multimodal large language model (MLLM) to generate robot-instruction code;

[0064] FIG. 2 illustrates architecture of an MLLM for generating robot-instruction code based on multimodal teaching data; P240392W001 - 8 - 26.08.2024

[0065] FIG. 3 illustrates a method for generating robot-instruction code based on multimodal teaching data; and

[0066] FIG. 4 illustrates a computing system that can be used in accordance with the systems and methods disclosed herein.

[0067] DETAILED DESCRIPTION

[0068] Described herein are the training and usage of an MLLM which receives multimodal teaching data during one or more teaching sessions and which generates code for instructing a robot to perform a task.

[0069] Generating training data

[0070] FIG. 1 illustrates a method 100 for generating training data for training a multimodal large language model (MLLM) to generate robot-instruction code.

[0071] The method 100 begins by obtaining preexisting robot instruction code, which in this non-limiting example comprises RAPID code 102, and using the preexisting RAPID code 102 as expected output (i.e. , target data) for training data 104 to be included in a training dataset 106. The RAPID code 102 may be available in one or more code repositories, in which code samples are available for specific robots as metadata.

[0072] Thereafter, multimodal teaching data is synthesized based on the preexisting RAPID code 102 using the following steps.

[0073] Execution

[0074] The RAPID code 102 is executed by a robot 108 in a real or simulated environment to cause the robot to perform at least one task defined by the code.

[0075] Trajectory teaching data

[0076] As the RAPID code 102 comprises trajectories 110 of the robot 108, the code 102 can be used to extract trajectory teaching data 112 directly therefrom. Additionally or alternatively, trajectory teaching data 112 can be recorded during the robot motion resulting from execution of the code 102, which may provide a more continuous and / or P240392W001 - 9 - 26.08.2024 smoother trajectory. The trajectories 110 may comprise TCP pose and / or joint values of the robot 108 attained during the robot motion.

[0077] Video teaching data

[0078] Using for example an RGB or RGB-D camera (robot mounted or external), video 114 of robot-performed actions can be recorded and used as video teaching data 116.

[0079] Detected objection teaching data

[0080] The recorded video stream can be also used by an object detection algorithm to extract the 6D pose of relevant objects to be used as detected object teaching data, which can be stored alongside the video teaching data. To deal with perception-based tasks in RAPID, a 6D pose estimation system may be employed to detect objects in the robot workspace and produce reference frames to be used as work objects in the RAPID code.

[0081] Text teaching data

[0082] The RAPID code 102 can be input to a large language model 118 and thereby converted into a natural language explanation 120 of the RAPID code 102. This natural language explanation 120 may serve as text teaching data. Additionally or alternatively, a visual language model (e.g., CLIP, GPT-4 Vision) can be used to describe one or more individual frames taken from the video teaching data for use as text teaching data.

[0083] Audio teaching data

[0084] The natural language explanation 120 can be converted by a text-to-speech converter 122 into voice teaching data 124.

[0085] Teaching data of various modalities is merged at 126 and added to the training dataset 106. This may include synchronization of training data 104. In particular, video teaching data 116 and voice teaching data 124 are synthesized in parallel with the execution of RAPID code 102 by the robot 108 so that the teaching data are time synchronous across different modalities. This can be achieved in the following way: execution of the RAPID code 102 is started in parallel with the recording 114 of the video. The execution is stopped at a preselected breakpoint at which the video recording is also P240392W001 - 10 - 26.08.2024 stopped. The RAPID code 102 for this specific timeframe is converted to text and the teaching data of various modalities is furnished with start and end timestamps.

[0086] The above-described steps are iteratively repeated over a plurality of code samples to result in further synthesized multimodal teaching data which is used as feature data for the required training dataset 106.

[0087] Training the model

[0088] Once a predetermined amount of training data 104 is synthesized, this is used to train a multimodal large language model to generate RAPID code based on multimodal teaching data provided by a user, who may record video, trajectories, generate text and / or audio for teaching the robot to perform a task.

[0089] Training the model may comprise pretraining and subsequent finetuning of the model. The model may be pretrained on the basis of the synthesized training data 104 using random masking for improved robustness. In masked training, at the pretraining stage, in order to learn more robust data representations, at each epoch, parts of input modalities are masked at random, such that for each sample, parts of inputs are masked at arbitrary positions, unaligned across modalities. At inference time, masking is not applied, and inputs are used in their entirety.

[0090] Model architecture

[0091] FIG. 2 illustrates architecture 200 of an MLLM for generating robot-instruction code based on multimodal teaching data.

[0092] The user-provided multimodal teaching data for teaching the robot to perform a task may comprise text teaching data 202, voice teaching data 204, video teaching data 206, trajectory teaching data 208, and / or any other suitable form of teaching data 210.

[0093] The multimodal architecture 200 is operable to fuse different modalities of the teaching data and generate corresponding RAPID code. The model architecture 200 comprises a plurality of modality-specific encoders 212-220 configured to produce embeddings of the teaching data in respective different modalities. That is, a text encoder 212 is provided for producing embeddings of the text teaching data 202. An audio encoder 214 is provided for producing embeddings of the voice teaching data 204. A video P240392W001 - 11 - 26.08.2024 encoder 216 is provided for producing embeddings of the video teaching data 206. A time series encoder 218 is provided for producing embeddings of the trajectory teaching data 208. A further encoder 220 is provided for producing embeddings of the at least one other form of teaching data 210. Thus, each encoder 212-220 is of an architecture suitable for the given modality (MLP, CNN, LSTM, Transformer, etc.). Embeddings produced by the individual encoders 212-220 are combined by a multimodal fusion algorithm 222 such as concatenation. The model architecture 200 further comprises a decoder 224 that generates RAPID code based on the combination of the embeddings. The decoder 224 can be of any suitable architecture such as a transformer.

[0094] In operation, the decoder 224 generates RAPID code one token at a time, at each step taking as input both the combined embedding of different modalities and the hidden representation of tokens generated in the previous steps. The model architecture 200 can take in the entirety of the user-provided multimodal teaching data and generate the entire output sequence at one time. Alternatively, given that teaching data across modalities are aligned (time synchronous), the data can be split into segments and, at each step, the model 200 can take in a sub-sequence and output respective RAPID code.

[0095] Usage of the trained model for teaching a robot to perform a task FIG. 3 illustrates a method 300 for generating robot-instruction code based on multimodal teaching data.

[0096] Step C.1 comprises obtaining initial multimodal teaching data for teaching a robot to perform a task. The user interacts with the robot for example by verbally explaining the task, physically moving the robot, and using the camera to detect relevant objects. Initial multimodal teaching data such voice teaching data 204, trajectory teaching data 208 (e.g., robot motion via lead-through or jogging), video teaching data 206 (recording using a gripper-mounted or externally mounted camera), as well as detected object teaching data (obtained for example using pretrained / preprogrammed specialized image recognition software, using an open-vocabulary ViT detector) are thus obtained during this initial multimodal teaching session 302 during which the user teaches the robot to perform the task. P240392W001 - 12 - 26.08.2024

[0097] Step C.2 comprises passing the initial multimodal teaching data to a multimodal large language model 304 and generating robot-instruction code 306 using the multimodal large language model. The model 304 is used for inference by inputting desired values from the user demonstration with multimodal inputs such as the voice teaching data 204, lead-through trajectory teaching data 208, and video teaching data 206). The multimodal large language model 304 has been trained in the manner described herein to generate the robot-instruction code 306 based on the multimodal teaching data. The robot instruction code in this non-limiting example comprises RAPID code for the robot (or another indirect representation which can be parsed at a later stage to RAPID).

[0098] Step C.3 comprises execution 308 of the generated robot-instruction code to cause the robot to perform the task. The user can run the generated RAPID code 306 on the robot using the vision services from step C.1.

[0099] In step C.4, the robot’s performance of the task resulting from the execution of the code is evaluated. Based on the evaluation, the user may then request fixes / changes by way of a subsequent multimodal teaching session 302, for obtaining subsequent multimodal teaching data for teaching the robot to perform the task, that is, for refining the robot task performance based on the evaluation. Execution of the code 306 on the robot facilitates verification that the code 306 was well formulated. Amendments to the code 306 may be effected by providing feedback during subsequent prompting with multimodal inputs, for example requesting fixes to the resulting motion by moving the robot in lead-through to a new location while explaining the fix that needs to be implemented. This subsequent multimodal teaching session may focus on one or more aspects that need improvement / modification. This subsequent multimodal input is combined with the initial multimodal input from step C.1 to prompt the multimodal LLM 304 to output a new version of the RAPID code 306, which can be tested again as in step C.3.

[0100] Step C.5 comprises providing natural language feedback to the user to assist the user in teaching the robot to perform the task. During the initial and / or subsequent multimodal teaching session (step C.1 and C.4), the robot can provide conversational engagement with the user to augment the information that is provided by the user and to explain what the robot understood from the teaching. P240392W001 - 13 - 26.08.2024

[0101] Conclusion

[0102] The present disclosure reenvisages the teaching framework to use an MLLM as centrepiece while providing compatible MLLM architectures with a novel data generation procedure to train the foundation robot teaching model.

[0103] Computer implementation

[0104] FIG. 4 illustrates an exemplary computing system 800 that can be used in accordance with the systems and methods disclosed herein. The computing system 800 may form part of or comprise any desktop, laptop, server, or cloud-based computing system. The computing system 800 includes at least one processor 802 that executes instructions that are stored in a memory 804. The instructions may be, for instance, instructions for implementing functionality described as being carried out by one or more components described herein or instructions for implementing one or more of the methods described herein. The processor 802 may access the memory 804 by way of a system bus 806. In addition to storing executable instructions, the memory 804 may also store conversational inputs, scores assigned to the conversational inputs, etc.

[0105] The computing system 800 additionally includes a data store 808 that is accessible by the processor 802 by way of the system bus 806. The data store 808 may include executable instructions, log data, etc. The computing system 800 also includes an input interface 810 that allows external devices to communicate with the computing system 800. For instance, the input interface 810 may be used to receive instructions from an external computer device, from a user, etc. The computing system 800 also includes an output interface 812 that interfaces the computing system 800 with one or more external devices. For example, the computing system 800 may display text, images, etc. by way of the output interface 812.

[0106] It is contemplated that the external devices that communicate with the computing system 800 via the input interface 810 and the output interface 812 can be included in an environment that provides substantially any type of user interface with which a user can interact. Examples of user interface types include graphical user interfaces, natural user interfaces, and so forth. For instance, a graphical user interface may accept input from a user employing input device(s) such as a keyboard, mouse, remote control, or the like and provide output on an output device such as a display. Further, a natural user interface may enable a user to interact with the computing system 800 in a P240392W001 - 14 - 26.08.2024 manner free from constraints imposed by input device such as keyboards, mice, remote controls, and the like. Rather, a natural user interface can rely on speech recognition, touch and stylus recognition, gesture recognition both on screen and adjacent to the screen, air gestures, head and eye tracking, voice and speech, vision, touch, gestures, machine intelligence, and so forth.

[0107] Additionally, while illustrated as a single system, it is to be understood that the computing system 800 may be a distributed system. Thus, for instance, several devices may be in communication by way of a network connection and may collectively perform tasks described as being performed by the computing system 800.

[0108] Various functions described herein can be implemented in hardware, software, or any combination thereof. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media include computer-readable storage media. Computer- readable storage media can be any available storage media that can be accessed by a computer. By way of example, and not limitation, such computer-readable storage media can comprise FLASH storage media, RAM, ROM, EEPROM, CD-ROM or other optical disc storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Disk and disc, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc (BD), where disks usually reproduce data magnetically and discs usually reproduce data optically with lasers. Further, a propagated signal may be included within the scope of computer-readable storage media. Computer-readable media also includes communication media including any medium that facilitates transfer of a computer program from one place to another. A connection, for instance, can be a communication medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fibre optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fibre optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio and microwave are included in the definition of communication medium. Combinations of the above should also be included within the scope of computer-readable media. P240392W001 - 15 - 26.08.2024

[0109] Alternatively, or in addition, the functionally described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), etc.

[0110] The applicant hereby discloses in isolation each individual feature described herein and any combination of two or more such features, to the extent that such features or combinations are capable of being carried out based on the present specification as a whole in the light of the common general knowledge of a person skilled in the art, irrespective of whether such features or combinations of features solve any problems disclosed herein, and without limitation to the scope of the claims. The applicant indicates that aspects of the present invention may consist of any such individual feature or combination of features.

[0111] It has to be noted that embodiments of the invention are described with reference to different categories. In particular, some examples are described with reference to methods whereas others are described with reference to apparatus. However, a person skilled in the art will gather from the description that, unless otherwise notified, in addition to any combination of features belonging to one category, also any combination between features relating to different category is considered to be disclosed by this application. However, all features can be combined to provide synergetic effects that are more than the simple summation of the features.

[0112] While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered exemplary and not restrictive. The invention is not limited to the disclosed embodiments. Other variations to the disclosed embodiments can be understood and effected by those skilled in the art, from a study of the drawings, the disclosure, and the appended claims.

[0113] The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used advantageously. P240392W001 - 16 - 26.08.2024

[0114] Any reference signs in the claims should not be construed as limiting the scope.

Claims

P240392W001 - 17 - 26.08.2024CLAIMS1. A method for generating robot-instruction code based on multimodal teaching data, the method comprising: obtaining initial multimodal teaching data for teaching a robot to perform a task; passing the initial multimodal teaching data to a multimodal large language model, wherein the multimodal large language model is trained to generate robot-instruction code based on the multimodal teaching data; and generating the robot-instruction code using the multimodal large language model.

2. The method as claimed in claim 1, further comprising executing the generated robot-instruction code to cause the robot to perform the task.

3. The method as claimed in claim 2, further comprising, following execution of the generated robot-instruction code, evaluating robot task performance.

4. The method as claimed in claim 3, further comprising, based on evaluation of the robot task performance, obtaining subsequent multimodal teaching data for teaching the robot to perform the task.

5. The method as claimed in claim 4, wherein the subsequent multimodal teaching data is obtained during a subsequent multimodal teaching session for refining the robot task performance.

6. The method as claimed in any preceding claim, wherein the initial multimodal teaching data is obtained during an initial multimodal teaching session during which a user teaches the robot to perform the task.

7. The method as claimed in claim 5 or 6, further comprising, during the initial multimodal teaching session or the subsequent multimodal teaching session, providing natural language feedback to the user to assist the user in teaching the robot to perform the task.P240392W001 - 18 - 26.08.20248. The method as claimed in any preceding claim, wherein the multimodal large language model comprises a plurality of modality-specific encoders, wherein the multimodal teaching data is input to the plurality of modality-specific encoders, and wherein the modality-specific encoders are configured to produce modality-specific embeddings of the multimodal teaching data.

9. The method as claimed in claim 8, wherein each encoder of the plurality of encoders is configured to produce embeddings for a respective modality.

10. The method as claimed in claim 8 or 9, wherein the multimodal large language model comprises a decoder configured to generate the robot-instruction code based on a combination of the embeddings.

11. The method as claimed in claim 10, wherein the multimodal large language model comprises a multimodal fusion algorithm configured to produce the said combination of the embeddings.

12. A method for generating training data for training a multimodal large language model, the method comprising: obtaining preexisting robot instruction code and using the preexisting robot instruction code as target data for the training data; synthesizing multimodal teaching data based on the preexisting robot instruction code; and using the synthesized multimodal teaching data as feature data for the training data.

13. The method as claimed in claim 12, further comprising using the generated training data to train the multimodal large language model.

14. A computing system configured to perform the method as claimed in any preceding claim.

15. A computer-readable medium comprising instructions which, when executed by a computing system, cause the computing system to perform the method as claimed in any of claims 1-13.

Images