Human-in-the-Loop Robot Training and Generative Artificial Intelligence (AI) Testing System

Abstract

This invention discloses a robot teaching and testing system and method for performing human-operated robot tasks according to instructions generated from a generative AI model. The process begins with a user prompt, which is combined with a predetermined prompt template to generate a well-formatted text prompt. The generative AI model takes the text prompt and converts it into high-level instructions or control code that can be deployed on the robot. The high-level instructions are then converted into human-operated robot tasks for a human data collector using a mixed reality (MR) device. The human data collector attempts to complete the human-operated robot task according to the instructions, can override the proposed instructions by performing different actions, can perform a task without instructions, or can leave feedback or comments on the task. The feedback data is captured and stored to improve the robot system.

Description

Technical Field

[0001] Cross - reference to related applications This application claims priority to U.S. Provisional Patent Application No. 63 / 472,144, filed on June 9, 2023, U.S. Provisional Patent Application No. 63 / 583,733, filed on September 19, 2023, and U.S. Provisional Patent Application No. 63 / 598,292, filed on November 13, 2023, and these U.S. Provisional Patent Applications are incorporated herein by reference in their entirety.

[0002] Field of the Invention The present invention relates to human - in - the - loop robot training (robot training, learning) and testing by generative artificial intelligence (AI).

Background Art

[0003] In order to control a robot in performing various robot control tasks, it is often necessary to train one or more machine learning models and other software used in a robot control system. However, it can be difficult to obtain sufficient data for training machine learning models and other software in an easy, scalable (extensible), and cost - effective way.

[0004] Generative artificial intelligence (AI) models, including but not limited to large language models (LLMs) and large - scale vision models, have shown potential in ways of creating instructions such as software programs for controlling robots. However, the execution of high - level instructions generated by LLMs heavily relies on low - level libraries such as computer vision libraries, motion planning libraries, and motion execution libraries. If one of the low - level libraries lacks the ability to perform a certain task, the entire program will malfunction.

[0005] For example, if a low-level computer vision library is not trained to detect avocados, it will fail the mission of picking up an avocado and placing it in a bowl. Furthermore, under the framework of such AI models that generate robot instructions, there is no clear way to improve the robot's performance. User adjustments to prompts for such AI models do not affect the execution of the low-level library.

[0006] In particular, detecting the root cause of all malfunctions in complex robotic movements involving multiple steps is equally challenging when generating robot commands using AI models. Typically, users must monitor the entire execution process or analyze log files to identify the cause of malfunctions. Users can only detect one cause at a time, and execution may fail at multiple steps, resulting in an inefficient, time-consuming, and costly process. [Overview of the project] [Problems that the invention aims to solve]

[0007] Therefore, current technology presents a need to provide systems and methods that enable the training of machine learning models and other software used in robot control systems in an easy, fast, and cost-effective manner. [Means for solving the problem]

[0008] A preferred example disclosed herein introduces a human-in-the-loop method for solving these problems. Instead of deploying high-level instructions generated from LLM onto the robot, the novel technology disclosed herein provides a framework for improving the robot system by collecting data on human behavior in the real world. The proposed method first converts high-level instructions into human-operated robot tasks, generates instructions for each task, and deploys these instructions onto mixed reality (MR) based on an instruction feedback system. A human data collector performs these tasks using a human-machine operation interface according to these instructions. The cause of malfunctions can be directly identified during this process. Once the cause of malfunctions in the corresponding library / algorithm is identified, data on human behavior and human feedback is stored to further improve the low-level library / algorithm.

[0009] Preferred examples disclosed herein relate to robot teaching and teaching systems that perform human-operated robot tasks by instructions generated from a generative AI model. The process begins with a user prompt, such as a text prompt, a graphic prompt (e.g., a graph or drawing), a voice prompt, etc. The teaching and testing system combines the user prompt with a predetermined prompt template to generate a well-formatted text prompt. The generative AI model takes the text prompt and converts it into high-level instructions or control code that can be deployed on the robot. The high-level instructions are then converted into human-operated robot tasks.

[0010] Human-operated robot tasks are sent to a teaching and feedback system, where a human data collector can use a mixed reality device to focus on visual commands in text or virtual marker format, and / or hear voice commands from an audio device specifying how to perform the human-operated robot task. The human data collector attempts to complete the human-operated robot task one by one according to the commands. In this process, the human data collector can override the proposed commands by performing different actions, demonstrate the task without commands, or leave feedback or comments on the task. The feedback data is captured and stored to improve the robot system.

[0011] In one preferred example, a system for testing and / or training a robot control system includes at least the following elements: 1) a visual sensing device, 2) an MR device, 3) a data acquisition device, 4) a computer device including a machine learning model or other software to be tested and / or trained, and 5) a storage device for recording sensing (detection) data and control data collected during testing and / or training.

[0012] In some preferred examples, the visual sensing device may include multiple cameras, such as a depth camera, a bird's-eye camera, and / or other cameras or various visual sensors as needed, with the depth camera positioned on the data acquisition device and the bird's-eye camera positioned on the wall or ceiling of the training location. The various cameras may be configured to detect the pose of the person performing the test and / or training, the pose of the data acquisition device, such as a human-machine interface, and other end effectors, such as a hand or glove, and / or other objects being manipulated by the data acquisition device.

[0013] In some preferred examples, the MR device is a platform that enables a human data collector to communicate with various other elements of the test and / or training system. The human data collector also receives and visualizes commands and feedback from a computer device via the MR device, and the computer device instructs the human data collector to perform various human-operated robotic tasks related to testing and / or training machine learning models.

[0014] In some preferred examples, the data acquisition device may include a human-machine interface, which is worn by a human data collector and used to perform human-operated robotic tasks related to testing and / or training machine learning models or other software. The data acquisition device may include various cameras and sensors, which are configured to collect data related to performing human-operated robotic tasks.

[0015] In some preferred examples, the data acquisition device may be equipped with a forearm-mounted human-machine interface, which is used to operate one or more robotic grippers or robotic hands in performing complex grasping and manipulation human-operated robotic tasks. In other preferred examples, the data acquisition device may be equipped with a palm-mounted human-machine interface, which is used to operate one or more robotic grippers or robotic hands in performing complex grasping and manipulation human-operated robotic tasks.

[0016] In yet another preferred example, the hands and / or arms of a human data collector can be considered as the data acquisition device. In such a preferred example, a visual sensing device mounted on the wall or ceiling of the test and / or training location can track the hands and / or arms of the human data collector in performing human-operated robotic tasks of grasping and manipulating. In an additional preferred example, the data acquisition device may comprise a sensing grab such as a motion capture grab or a hand pose tracking device. In yet another preferred example, the data acquisition device may comprise any combination of a human-machine operation interface, the hands and / or arms of a human data collector, a sensing grab, or a hand pose tracking device.

[0017] In some preferred examples, the computer device monitors the real-time synchronization of multiple data resources (data sources), data processing, and data visualization by providing commands (instructions) to other elements of the test and / or training system and receiving collected data from the other elements. As described above, the computer device may include machine learning models and other software being trained and / or tested. In some embodiments, the computer device includes processing power that enables the computer device to run the machine learning models and other software and to train and / or test the machine learning models and other software.

[0018] The memory device includes memory capacity to store data collected from other elements of the test and / or training system, and then use this data to train and / or test the robot control system.

[0019] These and other features, aspects, and advantages of the present invention will be better understood through the following description, the attached claims, and the attached drawings. [Brief explanation of the drawing]

[0020] [Figure 1] A diagram showing the user-in-the-loop method of applying generative AI in robotic applications. [Figure 2] A diagram showing how the performance of a robot can be improved by comparing the user-in-the-loop method with the human-in-the-loop method according to an embodiment of the present invention and using the human-in-the-loop method according to the embodiments disclosed herein. [Figure 3] A diagram showing the structure of a system for implementing the human-in-the-loop method according to the embodiments disclosed herein. [Figure 4] A diagram showing the workflow proposed by the human-in-the-loop method in robotic applications according to the embodiments disclosed herein. [Figure 5] A diagram showing an example of a prompt according to the embodiments disclosed herein. [Figure 6] A diagram showing an example of high-level instructions or code generated from a generative AI model according to the embodiments disclosed herein. [Figure 7] A diagram showing an example of a list of human-operated robot tasks converted from high-level instructions or code according to the embodiments disclosed herein. [Figure 8] A diagram showing an example of a list of human-operated robot tasks displayed in an MR device according to the embodiments disclosed herein. [Figure 9] A diagram showing an example of human-in-the-loop task execution. [Figure 10] A diagram showing an example of a visual marker displayed in an MR device, indicating an object of interest. [Figure 11] A diagram showing an example of a visual marker displayed in an MR device, indicating a position and the path towards this position. [Figure 12] A diagram showing the human-in-the-loop method of applying a generative AI model in robotic applications.

Embodiments for Carrying Out the Invention

[0021] The drawings are not necessarily drawn to scale. Instead, these drawings are intended to provide a better understanding of their components and to offer suitable examples, without limiting their scope.

[0022] Detailed description of various embodiments A better understanding of the different embodiments of the present invention can be obtained by reading the following description in conjunction with the drawings, in which similar reference letters refer to similar elements. While the present invention is prone to various modifications and alternative configurations, specific exemplary embodiments are shown in the drawings and described below. However, it should be understood that there is no intention to limit the present invention to any particular embodiment disclosed; on the contrary, the aim is to cover all modifications, alternative configurations, combinations, and equivalents that fall within the spirit and scope of the invention.

[0023] The reference numbers used are provided for convenience only and therefore do not define the scope of protection or embodiments. Unless a term is explicitly defined in this application to have the meaning described herein, it is understood that there is no intention to limit the meaning of such terms, explicitly or indirectly, beyond their plain or ordinary meaning. Any element in the claims that does not explicitly describe a “means” or a “step” performing a particular function should not be construed as a “means” or “step” provision in 35 U.S.C. § 112.

[0024] Figure 1 illustrates a user-on-the-loop method for applying generative AI to robotic applications such as deploying robots to perform one or more tasks. This user-on-the-loop method first combines a user prompt 102 with a given task-prompt template library 104 to generate a suitable prompt 106 to feed into a generative AI model 108, such as ChatGPT (registered trademark).

[0025] The user prompt 102 is an input, usually text or voice, that instructs the generating AI model 108 to generate a response. In the method illustrated in Figure 1, the user prompt 102 interacts with the generating AI model 108 to create a program, which, when executed on the target robot 114, causes the robot 114 to move an object from one position to another. The task-prompt template library 104 contains various templates that can be used to help create programs. For example, a template for moving a particular object may include information such as movement in the XYZ plane that can be used in the program.

[0026] The prompt 106, resulting from applying one or more templates from the task-prompt template library 104 to the user prompt 102, interacts with the generating AI model 108 to create a program that is deployed on the target robot 114 to move an object. The generating AI model 108 then creates high-level instructions 110, which include high-level control code or high-level executable code, such as Python code, where such code focuses on logic rather than execution. Thus, high-level instructions focus on the entire process that the robot will execute, while low-level instructions focus on individual steps within that process.

[0027] Next, a high-level command 110 is deployed (112) onto the target robot 114 and, when executed, triggers the robot 114 to perform one or more tasks defined by the generated AI model 108, which, in the illustrated manner, are to grasp and move an object. During task execution, the high-level command 110 can call predetermined low-level libraries 116, such as the computer vision library 116A, the motion planning library 116B, and the motion execution library 116C, and when the low-level libraries 116 are executed, the low-level libraries 116 can be used to move the object to the target robot 114.

[0028] There are problems associated with the user-on-the-loop method illustrated. First, the user-on-the-loop method does not provide a push for improving the method. Furthermore, it is difficult to identify the cause of the failure, especially when multiple steps of the task may have failures in some scenarios. The user-on-the-loop method can only identify one faulty step at a time. Similarly, the user is limited to changing prompts as a way to fix any failures. In most cases, the user can only guess the cause of the failure and must use this guess to try to correct any prompts supplied within the generating AI model. The user-on-the-loop method may not work unless the AI ​​model and algorithms are improved.

[0029] Figure 2 illustrates the application of the user-on-the-loop method described to real-world robot applications. In each application, a prompt 106 is supplied to the generating language model 108 to generate high-level instructions or code, which, when executed, cause the target robot 114 to perform various tasks. In application 202, the target robot 114 can perform all three steps revealed by the check mark 203.

[0030] However, in application 204, the target robot fails to execute step 2, which is revealed by "X" 205. In such cases, the user does not have an easy way to identify what caused the failure in step 2. All the user can do is observe the process, focus on the malfunction, guess what caused the malfunction, and then generate an updated prompt 206 based on this guess, which attempts to cause the generating AI model 108 to generate a high-level instruction or code, which, when executed, causes the target robot 114 to properly execute step 2. As mentioned above, the cause of the malfunction is a guess, which leads to inaccuracy and inefficiency in dealing with the malfunction.

[0031] In contrast, the human-in-the-loop method according to the embodiments disclosed herein translates high-level instructions into human-operated robot tasks and deploys them on a teaching-test system. A human data collector wears a robotic device and completes tasks according to the instructions provided by the teaching-test system. In the event of a malfunction, the human data collector knows in real time which step and which instruction caused the malfunction. This allows the human data collector to overwrite the incorrect instruction, leave feedback or notes, and move forward without having to restart the entire test or training from the beginning.

[0032] Furthermore, the human-in-the-loop method according to the disclosed embodiments allows a human data collector to identify the causes of multiple malfunctions in different steps. Similarly, for certain novel behaviors that trained AI models and algorithms cannot achieve, a human data collector can simply demonstrate the behavior, thereby collecting data to train the AI ​​model or improve the algorithm.

[0033] Figure 3 shows the structure of system 300, which implements the human-in-the-loop method of the present invention. System 300 may include the following three main components: a teaching-test system 302, an instruction feedback system 304, and a human-machine operation interface 306.

[0034] The teaching-test system 302 may comprise a software subsystem 308 and a hardware subsystem 310. The hardware subsystem 310 may comprise a sensing system 312 and a computer device 314. The sensing system 312 may comprise various different types of cameras such as RGB (red, green, blue) cameras, IR (infrared) cameras, depth cameras, QR (quick response) markers and ArUco (augmented reality library from the University of Cordoba) markers, motion capture systems such as Optitrack (registered trademark), and other types of sensors.

[0035] Computer device 314 can host computer computation tasks such as running an AI model interface, computer vision algorithms, capturing and processing audio signals, running a code-task interpreter and task completion check software, and running support software such as OptiTrack® software. Computer device 314 may comprise one or more processors and associated systems such as a desktop, laptop, wearable computer device (e.g., in a backpack), and a computer device inside the MR device 340.

[0036] The software subsystem 308 may comprise the following components: a task prompt template library 316, a generative AI model or interface 318, another generative AI model 320, a code-task interpreter 322, task completion check software 324, a memory system 326, and support software for the sensing system within the hardware subsystem. The elements of the software subsystem 308 and the hardware subsystem 310 are described in more detail below. The code-task interpreter 322 can also receive additional input other than computer code and interpret this input into a human-operated robot task, which is described in more detail below.

[0037] System 300 includes an interaction feedback system 304. The interaction feedback system 304 may comprise an instruction feedback software subsystem 330. The software subsystem 330 may include a virtual marker display 322, an instruction display 334, and a feedback collection 336. System 300 may comprise an instruction feedback hardware system 338. The instruction feedback hardware system 338 may comprise an MR device 340, other display / sound devices 342, a computer device 344, and an optical marker 346. The MR device 340 in the embodiments described may comprise a virtual reality / augmented reality (VR / AR) device or other human-usable interface for providing mixed reality viewing to a person, for example, mixed reality glasses. In embodiments, the MR device 340 may comprise a platform that a human data collector can use to communicate with other elements of System 300. The MR device 340 may comprise a bidirectional interface.

[0038] System 300 may include a human-machine operation interface 306. In some embodiments, the human-machine operation interface 306 may include a forearm-mounted human-machine operation interface, which is used to operate one or more robotic grippers or robotic hands to perform complex grasping and manipulating tasks. The forearm-mounted human-machine operation interface may include a forearm stabilizer platform attached to the forearm of a human data collector. A gripper support arm may have a first end connected to one end of the forearm stabilizer platform. A gripper coupling member may be connected to a second end of the gripper support arm. The gripper coupling member can connect one or more robotic grippers or robotic hands to the forearm-mounted human-machine operation interface, thereby allowing the data collector to easily operate one or more robotic grippers or robotic hands.

[0039] The grip handle can be connected to a gripper support arm to provide additional support. The grip handle can accommodate at least one input interface, which can receive user input and provide appropriate control commands to a microcontroller unit to control the operation of one or more robotic grippers or robotic hands. The forearm-mounted human-machine operation interface, and / or one or more robotic grippers and robotic hands, can be equipped with various sensors and control signals used for data acquisition. The acquired data can be provided to a wearable computer subsystem for recording.

[0040] In some embodiments, the human-machine operation interface 306 may include a palm-mounted human-machine operation interface, which is used to operate one or more robotic grippers or robotic hands to perform complex grasping and manipulating tasks. The palm-mounted human-machine operation interface may include an interface body and a palm support coupled to the interface body. A gripper coupling member may be coupled to the interface body. The gripper coupling member can connect one or more robotic grippers or robotic hands to the palm-mounted human-machine operation interface, thereby allowing the data collector to operate one or more robotic grippers or robotic hands.

[0041] The palm-mounted human-machine interface may include at least one input interface, which receives user input and provides appropriate commands to a microcontroller unit to control the operation of one or more robotic grippers or robotic hands. The palm-mounted human-machine interface, and / or one or more robotic grippers and robotic hands, may include various sensors and control signals used for data acquisition. The acquired data can be provided to a wearable computer subsystem for recording.

[0042] Figure 4 shows a workflow 400 of one embodiment of the human-in-the-loop method using the system described in relation to Figure 3. In the illustrated workflow, a user prompt 402 is combined with a predetermined task-prompt template library 316 to generate an appropriate prompt 404, which is then supplied into the generating AI model 318. The generating AI model 318 creates a high-level instruction 406, which can, but is not limited to, Python® code in the manner described in relation to Figure 1.

[0043] In some embodiments, generating appropriate prompts may include, for example, creating state and environment descriptions using a computer vision model. The state and environment descriptions may specify constraints or requirements related to the robot task corresponding to the prompt, describe the environment in which the robot task is performed, and / or describe the current state of the robot or robot system. For example, the state and environment descriptions may specify the weight, size, and shape of the object to be moved, describe the size and shape of the area related to the robot task and any obstacles or hazards that need to be avoided, and / or describe the current position and orientation of the related object and the robot or robot system.

[0044] Instead of deploying high-level instructions onto a target robot, the human-in-the-loop method according to the present invention translates high-level instructions 406 into a human-operated robot task 408 using a code-task interpreter 322. In various embodiments disclosed herein, a human-operated robot task can be considered as instructions for a human data collector operating the robot. Thus, the human data collector receives the instructions and can then use the human-machine interface 306 to operate the robot or at least a part of the robot, such as a robot gripper. Thus, the task is a human-operated robot task.

[0045] In some embodiments described herein, a human-operated robot task can be considered as the performance of a simple skill, such as grasping an object or moving it to a predetermined location. A robot mission based on user prompts can be divided into several steps, each of which can be translated into a human-operated robot task deployed on the command feedback system 304.

[0046] A human data collector 412 equipped with a teaching-test system 308 can perform tasks one by one via a human-machine interface 306 according to instructions shown in a proposed instruction feedback system 304. Some of the instructions can generate low-level libraries 410 within the system, such as a computer vision library 410A, a motion planning library 410B, and a motion execution library 410C (but not limited to these), which can provide instructions in a single suggestion or in multiple choices.

[0047] For example, if the task is to pick up a cup from a table and there are multiple cups, the command could indicate multiple cups, and the human data collector 412 could pick up the most appropriate one depending on the task context (the information needed for situational action) and objective. In another example, the motion planning library 410B could suggest multiple poses for grasping an object, and the human data collector 412 could select one and use the command feedback system 304 to score using this suggested pose. The human data collector 412 could also override an inappropriate suggestion if they felt it was necessary to use the command feedback system 304.

[0048] A specific example 416 of using the command feedback system 304 is described. Suppose a task presented on a task that a human can perform is to pick up a cup 418 from a table. However, as shown in 416, the computer vision library 410A in the system incorrectly displays a virtual marker display 322 on a kettle 420 in the MR device 340, thereby indicating the kettle 420. In such a case, the human data collector 412 knows that the kettle is not a cup and can override the task, leaving feedback 414 such as "This is a kettle, not a cup." All the data of the human performance and feedback can be recorded and processed to generate a new dataset, which can be used to preserve future AI models and / or to improve the task-prompt template library and / or to improve low-level libraries such as the computer vision library, motion planning library, motion execution library, etc. Thus, with the collected dataset, the system can be retrained to correctly recognize the cup and display a visual marker on the cup 418 in the MR device 340.

[0049] The human-in-the-loop method can be implemented in two modes. The first mode can be a test mode. In test mode, a human data collector 412 can be provided with a complete list of human-operated robot tasks 408 that they can perform. For each task, the command feedback system 304 can provide detailed commands via text, voice, or virtual markers. For example, if the task is to walk to a cup, a text or voice command can be displayed in the MR device to indicate that the human data collector needs to walk to a table. Virtual markers of the corresponding path and destination markers can be shown to the human data collector in the MR device. In another example, the task may be to pick up a cup, and markers for multiple possible grasping poses can be shown in the MR device.

[0050] The second mode may include a training mode. In the second mode, the instruction feedback system 304 may not provide instructions for some tasks, or may provide completely incorrect instructions, and a human data collector 412 can switch to a teaching mode to determine how to perform these tasks. All data in teaching mode is recorded by the teaching-test system 302. This data can be used to train the system on these tasks based on human data collector feedback, such as the human data collector's actions when performing the tasks, and / or to improve the task-prompt template library, and / or to improve low-level libraries such as the computer vision library, motion planning library, motion execution library, etc. In this way, real-time training can be achieved.

[0051] Although the above example was described using the human-machine interface 306, the use of the human-machine interface 306 is not necessarily required for all embodiments. For example, in one embodiment, data can be collected directly from the hand of a human data collector 412. In some embodiments, a depth camera or other camera of a sensing system, for example equipped with AI, can track the pose (i.e., the 3D orientation of the human data collector's hand) and / or the movement of the human data collector's hand as the human data collector 412 performs a human-operated robot task. The collected data or feedback can then be provided to a wearable computer subsystem for recording. In various embodiments, an additional step can be added to the workflow 400 in which the teaching-test system 302 maps the pose of the human hand to a pose corresponding to the pose of a robot hand or robot gripper.

[0052] In some embodiments, the sensing system 312 may include motion detection sensors. For example, a human data collector 412 may place markers on their arm, hand, fingers, or finger or hand joints. The motion detection sensors can then use these markers to track the poses and / or movements of the human data collector's hand as they perform a human-operated robot task. The collected data or feedback can then be provided to and recorded in a wearable computer subsystem. In some embodiments, an additional step can be added to the workflow 400 in which the teaching-test system 302 maps the human hand poses to poses corresponding to those of a robot hand or robot gripper.

[0053] In some embodiments, a human data collector 412 may wear an intelligent glove, which incorporates various sensors, such as those used in gloves for some game systems. The incorporated sensors can track the poses and / or movements of the human data collector's hand as the human data collector 412 performs a human-operated robot task. The collected data can then be provided to a wearable computer subsystem for recording. In some embodiments, an additional step can be added to the workflow 400 in the teaching-test system 302, which maps the human hand poses to poses corresponding to those of a robotic hand or robotic gripper.

[0054] In the example of workflow 400, the high-level instruction 400 is not directly deployed on any robot. Therefore, the high-level instruction 406 can be a program in any programming language or any natural language description. Furthermore, some novel human-operated robot tasks may not be adequately described by existing functions in a programming language, so data collection may rely solely on human demonstration. In such embodiments, a natural language description can be generated instead of code in a programming language.

[0055] Therefore, in some embodiments, the code-task interpreter 322 does not only need to translate high-level instructions into a list of human-operated robot tasks 408. Rather, in some embodiments, the code-task interpreter 322 can interpret (sequentially translate) the output of the generating AI model 318 and then generate a list of human-operated robot tasks 408 in natural language format that can be directly executed by a human data collector 412, and / or generate visual / speech instructions that the human data collector 412 can understand. Thus, the code-task interpreter 322 is a high-level interpreter that takes any input related to robot control and then generates a list of human-operated robot tasks 408 in any format that the human data collector 412 can understand. In each application, a prompt 404 can be supplied to the generating AI model 318 to generate a high-level instruction or control code 406, which is then translated by the code-task interpreter 322 into a human-operated robot task. In application 212, a human data collector 412 can perform all three steps revealed by the check mark 213.

[0056] However, in application 214, the reason why the human data collector 412 is unable to perform step 2, as revealed by "X" 215, is because there is a flaw in this step. As shown in 216, the human data collector 412 can determine the flaw in step 2. For example, if this step was to pick up a cup, but a kettle was marked for pickup, as shown in 217, the human data collector 412 can correct the flaw in step 2 and perform the entire task. Any additional steps taken by the human data collector 412 to correct the flaw in step 2 are collected and used for future training as described.

[0057] In application 218, the reason why the human data collector 412 is unable to perform step 2 or 3, as revealed by "X" 219, is that there are defects in both of these steps. As shown in 220, the human data collector 412 can perform the entire task by correcting the defects in step 2 and step 3. The additional steps taken by the human data collector 412 to correct the defects in steps 2 and 3 are collected and used for future training as described. If the human data collector 412 can correct the defects in steps 2 and 3 in a single run round, this is different from the user-on-the-loop method, which can only correct one defect at a time or per given test run.

[0058] Figure 5 shows an example of a generated prompt 404. The generated prompt includes the name, interface, and description of an available high-level function, a task description, an object / location name, and a prompt for generating code. For example, Figure 5 shows an example of a user prompt that a human user can generate. Examples of task-prompt templates 502, 504, and 506 (shown in boxes in the figure) describe available functions that can be used to generate high-level instructions or control codes.

[0059] The generative AI model takes the generated prompts and creates corresponding high-level instructions. Figure 6 shows an example of high-level instructions 406 generated from the LLM for a given task. In some embodiments, the generative AI model can be hosted locally within the system's computer equipment, while in other embodiments, the generative AI model can be hosted within a cloud computing facility and can invoke a corresponding interface or application program interface (API) into the teaching-test system to initiate the execution of the model in response to the generated prompts. Other AI models refer to AI models other than the AI ​​model used to generate high-level instructions or control codes. Other generative AI models may include image-text generation models, video-text generation models, motion planning generation models, grasp pose generation models, and reinforcement learning models for motion execution.

[0060] The code-task interpreter 322 is a component of the teaching-test system 308, which translates high-level instructions 406 into a list of human-operated robot tasks 408 that a human data collector can perform. Each human-operated robot task has a list of inputs, a list of outputs, a task description, and a criterion for achievement, such as checking completion or scoring the task's completion. The inputs can be indicators of objects, such as "bottle", "cup", or "desk".

[0061] Figure 7 shows an example of a human-operated robot task for a given application. Each task includes a list of inputs, a list of outputs, and a task description. While the criteria for success may not be easily described in text, they can be displayed using virtual markers and graphics within the MR (Mixed Reality) device. Virtual markers and voice commands can be generated for each task and sent to the command feedback system.

[0062] The task completion inspection software 324 is a set of algorithms that checks whether a task has been completed, or how well the task was completed. The task completion inspection software 324 can connect to a sensing system to acquire position / pose information of an object of interest, or it can connect to an MR device 340 to acquire the position / pose of a human data collector.

[0063] The memory system 326 can store all sensing data from the sensing system, human control data from the human-machine operation interface, and feedback from the command feedback system.

[0064] The instruction feedback system 304 can be configured to display virtual markers and text instructions for tasks, collect demonstrations from human data collectors, and collect audio feedback from human data collectors. The hardware subsystem may comprise a mixed reality device, and other display and audio devices such as monitors, speakers, and microphones, a computer device for running the MR program, and optical markers such as QR markers for localization.

[0065] Figure 9 shows an embodiment in which a human data collector 412 performs the human-in-the-loop method. Figure 9 shows the human data collector 412 wearing a human-machine operation interface 306 and an MR device 340, where a list of prompts 804 and human-operated robot tasks 808 can be displayed in the MR device 840, as shown in the illustration in Figure 8. In the illustrated embodiment, the human data collector 412 is using the human-machine operation interface 806 to pick up a cup 818 and move it to another location.

[0066] The following software components are present in the command feedback system: (1) Virtual marker display; displays position indicator (index) 1010, object indicators 1018, 1020, and the path to follow. Figures 10 and 11 show examples of object indicators and position and path indicators 1030, 1040. (2) Command display / reading; can be configured to display text information and play voice commands. Figure 8 shows an example of a human-operated robot task shown on MR device 840. (3) Feedback collection; collects human verbal feedback and notes, as well as feedback from mixed reality devices such as gesture, gaze, and / or pointer feedback.

[0067] Figure 12 illustrates a human-in-the-loop method for applying generated AI to robotic applications, such as deploying a robot to perform one or more tasks using system 300 as shown in Figure 3. The human-in-the-loop method according to the embodiments disclosed herein translates high-level instructions into human-operated robotic tasks and displays them on a teaching-test system. A human data collector wears a robotic device and completes the task according to the instructions provided by the teaching-test system. In case of failure, the human data collector knows in real time which step and which instruction caused the failure. This allows the human data collector to overwrite the incorrect instruction, leave feedback or notes, and move forward without having to restart the entire test or training from the beginning.

[0068] The embodiments described herein offer novel and non-obvious advantages over existing systems. For example, human-operated robot tasks are generated from high-level instructions of an AI model, allowing human data collectors to test the human-operated robot tasks and provide real-time feedback on the AI ​​model. Thus, instead of having to use an actual robot for multiple rounds of testing (which can be expensive and time-consuming), human data collectors can test the human-operated robot tasks and, in doing so, identify flaws in the high-level instructions provided by the AI ​​model and improve the future generation of these instructions. Furthermore, when testing is not performed virtually, but rather when human data collectors directly test the human-operated robot tasks, the data collected is based on actual data rather than merely virtual data, which can lead to better results.

[0069] The principles described herein can be implemented in relation to computer systems; therefore, some descriptions of computer system implementations are included to facilitate understanding. Computer systems are now taking on an increasingly diverse range of forms. Computer systems can include, for example, handheld devices, consumer electronics, laptop computers, desktop computers, mainframes, distributed computer systems, data centers, and even devices not traditionally considered computer systems, such as wearable devices (e.g., glasses). In this description and in the claims, “computer system” is broadly defined as any device or system (or combination thereof) that includes at least one physical, tangible processor and physical, tangible memory that can have computer-executable instructions that can be executed by the processor. Memory can take any form and depends on the nature and form of the computer system. Computer systems can be distributed across a network environment and may include computer systems of multiple components.

[0070] In most basic configurations, a computer system generally includes at least one hardware processing unit and memory. The processing unit may include a general-purpose processor, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or any other specialized circuit. The memory may be physical system memory and may be volatile, non-volatile, or any combination of the two. "Memory" is used herein to also refer to non-volatile mass storage (high-capacity storage devices) such as physical storage media. If the computer system is distributed, the processing, memory, and storage capacity may also be distributed.

[0071] A computer system also contains several structures, often referred to as "executable components." For example, the memory of a computer system may contain executable components. An "executable component" is a structure that is well understood by a person skilled in the art to be software, hardware, or a combination thereof. For example, a person skilled in the art will understand that when implemented in software, the structure of an executable component may include software objects, routines, methods, etc., that can be executed on a computer system, regardless of whether such executable components exist in a large number of computer systems or whether they reside on a computer-readable storage medium.

[0072] In such cases, a person of the art will recognize that the structure of the executable component exists on a computer-readable medium, and that when interpreted by one or more processors of a computer system (e.g., by a processor thread), it will cause the computer system to perform a certain function. Such a structure can be made directly computer-readable by the processor (as if the executable component were binary). Alternatively, the structure can be structured to be interpreted and / or compiled (either in a single or multiple stage) to produce a binary that can be directly interpreted by the processor. This understanding of examples of executable component structures is well within the understanding of a person of the art in the ordinary way of computer computing when using the term "executable component".

[0073] An “executable component” is a structure that is well understood by a person of the ordinary art, including structures such as hardcoded or hardwired logic gates, and is implemented exclusively or almost exclusively in hardware, such as within a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or any other specialized circuit. Thus, “executable component” is a term that describes a structure that is well understood by a person of the ordinary art, whether implemented in software, hardware, or a combination thereof. In this specification, terms such as “component,” “agent,” “manager,” “service,” “engine,” “module,” “virtual machine,” etc., may also be used. When used in this specification, and in such cases, these terms (whether expressed with or without a modifying clause) are intended to be synonymous with “executable component,” and therefore these terms also have structures that are well understood by a person of the ordinary art.

[0074] The above description refers to embodiments with reference to operations performed by one or more computer systems. When such operations are implemented in software, one or more processors (of the relevant computer systems performing the operations) direct the operation of the computer system in response to the execution of computer-executable instructions that constitute the executable components. For example, such computer-executable instructions can be embedded in one or more computer-readable media that form a computer program product. An example of such operations is the manipulation of data. When such operations are implemented in hardware, such as in an FPGA or ASIC, the computer-executable instructions can be hardcoded or hardwired logic gates. Computer-executable instructions (and the data being manipulated) can be stored in the computer system's memory. The computer system may also include communication channels that enable the computer system to communicate with other computer systems, for example, over a network.

[0075] Not all computer systems necessarily require a user interface, but in some embodiments, a computer system may include a user interface system used for interaction with a user. A user interface system may include output and input mechanisms. The principles described herein are not limited to specific output or input mechanisms and therefore depend on the nature of the device. However, output mechanisms may include, for example, speakers, displays, haptic outputs, holograms, etc. Examples of input mechanisms may include, for example, microphones, touchscreens, holograms, cameras, keyboards, mice or other pointer input devices, sensors of any kind, etc.

[0076] The embodiments described herein may comprise or utilize dedicated or general-purpose computer systems, which include computer hardware such as one or more processors or system memory, as will be described in more detail below. The embodiments described herein may also include physical or other computer-readable media for transporting or storing computer-executable instructions and / or data structures. Such computer-readable media may be any available media accessible by a general-purpose or dedicated computer system. A computer-readable medium that stores computer-executable instructions is a physical storage medium. A computer-readable medium that transports computer-executable instructions is a transmission medium. Thus, as an example, and not limited to, embodiments of the present invention may comprise at least two distinctly different types of computer-readable media: storage media and transmission media.

[0077] Computer-readable storage media include RAM (random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable ROM), CD-ROM (compact disc-ROM), other optical disk storage media, magnetic disk storage media, other magnetic storage devices, or any other physical tangible storage media, which can be used to store desired program code means in the form of computer-executable instructions or data structures, and which can be accessed by general-purpose or dedicated computer systems.

[0078] A “network” is defined as one or more data links that enable the transfer of electronic data between computer systems and / or modules and / or other electronic devices. When information is transferred to or provided to a computer system over a network or other communication connection (whether hardwired, wireless, or a combination of hardwired and wireless), the computer system appropriately considers this connection as a transmission medium. The transmission medium may include networks or data links, and these networks or data links may be used to carry desired program code means in the form of computer-executable instructions or data structures, and these networks or data links may be accessible by general-purpose or dedicated computer systems. The above combinations should also be included within the scope of computer-readable media.

[0079] Furthermore, upon reaching various components of a computer system, program code in the form of computer-executable instructions or data structures can be automatically transferred from a transmission medium to a storage medium (or vice versa). For example, computer-executable instructions or data structures received over a network or data link can be buffered in a network interface module (e.g., a "NIC (network interface card)") and then finally transferred to the computer system's RAM and / or a less volatile storage medium within the computer system. Thus, it should be understood that storage media can be included as components of a computer system, and these components also utilize transmission media (or rather, primarily transmission media).

[0080] Computer-executable instructions include, for example, instructions and data, which, when executed by a processor, cause a general-purpose computer system, a dedicated computer system, or a dedicated processing unit to perform a specific function or group of functions. Alternatively, or in addition to, computer-executable instructions can configure a computer system to perform a specific function or group of functions. Computer-executable instructions can be instructions that undergo some kind of translation (such as compilation) before being executed directly by the processor, such as instructions in binary, assembly language, or even intermediate formats like source code.

[0081] While the subject matter has been described using terminology specific to structural features and / or methodological behavior, it should be understood that the subject matter defined in the attached claims is not necessarily limited to the described features or behaviors. Rather, the described features and behaviors are disclosed as exemplary forms that realize the claims.

[0082] Those skilled in the art will understand that the present invention can be implemented in a number of computer system configurations within a network computing environment, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs (personal computers), minicomputers, mainframe computers, mobile phones, PDAs (personal digital assistants), pagers, routers, switches, data centers, wearable devices (such as glasses), etc. The present invention can also be implemented in a distributed system environment, in which local and remote computer systems linked through a network (either by hardwired data links, wireless data links, or a combination of hardwired and wireless data links) work together to perform tasks. In a distributed system environment, program modules can be located in both local and remote memory storage devices.

[0083] Those skilled in the art will also understand that the present invention can be implemented within a cloud computing environment. A cloud computing environment can be distributed, but this is not a requirement. When distributed, a cloud computing environment can be distributed internationally within an organization and / or have components across multiple organizations. In this specification and the subsequent claims, “cloud computing” is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage devices, applications, and services). The definition of “cloud computing” is not limited to any of the many other benefits that can be obtained when such models are properly deployed.

[0084] The drawings can illustrate various computer systems, which may include various components or functional blocks capable of implementing various embodiments disclosed herein. These components or functional blocks may be implemented on a local computer system, or on a distributed computer system including elements residing in a cloud, or on a distributed computer system implementing a form of cloud computing. The computer systems in the drawings may include more or fewer components than those shown in the rest of the drawings, and some of these components may be combined as circumstances permit. Although not necessarily shown, various components of a computer system may access and / or utilize processors and memory as necessary to perform their various functions.

[0085] In the processes and methods described herein, the actions performed during the processes and methods can be carried out in different orders. Furthermore, the actions outlined are provided only as examples, and some of the actions can be made optional, combined into fewer steps and actions, supplemented by additional actions, or extended to additional actions, without departing from the essence of the disclosed embodiments.

[0086] The present invention can be embodied in other specific forms without departing from its spirit and characteristics. The embodiments described should be considered in all respects to be illustrative and not limiting. Accordingly, the scope of the present invention is indicated not by the foregoing description but by the appended claims. All modifications that fall within the meaning and scope of the equivalents of the claims are encompassed within the claims.

Claims

1. A method for testing and / or improving a robot control system, The steps include: preparing a library of prompt templates on a computer device that defines one or more steps of one or more robot control tasks; A step of providing a prompt to one or more generative AI models, wherein the one or more generative AI models generate a high-level instruction, which, when executed, triggers a robot to perform one or more robot control tasks, which include one or more steps defined by the prompt template library. The steps include converting the aforementioned high-level instructions into a human-operated robot task, A step of providing the human-operated robot task to a mixed reality device worn by a human data collector, wherein the mixed reality device renders the human-operated robot task in a manner that shows the human data collector how to perform the human-operated robot task. The steps include receiving feedback data in response to the human data collector attempting to perform the human-operated robot task, The steps include updating the robot control system using the aforementioned feedback data. A method that includes this.

2. In the aforementioned mixed reality device, the step of rendering the human-operated robot task is: (1) Input / output variables such as the position of an object, (2) Visual / audio commands displayed on the mixed reality device, (3) Checkable events or measurements as a result of the task, which can be inspected / scored by the mixed reality device or by an external sensor / system. The method according to claim 1, comprising one or more of the following.

3. The method according to claim 1, further comprising the step of generating a prompt from a combination of a prompt template and a user prompt from a library of multiple prompt templates.

4. The method according to claim 1, wherein the high-level instruction calls a predetermined low-level library.

5. The method according to claim 4, wherein the predetermined low-level library includes one or more of a computer vision library, a motion planning library, and a motion execution library.

6. The method according to claim 1, wherein the feedback data includes overwriting one or more of the human-operated robot tasks by the human data collector.

7. The method according to claim 1, wherein the feedback data includes scores for one or more of the human-operated robot tasks provided by the human data collector.

8. The method according to claim 1, wherein the human data collector performs the human-operated robot task using a human-machine interface.

9. The method according to claim 8, wherein the human-machine interface comprises at least one input interface, the input interface receiving user input and providing a microcontroller unit with corresponding control commands for controlling the operation of one or more robot components.

10. The method according to claim 9, wherein the one or more robot components include one or more of a robot gripper, a robot rim, and a robot wheel.

11. The method according to claim 9, wherein the human-machine interface is a human-wearable human-machine interface.

12. The method according to claim 1, wherein the step of updating the robot control system using the feedback data includes updating one or more generated AI models using the feedback data.

13. A system for training and / or testing a robot control system, A computer device configured to train and / or test the robot control system, A data acquisition device configured to collect feedback data related to one or more human-operated robot tasks performed by a human data collector, One or more sensing devices positioned at the location of the human data collector, In a system comprising a mixed reality device worn by the aforementioned human data collector, the system, A library of prompt templates defining one or more steps of one or more robot control tasks is provided on the computer device. The system provides prompts to one or more generative AI models, which generate high-level instructions, which, when executed, trigger a robot to perform one or more robot control tasks, each including one or more steps defined by the prompt template library. The aforementioned high-level instructions are converted into human-operated robot tasks, The aforementioned human-operated robot task is provided to a mixed reality device worn by a human data collector, and the mixed reality device renders the human-operated robot task in a manner that shows the human data collector how to perform the human-operated robot task. In response to the human data collector attempting to perform the human-operated robot task, the robot receives feedback data. The robot control system is updated using the aforementioned feedback data. A system configured in such a way.

14. Rendering the human-operated robot task in the aforementioned mixed reality device is (1) Input / output variables such as the position of an object, (2) Visual / audio commands displayed on the mixed reality device, (3) Checkable events or measurements as a result of the task, which can be inspected / scored by the mixed reality device or by an external sensor / system. The system according to claim 13, comprising one or more of the following.

15. The system according to claim 13, further configured to generate the prompt from a combination of a prompt template from a library of multiple prompt templates and a user prompt.

16. The system according to claim 13, wherein the high-level instruction calls a predetermined low-level library, and the predetermined low-level library includes one or more of a computer vision library, a motion planning library, and a motion execution library.

17. The feedback data includes overwriting one or more of the human-operated robot tasks by the human data collector, The feedback data includes scores for one or more of the human-operated robot tasks provided by the human data collector. The system according to claim 13.

18. The system according to claim 13, wherein updating the robot control system using the feedback data includes updating one or more generated AI models using the feedback data.

19. The robot further comprises a human-machine interface configured to be operated by a human data collector when performing the aforementioned human-operated robot task, The human-machine interface comprises at least one input interface, which receives user input and provides a microcontroller unit with corresponding control commands for controlling the operation of one or more robotic components. The system according to claim 13.

20. The system according to claim 19, wherein the one or more robot components include one or more of a robot gripper, a robot rim, and a robot wheel.

Images