Alleviate the reality gap by training simulation-to-real models using vision-based robotics task models

Abstract

Implementations disclosed herein are directed to mitigating the reality gap by training a simulation-to-real machine learning model ("Sim2Real" model) using a vision-based robotics task machine learning model. The vision-based robotics task machine learning model can be, for example, a reinforcement learning ("RL") neural network model (RL network), such as an RL network that represents a Q function.

Description

Background Art

[0001] Various machine learning-based robot control methods have been proposed. Some of these methods train machine learning models (e.g., deep neural network models) that can be used to generate one or more predictions for controlling a robot, and use training data based only on data from real-world physical robots to train the machine learning models. However, these and / or other methods may have one or more disadvantages. For example, generating training data based on data from real-world physical robots requires a large amount of data using one or more physical robots to generate the training data. This may be time-consuming (e.g., actually operating a real-world physical robot requires a large amount of time), may consume a large amount of resources (e.g., the power required to operate the robot), may cause wear and tear on the robot being used, and / or may require a large amount of human intervention.

[0002] In view of these and / or other considerations, it has been proposed to use a robot simulator to generate simulated data that can be used in generating simulated training data that can be used in training a machine learning model. However, there is often a meaningful "reality gap" between a real robot and a simulated robot (e.g., a physical reality gap) and / or between a real environment and a simulated environment simulated by a robot simulator (e.g., a visual reality gap). This may result in the generation of simulated training data that does not accurately reflect what will occur in a real environment. This may affect the performance of a machine learning model trained on such simulated training data, and / or may require that a large amount of real-world training data also be used in training to help mitigate the reality gap.

[0003] More specifically, various techniques have been proposed to address the visual reality gap. Some of these techniques randomize the parameters of the simulated environment (e.g., texture, lighting, cropping, and camera position) and generate simulated images based on these randomized parameters. Such techniques infer that a model trained based on training instances including such simulated images will better adapt to the real-world environment (e.g., because the real-world environment may be within the range of these randomized parameters). However, this randomization of parameters requires the user to manually define which parameters of the simulated environment will be randomized. In addition, the strategies generated using machine learning models trained based on simulated training data from randomized simulated environments are task-agnostic (i.e., general strategies, script strategies, etc.), rather than task-aware (i.e., specific to a given robot task), which may result in semantics (e.g., objects, (one or more) robot postures and / or other semantics) and styles (e.g., lighting, textures, and / or other styles) being removed during adaptation to the real-world environment.

[0004] Some other techniques include training a Generative Adversarial Network (“GAN”) model or a Cycle Generative Adversarial Network (“CycleGAN”) model to adapt image-to-image translation between a simulated environment and a real-world environment. Although both the GAN model and the CycleGAN model produce more realistic adaptations for real-world environments, the policies generated using the GAN model and the CycleGAN model are still task-agnostic rather than task-aware, which may result in semantics (e.g., objects, (one or more) robot poses, and / or other semantics) and styles (e.g., lighting, textures, and / or other styles) being removed during adaptation to the real-world environment. In addition, due to the task-agnostic nature of these various techniques, the results of robotic tasks (e.g., grasping tasks, navigation tasks, path planning tasks, etc.) may be negatively affected. Summary of the invention

[0005] Implementations disclosed herein relate to mitigating the reality gap by training a simulation-to-real machine learning model ("Sim2Real" model) using (one or more) vision-based robotic task machine learning models. (One or more) vision-based robotic task machine learning models can be, for example, reinforcement learning ("RL") neural network models (RL networks), such as RL networks that represent Q functions or other value functions. For example, the RL network that represents a value function can be used to process a given image and a given action, and generate a predicted value (e.g., a Q value) representing the value of achieving a given action given the given image. As described herein, (one or more) vision-based robotic task machine learning models can be trained based on the output generated by processing predicted real images using (one or more) vision-based robotic task machine learning models. Predicted real images are generated by processing simulated images using the Sim2Real model. In addition, the Sim2Real model can also be trained in part based on the output generated using (one or more) vision-based robotic task machine learning models as well as training (one or more) vision-based robotic task machine learning models. For example, a Sim2Real model can be trained based on the variance of the output generated by processing simulated images using a vision-based robotic task machine learning model, corresponding predicted real images (generated by processing the simulated images using a Sim2Real model), and corresponding predicted simulated images (generated by processing the predicted real images using a real-to-simulation (“Real2Sim”) machine learning model). In these and other ways, the Sim2Real model can be trained to generate predicted real images that are customized for the robotic task for which the (one or more) vision-based robotic task machine learning models are being trained. This can help mitigate the reality gap when the Sim2Real model is trained to generate predicted real images that include attributes relevant to the robotic task. Therefore, when the (one or more) vision-based robotic task machine learning models (trained based on predicted real images and / or predicted simulated images generated using the Sim2Real model) are subsequently used on a real robot and process real images from the (one or more) visual components of the real robot, the performance of the (one or more) vision-based robotic task machine learning models is improved.

[0006] In various implementations, the Sim2Real model can be further trained based on additional and / or alternative losses. The additional and / or alternative losses may include, for example, one or more adversarial losses, one or more cycle consistency losses, one or more temporal difference losses, and / or other losses. For example, assume that the Sim2Real model is a Sim2Real generator model of a generative adversarial network model ("GAN" model), and the generative adversarial network model includes a Sim2Real generator model and a Sim2Real discriminator model. The Sim2Real generator model of the Sim2Real GAN ​​model can generate (one or more) predicted real images by processing (one or more) simulated images, and the discriminator component of the Sim2Real model can determine whether the (one or more) predicted real images are actual real images or (one or more) predicted real images by processing (one or more) predicted real images. Based on this determination, a Sim2Real adversarial loss for the Sim2Real GAN ​​model can be generated and used to further train the generator Sim2Real model of the Sim2Real GAN ​​model. The adversarial loss of the Real2Sim GAN model can also be generated in a similar manner and can also be used to further train the Sim2Real model. As another example, assume that the Sim2Real model is part of a cyclic generative adversarial network model ("CycleGAN" model) that also includes a Real2Sim model ("Sim2Real2Sim CycleGAN" model). The Sim2Real GAN ​​model of the Sim2Real2Sim CycleGAN can generate (one or more) predicted real images by processing (one or more) simulated images, and the Real2Sim GAN model of the Sim2Real2SimCycleGAN model can generate (one or more) predicted simulated images by processing (one or more) predicted real images. The (one or more) simulated images and the (one or more) predicted simulated images can be compared, and the Sim2Real2Sim cycle consistency loss for the Sim2Real2Sim model can be determined based on the comparison. The cycle consistency loss can be used to further train the Sim2Real generator model of the Sim2RealGAN model of the Sim2Real2Sim CycleGAN model. The Real2Sim2Real cycle consistency loss for the Real2Sim2Real CycleGAN model can also be generated in a similar way and can also be used to further train the Sim2Real generator model of the Sim2Real GAN ​​model of the Real2Sim2Real CycleGAN model.As yet another example, a standard temporal difference loss (“TD-loss”) can be generated based on processing images of Sim2Real2Sim and Real2Sim2Real image triplets and the corresponding simulated or real robot actions using a vision-based robotics task machine learning model, and the TD-loss can also be used to further train the Sim2Real model.

[0007] In some implementations, a method implemented by one or more processors is provided, the method comprising processing a simulated image using a simulation-to-real generator model to generate a simulated-episode predicted real image. The simulated image is generated by a robot simulator during a simulated-episode in which a simulated robot attempts to perform a robot task. The method further comprises processing the simulated-episode predicted real image using a real-to-simulation generator model to generate a simulated-episode predicted simulated image. The method further comprises: processing the simulated image and simulated robot action using a task machine learning model used in the robot task to generate a first prediction value; processing the simulated-episode predicted real image and the simulated robot action using the task machine learning model to generate a second prediction value; and processing the simulated-episode predicted simulated image and the simulated robot action using the task machine learning model to generate a third prediction value. The method further comprises generating a loss based on a comparison of the first prediction value, the second prediction value, and the third prediction value; and updating the simulation-to-real generator model based on the generated loss.

[0008] Other implementations of the methods and techniques disclosed herein may each optionally include one or more of the following features.

[0009] In some implementations, the method may further include processing the real image using the real-to-simulation generator model to generate a real-turn predicted simulated image. The real image is captured by a real camera associated with the real robot during a real turn in which the real robot attempts to perform the robot task. The method may further include processing the real-turn predicted simulated image using the simulation-to-real generator model to generate a real-turn predicted real image. The method may further include: processing the real image and the real robot action using the task machine learning model or the additional task machine learning model used in the robot task to generate a fourth prediction value; processing the real-turn predicted simulated image and the real robot action using the task machine learning model or the additional task machine learning model to generate a fifth prediction value; and processing the real-turn predicted real image and the real robot action using the task machine learning model or the additional task machine learning model to generate a sixth prediction value. In some versions of these implementations, generating the loss may be further based on additional comparisons of the fourth prediction value, the fifth prediction value, and the sixth prediction value.

[0010] In some implementations, the comparison of the first predicted value, the second predicted value, and the third predicted value includes three comparisons. Each of the three comparisons can be between a unique pair of the first predicted value, the second predicted value, and the third predicted value (i.e., the first predicted value is compared to the second predicted value, the second predicted value is compared to the third predicted value, and the first predicted value is compared to the third predicted value). In some versions of those implementations, the additional comparison of the fourth predicted value, the fifth predicted value, and the sixth predicted value includes three additional comparisons. Each of the three additional comparisons can be between a unique pair of the fourth predicted value, the fifth predicted value, and the sixth predicted value.

[0011] In some implementations, generating the loss may be further based on an adversarial loss and / or a cycle consistency loss, wherein both the adversarial loss and the cycle consistency loss are generated independently of any output generated using the task machine learning model or the additional task machine learning model. In some versions of those implementations, the adversarial loss may be generated based on a simulated-to-real discriminator model predicting whether the predicted real image is an actual real image or a predicted real image generated by the simulated-to-real generator. In some versions of those implementations, the cycle consistency loss may be generated based on a comparison of the simulated image and the simulated round predicted simulated image. In some other versions of those implementations, generating the loss may be further based on both the adversarial loss and the cycle consistency loss.

[0012] In some implementations, the task machine learning model may represent a Q-function. The task machine learning model may be trained during reinforcement learning based on the simulation episodes and the additional simulation episodes, and the first predicted value may be a first Q-value, the second predicted value may be a second Q-value, and the third predicted value may be a Q-value.

[0013] In some implementations, the method may further include generating a task machine learning model loss based on the second prediction value; and updating the task machine learning model based on the task machine learning model loss. In some versions of those implementations, generating the task machine learning model loss is independent of the first prediction value, the third prediction value, the fifth prediction value, and the sixth prediction value. In some other versions of those implementations, generating the task machine learning model loss is further based on the fourth prediction value.

[0014] In some implementations, the simulated round can be an offline round. In some alternative implementations, the simulated round can be an online round.

[0015] In some implementations, the robotic task may be an object manipulation task or a navigation task. In some versions of those implementations, the robotic task may be an object manipulation task, and the object manipulation task may be a grasping task.

[0016] In some implementations, a method implemented by one or more processors is provided, including training a simulation-to-reality generator model, wherein the training of the simulation-to-reality generator model includes iteratively updating the simulation-to-reality generator model based on a reinforcement learning scenario consistency loss, wherein the reinforcement learning scenario consistency loss is generated based on multiple prediction values ​​generated using a task machine learning model or an additional task machine learning model. The method further includes, after training the simulation-to-reality generator model, generating a robot control policy for a real robot based on simulated training data generated using the simulation-to-reality generator model; and causing the real robot to be controlled based on the generated robot control policy.

[0017] Other implementations of the methods and techniques disclosed herein may each optionally include one or more of the following features.

[0018] In some implementations, the plurality of prediction values ​​generated using the task machine learning model are generated based on processing simulated-to-real-to-simulated image triplets using the task machine learning or the additional task machine learning model. In addition, the reinforcement learning scene consistency loss generated based on the plurality of prediction values ​​is generated based on a comparison of the plurality of prediction values.

[0019] Other implementations may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., central processing unit(s) (CPUs), graphics processing unit(s) (GPUs), and / or tensor processing units (TPUs)) to perform methods such as one or more of the methods described above and / or elsewhere herein. Other implementations may include a system of one or more computers and / or one or more robots including one or more processors operable to execute stored instructions to implement methods such as one or more of the methods described above and / or elsewhere herein.

[0020] It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein are considered part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered part of the subject matter disclosed herein.

[0021] The above description is provided as an overview of some implementations of the present disclosure. Further descriptions of those implementations and other implementations are described in more detail below. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 An example robot and an example training system according to various implementations disclosed herein are illustrated.

[0023] Figure 2 According to the various implementations disclosed in this article, the use of Figure 1 Example process flow for training a simulation-to-real model of an example training system and (one or more) vision-based RL task models.

[0024] Figure 3A is an example process flow for using a simulated-to-real-to-simulated cycle generative adversarial network model to generate an adversarial loss and a simulated-to-real-to-simulated cycle consistency loss for updating a simulated-to-real model according to various implementations.

[0025] Figure 3B is an example process flow for using a simulated-to-real-to-simulated loop generative adversarial network model to generate a reinforcement learning scenario consistency loss for updating a simulated-to-real model according to various implementations.

[0026] Figure 4A is an example process flow for using a real-to-simulated-to-real cycle generative adversarial network model to generate an adversarial loss and a real-to-simulated-to-real cycle consistency loss to update a simulated-to-real model according to various implementations.

[0027] Figure 4BAn example process flow for using a real-to-simulated-to-real recurrent generative adversarial network model to generate a reinforcement learning scenario consistency loss for updating a simulated-to-real model according to various implementations.

[0028] Figure 5 is a flow chart illustrating an example method for updating a simulation-to-real model using a reinforcement learning scene consistency loss in accordance with various implementations disclosed herein.

[0029] Figure 6 An example architecture of a robot according to various implementations disclosed herein is schematically depicted.

[0030] Figure 7 An example architecture of a computer system according to various implementations disclosed herein is schematically depicted. DETAILED DESCRIPTION

[0031] Now turning to the attached figure, Figure 1 An example robot 130 and an example training system 150 are illustrated. The robot 130 includes a base 133 having wheels disposed on opposite sides thereof for movement of the robot 130. The base 133 may include, for example, one or more motors for driving the wheels of the robot 130 to achieve a desired movement direction, speed, and / or acceleration of the robot 130. Although the robot 130 is Figure 1 130 is depicted as a mobile robot having a base 133 with wheels disposed on opposite sides thereof for movement of the robot 130, but it should be understood that this is for purposes of example and is not meant to be limiting, and that additional and / or alternative robots, mobile or stationary, may be provided. For example, in some implementations, a mobile telepresence robot, a humanoid robot, a robotic arm, and / or other robots may be provided as a distinct robot and / or in conjunction with other robots.

[0032] The robot 130 also includes (one or more) visual components 131. In some implementations, (one or more) visual components 131 may include a monocular camera that includes a single sensor (e.g., a charge coupled device (CCD)) and generates images based on physical properties sensed by the sensor, each image including multiple data points defining color values ​​and / or grayscale values. For example, the monocular camera can generate images including red, blue, and / or green channels. Each channel can define a value for each of a plurality of pixels of the image, such as defining a value from 0 to 255 for each pixel of the image. In some implementations, (one or more) visual components 131 may include a stereo camera that includes two or more sensors, each sensor being located at a different vantage point. In some of these implementations, the stereo camera generates images based on characteristics sensed by the two sensors, each image including multiple data points defining depth values ​​and color values ​​and / or grayscale values. For example, the stereo camera can generate images including a depth channel and red, blue, and / or green channels. In some implementations, the vision component(s) 131 may include a stereo camera, but only one of the sensors may be used to generate an image that does not include a depth channel. Additional vision components(s) separate from the robot 130 may also be included to capture real image(s).

[0033] The robot 130 also includes one or more processors that, for example, provide control commands to actuators and / or other operating components thereof, generate control commands for various tasks, etc. For example, the one or more processors of the robot 130 can provide control commands to servo motors that drive the wheels of the robot 130 to autonomously and / or semi-autonomously navigate the robot 130 to various locations in the environment. In addition, for example, the one or more processors of the robot 130 can implement all or various aspects of the training system 150 and / or perform Figure 5 All or aspects of method 500. (For example, with respect to Figure 6 Robot 620) Additional description of some examples of the structure and functionality of various robots is provided herein.

[0034] The robot 130 also includes robotic arms 134A and 134B having respective end effectors 135A and 135B, each of which takes the form of a gripper having two opposing "fingers" or "toes". The robotic arms 134A, 134B and the end effectors 135A, 135B may be controlled autonomously, semi-autonomously, and / or based on control commands provided by a user via the user's computing device. For example, the robotic arm 134B may be autonomously controlled to position the end effector 135B near an object, and the end effector 135B may be autonomously controlled to grasp an object. In some of these implementations, the processor(s) of the robot 130 may generate control commands to control the robotic arm 134B and / or the end effector 135B, and the generation of these control commands may be at least partially influenced by the current pose of the robot 130 determined according to the techniques described herein. Furthermore, the robot 130 may perform various tasks using the robotic arms 134A, 134B and the end effectors 135A, 135B, such as object manipulation tasks, navigation tasks, path planning tasks, and / or other robotic tasks.

[0035] In some implementations, all or aspects of the training system 150 may be implemented by the robot 130. In some implementations, all or aspects of the training system 150 may be implemented by one or more remote computing systems and / or devices (e.g., computing device 910 of FIG. 9 ) that communicate with the robot 130 via (one or more) networks (e.g., local area network (LAN), wide area network (WAN), Bluetooth, and / or other networks). Various modules or engines may be implemented as part of the training system 150 as software, hardware, or any combination of both. For example, Figure 1 As shown, the training system 150 may include a robot simulator 152, (one or more) simulation-to-real (“Sim2Real”) engines 154, (one or more) real-to-simulation (“Real2Sim”) engines 156, (one or more) vision-based robot task engines 158, and (one or more) loss engines 160.

[0036] In some implementations, the vision-based robotic task engine(s) 158 may apply a simulated-to-real-to-simulated (“Sim2Real2Sim”) image triple of [simulated image, predicted real image, looped simulated image] as an input across the vision-based robotic task model(s) 190 (e.g., RL network) to generate a predicted value for each image in the Sim2Real2Sim image triple. The simulated images of the image triples may be generated by the robotic simulator 152 during a simulation episode of executing the robotic task for which the vision-based robotic task model(s) 190 is being trained. The predicted real images of the Sim2Real2Sim image triples are generated by (one or more) Sim2Real engines 154 processing the simulated images using the Sim2Real models stored in (one or more) Sim2Real and Real2Sim model databases 180, and the predicted simulated images of the image triples are generated by (one or more) Real2Sim engines 156 processing the predicted real images using the Real2Sim models stored in (one or more) Sim2Real and Real2Sim model databases 180.

[0037] Since each image in a Sim2Real2Sim image triplet represents the same scene, there should be little variance in the predicted values ​​for the Sim2Real image triplet. The loss engine(s) 160 may generate a Sim2Real2Sim RL scene consistency loss based on the variance of the three predicted values ​​using the loss function(s) 195, each predicted value being generated based on processing a corresponding image of the Sim2Real2Sim image triplet using the vision-based robotic task model(s) 190. The loss engine(s) 160 may back-propagate the Sim2Real2Sim RL scene consistency loss on the Sim2Real model to update one or more weights of the Sim2Real model, thereby training the Sim2Real model in view of the specific robotic task for which the vision-based robotic task model(s) 190 is being trained. This training of the Sim2Real model using (one or more) vision-based robotic task models enables the generation of trained RL policies (e.g., reflected by (one or more) vision-based robotic task models stored in (one or more) vision-based robotic task model databases 190) based on simulated rounds from the simulated environment and more accurately reflects the real-world environment. In addition, by training the Sim2Real model and (one or more) vision-based robotic task models 190, the Sim2Real models stored in (one or more) Sim2Real and Real2Sim model databases 180 are task-aware (i.e., specific to a given robotic task) rather than task-agnostic (i.e., general policies, scripted policies, etc.). In other words, the training technique can be performed to train various Sim2Real models specific to a given robotic task (e.g., Sim2Real models for grasping tasks, Sim2Real models for navigation tasks, Sim2Real models for path planning tasks, and / or other Sim2Real models for other robotic tasks). Furthermore, in various implementations, such training of the Sim2Real model may be performed in conjunction with training of the vision-based robotic task model(s). Thus, when a policy generated in a simulated environment based on simulated training data is transferred to a real robot (e.g., robot 130) for use in a real-world environment, the robotic performance for a given robotic task may be improved compared to policies trained using various other techniques.

[0038] In some implementations, the Sim2Real model trained according to the techniques described herein is a Sim2Real generator model of a generative adversarial network ("GAN") model. The GAN model includes at least a Sim2Real generator model and a Sim2Real discriminator model stored in (one or more) Sim2Real and Real2Sim model databases 180. In some versions of those implementations, the Sim2Real generator model is used to process a simulated image to generate a predicted real image corresponding to the simulated image. In addition, the Sim2Real discriminator model can be used to evaluate the predicted real image to predict whether the predicted real image is a real image or a predicted real image generated using the Sim2Real generator model. In some versions of those implementations, the Real2Sim generator model can be used to process a real image to generate a predicted simulated image corresponding to a real image. In addition, the Real2Sim discriminator model can evaluate the predicted simulated image to predict whether the predicted simulated image is a real simulated image or a predicted simulated image generated by the Real2Sim generator model. The loss engine (s) 160 can generate an adversarial loss based on these predictions generated using the discriminator model (s), and the adversarial loss can be used as part of the loss back-propagated on the GAN model. Thus, the goal of the generator model is to "fool" the discriminator model to maximize the adversarial loss, and the goal of the discriminator model is to minimize the adversarial loss, thereby improving the GAN model.

[0039] In additional and / or alternative implementations, the Sim2Real model trained according to the techniques described herein is a Sim2Real generator model of a cyclic generative adversarial network model ("CycleGAN" model). The CycleGAN model includes at least a Sim2Real GAN ​​model and a Real2Sim GAN model stored in (one or more) Sim2Real and Real2Sim model databases 180, wherein both the Sim2Real GAN ​​model and the Real2Sim GAN model each include a corresponding generator model and a corresponding discriminator model. In some versions of those implementations, the Sim2Real generator model is a Sim2Real generator model of a Sim2Real2Sim CycleGAN model that can generate simulated images based on a distribution of real images, and can process the simulated images to generate predicted real images corresponding to the simulated images (or process simulated images generated by the robot simulator 152). In addition, the Sim2Real discriminator model of the Sim2Real2Sim CycleGAN model can evaluate the predicted real image to predict whether the predicted real image is a real image or a predicted real image generated by the Sim2Real generator model. In addition, the Real2Sim generator model of the Sim2Real2Sim CycleGAN model can process the predicted real image to generate a predicted simulated image corresponding to the predicted real image. Therefore, the predicted simulated image should also correspond to the original simulated image, and the (one or more) loss engines 160 can generate a Sim2Real2Sim cycle consistency loss for the Sim2Real2Sim CycleGAN model based on comparing the simulated image with the predicted simulated image. In addition, the Real2Sim discriminator model of the Sim2Real2Sim CycleGAN model can evaluate the predicted simulated image to predict whether the predicted simulated image is a real real image or a predicted simulated image generated by the Real2Sim generator model. The Sim2Real2Sim CycleGAN technique is used to generate Sim2Real2Sim image triplets of [simulated image, predicted real image, predicted simulated image]. In addition, the (one or more) loss engines 160 can use Sim2Real and / or Real2Sim discriminator models to generate adversarial losses based on the above determinations, and the adversarial losses can be used as part of the loss back-propagated on the Sim2Real2Sim CycleGAN model to update the Sim2Real generator model.

[0040] In some other versions of those implementations, the Real2Sim generator model of the Real2Sim2Real CycleGAN model stored in the (one or more) Sim2Real and Real2Sim model databases 180 can process the real image to generate a predicted simulated image corresponding to the real image. In addition, the Real2Sim discriminator model of the Real2Sim2Real CycleGAN model can evaluate the predicted simulated image to determine whether the predicted simulated image is a real simulated image or a predicted simulated image generated by the Real2Sim generator model. In addition, the Sim2Real generator model of the Real2Sim2Real CycleGAN model can process the simulated image to generate a predicted real image corresponding to the predicted simulated image. Therefore, the predicted real image should also correspond to the original real image, and the (one or more) loss engines 160 can generate the Real2Sim2Real cycle consistency loss of the Real2Sim2Real CycleGAN model based on comparing the real image with the predicted real image. In addition, the Sim2Real discriminator model of the Real2Sim2Real CycleGAN model can evaluate the predicted real image to predict whether the predicted real image is a real image or a predicted real image generated by the Sim2Real generator model. The Real2Sim2Real image triple of [real image, simulated image, predicted real image] is generated using the Sim2Real2Sim CycleGAN technology. (One or more) loss engines 160 can generate adversarial losses based on the above determination of the Real2Sim and / or Sim2Real discriminator models, and the adversarial losses can be used as part of the loss back-propagated on the Real2Sim2Real CycleGAN model. Real2Sim2Real adaptation can be performed outside the policy and can be used to quickly improve the performance of these models in generating policies using (one or more) vision-based robot task models stored in (one or more) vision-based robot task model databases 190 to perform robot tasks.

[0041] In some implementations, the vision-based robotic task model database(s) 190 include Q-functions that can be used for RL training for vision-based RL, and can be based on continuous action generalization of Q-learning, which is sometimes referred to herein as "QT-opt". Unlike other continuous action Q-learning methods that are generally unstable, QT-opt does not require training explicit actors, but instead uses stochastic optimization to select actions (during inference) and target Q-values ​​(during training). In addition, QT-opt can be used to train Q-functions to select actions to be evaluated in view of the current state, and a strategy that generates Q-values ​​is used to select a given action (from the evaluated actions) to be implemented in view of the current state. The strategy for generating these Q-values ​​can be generated using a Sim2Real model trained with simulated data and can be transmitted to a real robot (e.g., robot 130). In some implementations, the Sim2Real model used to generate the strategy for generating Q-values ​​is a Sim2Real generator model of a GAN model stored in the Sim2Real and Real2Sim model database(s) 180. In some other implementations, the Sim2Real model used to generate the strategy for generating Q-values ​​is a Sim2Real generator model of a CycleGAN model stored in (one or more) Sim2Real and Real2Sim model databases 180. It is worth noting that the Sim2Real generator model of the GAN model and the CycleGAN model can be the same Sim2Real generator model.

[0042] In some implementations, a Sim2Real2Sim image triple of [simulated image, predicted real image, predicted simulated image] may be applied as an input to the Q network to generate a Q value for each image. The simulated images of the Sim2Real2Sim image triples may be generated by the robot simulator 152 during a simulation round of performing the robot task for which the Q network is being trained. As described above, the predicted real images of the Sim2Real2Sim image triples may be generated by processing the simulated images using the Sim2Real generator model of the GAN model and / or the CycleGAN model, and the predicted simulated images of the Sim2Real2Sim image triples may be generated by processing the predicted real images using the Real2Sim generator model of the GAN model and / or the CycleGAN model.

[0043] Since each image in the Sim2Real2Sim image triplet represents the same scene, the Q-values ​​of the Sim2Real2Sim image triplet should have almost no variance. The Sim2Real2Sim RL scene consistency loss can be determined based on the variance of the Q-values ​​in the Sim2Real2Sim image triplet. In some versions of those implementations, the Real2Sim2Real image triplet of [real image, simulated image, predicted real image] can be applied as input to a Q network or an additional Q network to generate a Q-value for each of the images. Since each of the images in the Real2Sim2Real image triplet also represents the same scene, the Q-values ​​of the Real2Sim2Real image triplet should have almost no variance. The Real2Sim2Real RL scene consistency loss can be determined based on the variance of the Q-values ​​in the Real2Sim2Real image triplet. In addition, when using the Q-function trained by RL, a standard time difference loss ("TD loss") can also be generated.

[0044] The RL scene consistency loss, Sim2Real2Sim RL scene consistency loss, and / or Real2Sim2Real RL scene consistency loss may be back-propagated to the Sim2Real model via the loss engine(s) 160, resulting in training of the Sim2Real model, and optionally resulting in joint training of the vision-based robotic task model(s) stored in the vision-based robotic task model(s) database(s) 190. In some implementations, one or more additional or alternative losses may be back-propagated on the GAN model and / or the CycleGAN model to train the Sim2Real model. The one or more losses may include: for each of Sim2Real2Sim and Real2Sim2Real, a first adversarial loss of a first GAN model of the CycleGAN model, a second adversarial loss of a second GAN model of the CycleGAN model, a cycle consistency loss of the CycleGAN, and / or a TD loss of the RL network. Notably, by back-propagating the RL scene consistency loss on the CycleGAN model, the (one or more) vision-based robotic task models ensure that the semantics (e.g., objects, (one or more) robot poses and / or other semantics) and / or styles (e.g., lighting, textures and / or other styles) of a given robotic task for the (one or more) vision-based robotic task models are preserved in the scene. Therefore, the policies generated in the simulated environment using the (one or more) vision-based robotic task models trained according to the techniques described herein are more suitable for use in the real-world environment. Therefore, such training of the Sim2Real model (and optionally joint training with (one or more) vision-based robotic task models) results in policies generated in the simulated environment that more accurately reflect the real-world environment and are customized for specific tasks. In addition, the (one or more) Sim2Real models can be trained to preserve different semantics (e.g., objects, (one or more) robot poses and / or other semantics) and / or styles (e.g., lighting, textures and / or other styles) based on the actions evaluated when training the (one or more) vision-based robotic task models stored in the (one or more) vision-based robotic task model database 190.

[0045] For example, based on the distribution of real images, assume that the robot simulator 152 generates a simulated environment for a simulated round of a robot task, wherein the distribution of real images can be based on the real round of the robot task and the data log associated with the real round of the robot task. In addition, (one or more) Sim2Real engines 154 can use the Sim2Real generator model stored in (one or more) Sim2Real and Real2Sim model databases 180 to generate predicted real images based on simulated images, and (one or more) Real2Sim engines 156 can use the Real2Sim generator model stored in (one or more) Sim2Real and Real2Sim model databases 180 to generate predicted simulated images based on predicted real images. This results in Sim2Real2Sim image triplets of [simulated image, predicted real image, predicted simulated image] that each correspond to the same scene.

[0046] In addition, the (one or more) vision-based robot task engines 158 can use the Sim2Real2Sim image triples of [simulated image, predicted real image, predicted simulated image] to train the vision-based robot task model stored in the (one or more) vision-based robot task model 190. In some implementations, the vision-based robot task model is a value network that uses each image of the Sim2Real2Sim image triple and the candidate action to be performed for the robot task as input on the network. The value network can be used to determine the predicted value of each image. Each predicted value can be determined based on evaluating the candidate action to be performed for the robot task in view of the corresponding image in the image of the Sim2Real2Sim image triple (e.g., indicating the probability of successful execution of the robot task if one or more candidate actions are applied). In some other implementations, the vision-based robot task model is a Q-function that uses each image of the Sim2Real2Sim image triple and the one or more candidate actions to be performed for the robot task as input to the Q-function. The Q function can determine a Q value for each image based on evaluating one or more candidate actions to be performed for the robotic task, given an image in the Sim2Real2Sim image triplet, to measure the successful performance of the robotic task. In these implementations, there should be little or no variance between the predicted values ​​and / or Q values ​​because each image in the Sim2Real2Sim image triplet corresponds to the same scene. Based on these variances, the loss engine(s) 160 can generate an RL scene consistency loss for the vision-based robotic task model, and can back-propagate the RL scene consistency loss on the Sim2Real model.

[0047] For example, if a given robotics task is an object manipulation task, such as a grasping task, but the predicted simulated images of the Sim2Real2Sim image triples have removed the object that is the focus of the grasping task, there will be large variance in the predicted values ​​and / or Q-values ​​in the Sim2Real2Sim image triples. Therefore, by back-propagating the RL scene consistency loss of the vision-based robotics task model on the Sim2Real model, the Sim2Real model can learn to preserve the object. In this way, the Sim2Real model can learn to preserve not only the object, but also other semantics and / or styles that are important for a given task.

[0048] In some additional and / or alternative implementations, the techniques disclosed herein can be used in conjunction with other non-reinforcement learning training. For example, rather than using RL techniques such as Q-learning, a Sim2Real model can be trained using a value network that predicts whether a given action will result in successful performance of a task (e.g., one or more motion commands result in a successful grasp) without considering a reward function during training. In some versions of those implementations, the value network can be jointly trained with the Sim2Real model. For example, the value network can be trained using supervised or semi-supervised training instances, each of which includes an image and a candidate action as a training instance input, as well as a labeled probability indicating the likelihood that the candidate action will result in success of the robot task given the image.

[0049] Thus, the techniques described herein implement Sim2Real2Sim CycleGAN models and / or Real2Sim2RealCycleGAN models to mitigate the visual-reality gap trained based at least on losses from (one or more) vision-based robotic task models stored in (one or more) vision-based robotic task model databases 190, thereby improving real-world vision-based control policies. In some implementations, by combining the RL scene consistency loss of the current Q-function and the losses of the Sim2Real2Sim CycleGAN model and / or Real2Sim2Real CycleGAN model, the style and / or semantics of the Sim2Real images can be improved for a given robotic task. For example, these losses can adapt the simulated images to look realistic, while the RL scene consistency loss ensures that the relevant semantics of the Sim2Real images are preserved. Furthermore, by training the Sim2Real model for a given task using (one or more) vision-based robotic task models stored in the (one or more) vision-based robotic task model database 190 (and optionally jointly training the Sim2Real model and (one or more) vision-based robotic task models), there is no need to generate scene segmentation masks for grasping tasks or to define canonical scenes and simulation randomization for simulated images.

[0050] As used herein, a "reality gap" (e.g., a physical reality gap) exists between a real robot and a simulated robot and / or a "reality gap" (e.g., a visual reality gap) exists between a real environment and a simulated environment simulated by a robot simulator. The technology disclosed herein relates to mitigating the visual reality gap between a real environment and a simulated environment simulated by a robot simulator.

[0051] The training of machine learning models that are robust and accurate and can be used to control a real-world physical robot (e.g., robot 130) is often limited by the scalability of using a real-world physical robot to generate a sufficient amount of training data and / or to generate sufficiently diverse training data. For example, for many challenging robotic tasks (e.g., object manipulation tasks, navigation tasks, path planning tasks, and / or other robotic tasks), it is not feasible to repeatedly run real robot experiments to collect real training data in the real world. Therefore, training a machine learning model using simulated training data from a robot simulator 152 is a faster and cheaper alternative than collecting and using real training data. However, simulated training data is not perfect, and machine learning models trained using simulated training data do not always transfer to the real world due to the reality gap.

[0052] The implementations described herein propose techniques for mitigating the reality gap between the robot simulator 152 and (one or more) real-world physical robots (e.g., robot 130) and / or the real-world environment by training a Sim2Real model using (one or more) vision-based robot task models (e.g., RL) for the robot (e.g., robot 130). By training the Sim2Real model in this manner, and optionally jointly training the Sim2Real model and (one or more) vision-based robot task models (e.g., RL) for the robot (e.g., robot 130), the reality gap in the image-to-image conversion of Sim2Real when generating policies can be mitigated. The Sim2Real model can be a Sim2Real generator model of a GAN model, a CycleGAN model, and / or any other suitable Sim2Real model stored in (one or more) Sim2Real and Real2Sim model databases 180. For example, a GAN model can be used to adapt the pixels of a simulated image to be more realistic before being used by a deep RL model stored in (one or more) vision-based robot task model databases 190. As another example, the CycleGAN model can be used to adapt the pixels of simulated images to be more realistic before being used by the deep RL model stored in the (one or more) vision-based robotic task model databases 190, and can also be used to implement cycle consistency between Sim2Real2Sim and Real2Sim2Real adaptations to produce images that are better for the deep RL model than the GAN or CycleGAN model alone. In this way, the Sim2Real model of the CycleGAN model can be trained using the deep RL model (referred to as "RL-CycleGAN" training herein) to ensure that the vision-based robotic task model (e.g., RL model) is consistent in all adaptations. In addition, by training the Sim2Real model of the CycleGAN model using the deep RL model (and optionally jointly training the RL model and the Sim2Real model), the real-world performance of various robotic tasks (e.g., grasping, path planning, navigation, etc.) may be greatly improved and only a portion of the real training data is required compared to known techniques. This RL-CycleGAN training technique can be used to train various CycleGAN models to generate task-aware (i.e., specific to a given robotic task) policies, rather than task-agnostic (i.e., general policies, scripted policies, etc.) policies as in general GAN ​​models or CycleGAN models.

[0053] RL provides a powerful tool to automatically acquire visual representations for end-to-end training directly on task-specific goals (e.g., grasping, path planning, navigation, etc.). For example, RL can be used to train a deep neural network model (e.g., stored in (one or more) vision-based task model databases 190) of a robot (e.g., robot 130) to grasp objects directly using image observations, or to perform navigation using a mobile robot (e.g., robot 130) directly from the mobile robot's onboard sensor readings. However, the ability to learn visual representations end-to-end with a task controller is expensive in terms of collecting training data, and since the data required for RL is often task- and policy-specific, collecting this training data can be cumbersome. Therefore, an attractive alternative is to generate simulated training data in a simulated environment using simulated visual representations, and then use the policy generated based on the simulated training data in a real-world system. However, by using this simulated training data, the policy results in an improved representation of the simulated environment, which may not accurately reflect the real-world environment visually or physically (e.g., reality gap).

[0054] To mitigate this reality gap, the implementations described herein enable the transfer of task-aware vision-based policies from a simulated environment to a real-world environment. In some implementations, simulated images can be automatically converted to realistic images via a CycleGAN model (e.g., using (one or more) Sim2Real engines 154 and (one or more) Real2Sim engines 156 as described above), which has access to a real-observed "offline policy" dataset, but no simulated-realistic image pairs. For example, assume that the robot task is a closed-loop vision-based grasping task, and further assume that a policy for the vision-based grasping task is learned via a deep Q-learning technique (such as QT-opt) based on simulated grasping rounds in a simulated environment or real grasping rounds in a real environment. If these real rounds are collected with a scripted policy or a previously trained machine learning model, these real rounds are considered "offline policies", and if these real rounds are collected using a Q-learning technique on a real robot, these real rounds are considered "online policies". It is more cost-effective to train a machine learning model based on offline policy data because the same offline policy data can be reused as training data and no new real rounds are required.

[0055] In some versions of these implementations, since there are no simulated realistic image pairs, the CycleGAN model can adopt a cycle consistency method when training the Sim2Real model of the CycleGAN model. This method provides pixel-level domain adaptation for simulated images and allows the CycleGAN model to be trained in simulation on images that reflect those images that the policy will encounter in the real world. In addition, by implementing cycle consistency during training the Sim2Real model of the CycleGAN model, geometric information is retained in the simulated images, which may be critical for a given task (e.g., a grasping task of a specific object) when encountered in real images. However, and as described above, the Real2Sim model of the CycleGAN model is a task-agnostic model, and task-related details can change between simulated images and real images encountered in the real world. These changes may have a negative impact on the performance of the RL policy generated using (one or more) vision-based robotic task models that are trained using these simulated images. To reduce the effects caused by the task-agnostic model, the Sim2Real model of the CycleGAN model can be trained using an RL scene consistency loss (hence the term "RL-CycleGAN training"). In some implementations, the vision-based robotic task engine(s) 158 can use an RL-trained Q-function, and due to the RL scene consistency loss, the Q-values ​​predicted by the RL-trained Q-function are invariant under the CycleGAN transformation. The RL scene consistency loss is general in that it can be used for any RL problem.

[0056] Thus, the implementation described in this paper provides a Sim2Real transfer of a vision-based reinforcement learning policy, a Sim2Real transfer based on this RL scene consistency loss, and a Sim2Real model of a CycleGAN model with an RL scene consistency loss, resulting in an RL-CycleGAN model. The CycleGAN cycle consistency loss ensures that high-level scene geometry and content are preserved, while the CycleGAN RL scene consistency loss focuses on features that are critical to the current RL Q-function that the vision-based robotic task model is being trained on. The combined CycleGAN cycle consistency loss and CycleGAN RL scene consistency loss of the current Q-function can be used to produce a Sim2Real image translation model that can be used to generate policies that effectively generalize to the real world.

[0057] As described above, a task-aware vision-based policy can be generated using the RL-CycleGAN model and can be transferred from a simulated environment to a real-world environment. For example, given a current RL Q-function for a grasping task as a vision-based robotic task model, when the robot attempts to grasp an object based on a vision-based robotic control policy generated using the RL-CycleGAN model, the real RGB images captured by the robot's vision sensor and the robot's candidate motion parameters can be used to continuously make grasp predictions. These techniques can be a supplement or alternative to manually marking grasp positions for objects and / or predicting overall grasp results based on a given set of candidate motion parameters for the robot.

[0058] As also described above, using simulated training data in a simulated environment to train a machine learning model (e.g., stored in (one or more) vision-based robot task model databases 190) and then using the policy generated based on the simulated training data in a real-world system is an attractive approach. However, by using the simulated training data, the policy may not accurately reflect the real-world environment visually or physically (e.g., reality gap). More specifically, the simulated images may have unrealistic parameters such as texture, lighting, color, and / or objects, and the policy generated based on the simulated images may not accurately reflect the real-world environment visually.

[0059] In addition, the technology described herein can be considered as visual domain-level adaptation technology.Domain-level adaptation technology is intended to use training data from a source domain (e.g., a simulated environment) in combination with training data from a target domain (e.g., a real-world environment) to train a machine learning model. In some implementations, domain-level adaptation technology includes feature-level adaptation technology. Feature-level adaptation technology can be used to learn the invariant features of an environment that is well defined in both simulation and real-world environments. In some other implementations, domain-level adaptation technology includes pixel-level adaptation technology. Pixel-level adaptation technology can be used to re-stylize pixels from a source image (e.g., a simulated image) to look like an image from a target domain (e.g., a real-world environment) without any simulated realistic image pairs.

[0060] In addition, the pixel-level adaptation technique uses the Sim2Real generator model of the GAN model to re-stylize pixels from the source image (e.g., simulated image) to look like an image from the target domain (e.g., real-world environment). However, the challenge of using the GAN model for Sim2Real transfer is that the GAN model generates images based on the distribution of real-world images rather than simulated images. In order to generate a robust GAN model for Sim2Real transfer, it is desirable to have a realistic version of the simulated image, not just any realistic image. For example, for a robotic grasping task, an additional loss term can be applied to the GAN model to ensure that the semantics of the grasped image is preserved in the Sim2Real adaptation. In addition, a content-based loss can also be applied to the GAN model to ensure that the generated adapted image does not deviate too much from the input image. Furthermore, a special mask can also be created for the simulated image of the object, and the GAN model can reproduce the special mask in addition to the generated adapted image. Therefore, when training an RL task-based model for a robotic grasping task (e.g., stored in (one or more) vision-based robotic task model databases 190), real images can be mixed with adapted images, and domain adversarial losses can be applied across intermediate layers of the RL task-based model to learn domain-invariant features.

[0061] In addition, and as described above, the CycleGAN model employs two GAN models, each of which includes a simulator model and a discriminator model for Sim2Real and Real2Sim (e.g., stored in the (one or more) Sim2Real and Real2Sim model databases 180). In some implementations, the first GAN generator model of a given CycleGAN model can be the (one or more) Sim2Real generator model that the (one or more) Sim2Real engine 154 can use to adapt images from a source domain (e.g., a simulated environment) to a target domain (e.g., a real-world environment). In some other versions of these implementations, the second GAN generator model of a given CycleGAN model can be the (one or more) Real2Sim generator model that the (one or more) Real2Sim engine 156 can use to adapt images from a target domain (e.g., a real-world environment) to a source domain (e.g., a simulated environment). In addition, the loss engine (s) 160 may generate a cycle consistency loss that ensures that when a given image is applied to the first GAN model and the second GAN model in succession, the adapted image is the given image. This enables the given CycleGAN model to retain as much of the scene as possible during adaptation as the image is reproduced from the adapted image by the second GAN model. Thus, visual differences caused by the reality gap may be mitigated because the semantics of the scene (e.g., objects, robot poses (s) and / or other semantics) are preserved.

[0062] In some implementations, for Sim2Real image-to-image translation, a CycleGAN model can be trained to learn a mapping between two image domains X and Y, where X can represent a source domain of images (e.g., simulated images) and where Y can represent a target domain of images (e.g., real-world images). Furthermore, in some versions of these implementations, the mapping can be based on unpaired images from each domain, such as and Furthermore, as described above, the CycleGAN model employs two GAN models (e.g., stored in (one or more) Sim2Real and Real2Sim model databases 180), and training the CycleGAN model includes learning a mapping of two generators: a Sim2Real generator G: X→Y and a Real2Sim generator F: Y→X. Furthermore, each of the generators includes a corresponding adversarial discriminator D X and D Y , and train the CycleGAN model so that the corresponding adversarial discriminator D X and D YEach of is capable of distinguishing the simulated image {x} from the adapted simulated image {F(y)}, and distinguishing the real image {y} from the adapted real image {G(x)}. Thus, in some versions of these implementations, when the CycleGAN model is trained to learn these mappings of the two generators, an adversarial loss can be applied to the two mappings. For example, for the Sim2Real2Sim CycleGAN model, the (one or more) loss engine 160 can generate an adversarial loss as shown in Equation 1 using the (one or more) loss functions stored in the (one or more) loss function database 195:

[0063]

[0064] The Sim2Real generator G can be obtained by comparing it to the adversarial generator D Y The loss of D minimizes the adversarial loss to produce realistic images, while the adversarial generator D Y Trying to maximize this adversarial loss of the Sim2Real generator G, resulting in an updated adversarial loss of the Sim2Real generator G In contrast, the Real2Sim generator F can be trained against the adversarial generator D by minimizing X The loss is used to generate simulated images, while the adversarial generator D X Trying to maximize this adversarial loss of the Real2Sim generator F, resulting in an updated adversarial loss of the Real2Sim generator F

[0065] In some implementations, the loss engine(s) 160 may generate a cycle consistency loss for the CycleGAN model using the loss function(s) stored in the loss function database(s) 195, as shown in Equation 2:

[0066]

[0067] This cycle consistency loss ensures that x→G(x)→F(G(x))≈x and y→F(y)→G(F(y))≈y, where d is some distance metric. In other words, the cycle consistency loss uses mean squared error to prevent drastic deviations when generating Sim2Real and Real2Sim images on the CycleGAN model, because the original images (whether simulated or real) are recoverable. However, the CycleGAN model trained in this way is still task-agnostic (i.e., general policy, scripted policy, etc.) rather than task-aware (i.e., specific to a given robotic task), which may be problematic when simulated policies generated using the CycleGAN model are transferred to real-world robots to perform challenging robotic tasks (e.g., grasping tasks, navigation tasks, path planning tasks, etc.).

[0068] Thus, the training of the CycleGAN model can be combined with RL techniques to generate a task-aware CycleGAN model that is task-agnostic as opposed to task-agnostic. In some implementations, Q-learning can be used to learn a Q function (e.g., stored in a (one or more) vision-based robotic task model database 190) for a vision-based task representing a total expected future reward, where s is an input image and where α is a candidate action executable based on the input image. In addition, the (one or more) loss engines 160 can utilize a given state transition (s, a, r, s′) and can update the Q function Q(s, a) using the (one or more) loss functions stored in the (one or more) loss function database 195 to minimize the temporal difference loss (“TD loss”), as shown in Equation 3:

[0069] l(Q(s,a),r+γV(s′)) (Equation 3)

[0070] Where V(s′) is the target value, and where l is the loss function. The target value V(s′) can be estimated using clipped double Q learning. In addition, the Q function can be jointly trained with the CycleGAN model (e.g., RL-CycleGAN training), and the Q value can be used to add an additional consistency loss to the CycleGAN training.

[0071] One advantage of using a task-aware CycleGAN model is that a Sim2Real generator model can be used to adapt simulated images to real images while retaining semantics for the purpose of performing the task. For example, with respect to a robotic grasping task, a typical GAN ​​model and / or a CycleGAN model can generate realistic images based on simulated images, but when the realistic images are generated by (one or more) Sim2Real engines 154, some semantics (e.g., objects, robot poses, etc.) and styles (e.g., lighting, textures, etc.) present in the simulated images are removed. These removed objects can change the results of the robotic grasping task. In addition, the distinction between semantics and style is not always clear and can depend on the specific robotic task to be performed. However, by using a task-aware Sim2Real generator model (e.g., stored in (one or more) Sim2Real and Real2Sim model databases 180) trained using a vision-based robotic task model (e.g., stored in (one or more) vision-based robotic task model databases 190), the vision-based robotic task model can constrain the Sim2Real generator model to indirectly retain the semantics and / or style of the scene.

[0072] As described above, in some implementations, the vision-based robotic task model used by the vision-based robotic task engine(s) 158 may be a deep Q-learning network (e.g., stored in the vision-based robotic task model database(s) 190) that learns a Q-function Q(s,a) for a vision-based task, where s is an input image, and where α is a candidate action (s,a) executable based on the input image. In some implementations, a single Q-network Q(s,a) may be used to represent the Q-function trained on simulated and real input images and candidate actions (s,a). In some other versions of these implementations, multiple Q-networks Q sim (s,a) and Q real (s, a) can be used to represent the Q function trained on the simulated and real input images and candidate actions (s, a) respectively. The RL-CycleGAN model can jointly train the RL model with the Sim2Real generator model of the CycleGAN model, and each of the six image Sim2Real2Sim image triplets {x, G(x), F(G(x))} and Real2Sim2Real image triplets {y, F(y), G(F(y))} as described above is passed to Q sim (s,a) and Q real (s, a) to generate the Q value of each of the six images. The Q value of each of the six images can be expressed as:

[0073]

[0074]

[0075]

[0076]

[0077]

[0078]

[0079] Each of these Q values ​​corresponds to the Q value of a given image among the six images included in the Sim2Real2Sim image triplet {x, G(x), F(G(x))} and the Real2Sim2Real image triplet {y, F(y), G(F(y))}. In addition, each of these six images should represent the same scene. Therefore, the Q values ​​for each corresponding image triplet should be similar, and in order to minimize the difference in the Q values ​​for each corresponding image triplet, the (one or more) loss engine 160 can use the (one or more) loss functions stored in the (one or more) loss function database 195 to generate the RL scene consistency loss as shown in Equation 4:

[0080]

[0081] where d is some distance metric. In other words, the RL scene consistency loss uses mean squared error to penalize the variation of the Q-value of a given one of the Sim2Real2Sim and Real2Sim2Real image triplets to preserve the RL scene during adaptation.

[0082] In addition, the Q-network (e.g., stored in the (one or more) vision-based robot task model database 190), whether a single Q-network Q(s,a) or multiple Q-networks Q sim (s,a) and Q real (s, a), can be trained using standard TD loss on all images of Sim2Real2Sim and Real2Sim2Real image triplets {x, G(x), F(G(x)), y, F(y), G(F(y))}, and the (one or more) loss engine 160 can use the (one or more) loss functions stored in the (one or more) loss function database 195 to generate the TD loss, as shown in Equation 5:

[0083]

[0084] In some implementations, the Sim2Real model of the CycleGAN model can be trained based only on the RL scene consistency loss. In some versions of those implementations, the Sim2Real model of the CycleGAN model can be trained based on a combination of one or more additional losses. The one or more additional losses can include an adversarial loss for each GAN of the CycleGAN, a CycleGAN consistency loss, and / or a TD loss. Therefore, the loss for training the CycleGAN model can generally be expressed by Equation 6:

[0085]

[0086] Thus, the techniques described herein implement a CycleGAN model to mitigate the visual-reality gap that is trained based on at least a loss from a vision-based robotics task model, thereby improving a real-world vision-based control policy. In some implementations, by combining the RL scene consistency loss of the current Q-function with the CycleGAN loss, the style and / or semantics of the Sim2Real image can be improved for a given robotics task. For example, the CycleGAN loss can adapt the simulated image to look realistic, while the RL scene consistency loss ensures that the relevant semantics of the Sim2Real image is preserved. In addition, by training the CycleGAN model using a vision-based robotics task model for a given task (and optionally jointly training with a vision-based robotics task model), there is no need to generate scene segmentation masks for grasping tasks or define canonical scenes and simulation randomization for simulated images.

[0087] Now go to Figure 2 , depicting the use of Figure 11 and 1. Example processing flow of a Sim2Real model and (one or more) vision-based RL model of an example training system. In some implementations, the robot simulator 152 can generate (one or more) simulated images 210. The (one or more) simulated images 210 can be applied as input to the cross-Sim2Real2Sim CycleGAN model 205 to generate (one or more) simulated round predicted simulated images 212. More specifically, the CycleGAN model 205 may include (one or more) Sim2Real engines 154 for processing (one or more) simulated images 210 using (one or more) Sim2Real models stored in (one or more) Sim2Real and Real2Sim model databases 180 to generate (one or more) simulated round predicted real images, and may include (one or more) Real2Sim engines 156 for processing (one or more) simulated round predicted real images using (one or more) Real2Sim models stored in (one or more) Sim2Real and Real2Sim model databases 180 to generate (one or more) simulated round predicted simulated images 212.

[0088] The simulation-episode predicted real image(s) and / or the simulation-episode predicted simulated image(s) 212 provide a more accurate representation of the real-world environment when compared to the original simulated image(s) 210 generated by the simulator 152. In addition, when the simulation-episode predicted real image(s) and / or the simulation-episode predicted simulated image(s) 212 are generated using the CycleGAN model 205 including the Sim2Real model trained using the techniques described herein, style (e.g., lighting, texture, and / or other styles) and / or semantics (e.g., objects, object(s) poses, and / or other semantics) are preserved in the simulation-episode predicted real image(s) and the simulation-episode predicted simulated image(s) 212. Thus, when the CycleGAN model 205 including the Sim2Real model trained using the techniques described herein is trained jointly with the vision-based robotic task model(s) 190, these styles and / or semantics preserved on the images are relevant to the performance of the given simulated robotic task for which the vision-based robotic task model(s) 190 are being trained. Once the Sim2Real model is trained using the techniques described herein, the vision-based robotic task model(s) 190 may be further trained using the simulated bout predicted real image(s) and / or the simulated bout predicted simulated image(s) 212 generated using the CycleGAN model 205 including the trained Sim2Real model. Furthermore, once the vision-based robotic task model(s) 190 are further trained, the vision-based robotic task model(s) 190 may be tested using the real image(s) 213 captured by the vision component(s) 131 of the robot 130 (or other vision component(s) external to the robot 130). In this manner, the vision-based robotic task model(s) 190 may be effectively trained using simulated training data that retains the style and / or semantics associated with a given task of a real robot, thereby alleviating the vision gap and eliminating the need to collect real training data, which may be an expensive and time-consuming process.

[0089] Now go to Figure 3A, depicts an example processing flow for using a Sim2Real2Sim CycleGAN model to generate an adversarial loss and a Sim2Real2Sim cycle consistency loss to update a Sim2Real model. In some implementations, the Sim2Real model may be a Sim2Real generator model of a Sim2Real GAN ​​model of a Sim2Real2Sim CycleGAN model. The Sim2Real2Sim CycleGAN model may also include a Real2Sim GAN model. In addition, each GAN model of the Sim2Real2SimCycleGAN model may include a corresponding generator model and a discriminator model. In some versions of these implementations, the Sim2Real model may be trained based on the adversarial loss and / or the Sim2Real2Sim cycle consistency loss of the Sim2Real2Sim CycleGAN model. For example, Figure 3A As shown, the robot simulator 390 (e.g., Figure 1 The robot simulator 152 of the training system and / or (one or more) other robot simulators) can generate simulated images 311 during a simulation session of performing a vision-based robotics task (e.g., a simulated object manipulation task, a simulated navigation task, a simulated path planning task, and / or other vision-based robotics task). The Sim2Real generator 321 can use (e.g., stored in Figure 1 The (one or more) Sim2Real and Real2Sim model databases 180 of the (one or more) Sim2Real generator models process the simulated image 311 to generate a simulated round predicted real image 312. The Sim2Real discriminator 322 may use (for example, stored in Figure 1 The Sim2Real and Real2Sim model database 180 (one or more) Sim2Real discriminator models are used to process the simulated round predicted real image 312 to determine whether the simulated round predicted real image 312 is a real image or a simulated round predicted real image generated by the Sim2Real generator 321. Based on this determination, the loss engine 340A can generate a Sim2Real adversarial loss 340A1. (For example, refer to Figure 1 and Equation 1) The paper describes the generation of the Sim2Real adversarial loss in more detail.

[0090] In addition, Real2Sim generator 323 may use (for example, stored in Figure 1The (one or more) Sim2Real and Real2Sim model databases 180 of the (one or more) Real2Sim generator models are used to process the simulated round predicted real image 312 to generate the simulated round predicted simulated image 313. The Real2Sim discriminator 324 may use (for example, stored in Figure 1 (one or more) Sim2Real and Real2Sim model databases 180) (one or more) Real2Sim discriminator models are used to process the simulated episode prediction simulated image 313 to determine whether the simulated episode prediction simulated image 313 is a real simulated image or a simulated episode prediction simulated image generated by the Real2Sim generator 323. Based on this determination, the loss engine 340B can generate a Real2Sim adversarial loss 340B1. (For example, refer to Figure 1 and Equation 1) The paper also describes the generation of the Real2Sim adversarial loss in more detail.

[0091] In addition, the loss engine 340C may compare the simulation image 311 generated by the robot simulator 390 with the simulation round prediction simulation image 313 generated by the Real2Sim generator 323 to generate a Sim2Real2Sim cycle consistency loss 340C1. (For example, refer to Figure 1 and Equation 2) The present invention describes the generation of the Sim2Real2Sim cycle consistency loss in more detail. It is worth noting that the simulated image 311, the simulated round predicted real image 312, and the simulated round predicted simulated image 313 form a Sim2Real2Sim image triplet, and there should be little or no variance between the images in the Sim2Real2Sim image triplet. In other words, the Sim2Real2Sim cycle consistency loss 340C1 can be used to update the Sim2Real generator model 321 to ensure that the features of the original simulated image 311 are preserved when processed by the generators 321 and 323 of the Sim2Real2Sim CycleGAN model. It can be used (for example, referring to Figure 1 , 3B , 4A and 4B) Sim2Real adversarial loss 340A1, Real2Sim adversarial loss 340B1, Sim2Real2Sim cycle consistency loss 340C1 and / or one or more of the additional losses are generated by Figure 3A The updates of the Sim2Real model used by the Sim2Real generator 321 of the Sim2Real2Sim CycleGAN model. The updates can be back-propagated on the Sim2Real2Sim CycleGAN model to train the Sim2Real model during simulation rounds of the robotic task.

[0092] In some additional and / or alternative versions of those implementations, the Sim2Real model can be trained based on additional or alternative losses. For example, Figure 3B , depicting an example process flow for generating RL scene consistency loss using a Sim2Real2SimCycleGAN model for updating a Sim2Real model. In some versions of those implementations, the loss can be generated based on the Figure 3A The Sim2Real2Sim RL scene consistency loss generated during the same simulation round is used to train the Sim2Real model. In addition, the adversarial loss (e.g., Sim2Real adversarial loss 340A1 and Real2Sim adversarial loss 340B1) and the cycle consistency loss (e.g., Sim2Real2Sim cycle consistency loss 340C1) can be used in the same way as in the above reference. Figure 3A The same method as described above can be used based on reference Figure 3A and 3B Any one or any combination of the losses described above can be used to update the Sim2Real model.

[0093] For example, Figure 3B As shown, the vision-based robot task engine 332 can process the simulated image 311 and the simulated robot action 399 to generate a first prediction value 361, process the simulated round predicted real image 312 and the simulated robot action 399 to generate a second prediction value 362, and process the simulated round predicted simulated image 313 and the simulated robot action 399 to generate a third prediction value 363. The simulated robot action 399 can be any vision-based action that can be performed by the simulated robot during a simulated robot task (e.g., a simulated object manipulation task, a simulated navigation task, a simulated path planning task, and / or any other simulated vision-based action). In addition, the vision-based robot task engine 332 can use (e.g., stored in Figure 1 The vision-based robot task model(s) in the vision-based robot task model database 190 is used to process each of the images 311, 312, and 313 of the Sim2Real2Sim image triplet.

[0094] In some implementations, the vision-based robotic task engine 332 uses a single vision-based robotic task model to process each of the images 311, 312, and 313 of the Sim2Real2Sim image triplet, while in other implementations, the vision-based robotic task engine 332 uses multiple vision-based robotic task models to process each of the images 311, 312, and 313. As a non-limiting example, a first vision-based robotic task model trained based on a simulated training round can be used to process the simulated image 311 and the simulated round predicted simulated image 313, and a second vision-based robotic task model trained based on a real training round can be used to process the simulated round predicted real image 312. (For example, refer to Figure 1 ) The vision-based robotics task model(s) used by the vision-based robotics task engine 332 are described in more detail herein. In some implementations, the vision-based robotics task model(s) used by the vision-based robotics task engine 332 is a Q-network, and the prediction values ​​361, 362, and 363 are the corresponding Q-values ​​for each image 311, 312, and 313 of the Sim2Real2Sim image triplet. (For example, see Figure 1 )The Q network used to generate the corresponding Q values ​​is described in more detail above.

[0095] The loss engine 340D may process the first prediction value 361, the second prediction value 362, and the third prediction value 363 to generate a Sim2Real2Sim RL scene consistency loss 340D1. More specifically, the loss engine 340D may determine a mean square error based on the first prediction value 361, the second prediction value 362, and the third prediction value 363, and generate a Sim2Real2Sim RL scene consistency loss 340D1 based on these errors. (For example, refer to Figure 1 and Equation 4) The above describes in more detail the generation of Sim2Real2Sim RL scene consistency loss 340D1. In addition, although not in Figure 3B depicted in (e.g., reference Figure 1 and Equation 5) As described in more detail above, a temporal difference (“TD”) loss may be generated based on processing by the vision-based robotic task engine 332. Sim2Real2Sim RL scene consistency loss 340D1 and / or TD loss may be used to generate Figure 3B The updates are back-propagated on the Sim2Real2Sim CycleGAN model of the proposed framework, thereby training the Sim2Real model during simulated episodes of the robotics task.

[0096] In this way, the Real2Sim2Real cycle consistency loss 440C1 can be used to update the Sim2Real generator model 423 to ensure that the features of the original real image 411 are preserved when processed by the generators 321 and 323 of the Sim2Real2Sim CycleGAN model, and the preserved features are relevant to the simulated task being performed during the simulation round, so that the trained Sim2Real model is task-aware. In addition, the updates used to train the Sim2Real model can be based on the reference 440C1 of this article. Figure 3A , 2 Although each of loss engines 340A, 340B, 340C, and 340D are depicted as a different loss engine, it should be understood that this is for purposes of illustration and loss engines 340A, 340B, 340C, and 340D may be a single loss engine and / or different loss engines.

[0097] Now go to Figure 4A , depicts an example processing flow for using a Real2Sim2Real CycleGAN model to update a Sim2Real model to generate an adversarial loss and a Real2Sim2Real cycle consistency loss. In some implementations, the Sim2Real model can be a Sim2Real generator model of a Sim2Real GAN ​​model of a Real2Sim2Real CycleGAN model. The Real2Sim2Real CycleGAN model can also include a Real2Sim GAN model. In addition, each of the GAN models of the Real2Sim2Real CycleGAN model can include a corresponding generator model and a discriminator model. In some versions of those implementations, the Sim2Real model can be trained based on the adversarial loss and / or the Real2Sim2Real cycle consistency loss of the Real2Sim2Real CycleGAN model. For example, Figure 4A As shown, (one or more) visual components 492 (e.g., Figure 1 The real image 411 may be generated by the (one or more) vision components 131 and / or (one or more) other vision components of the robot 130 during a real round of performing a vision-based robotic task (e.g., a real object manipulation task, a real navigation task, a real path planning task, and / or other vision-based robotic task). The Real2Sim generator 421 may use (e.g., stored in Figure 1The Real2Sim discriminator 422 may use (for example, stored in) the (one or more) Sim2Real and Real2Sim model databases 180 to process the real image 411 to generate the real round prediction simulation image 412. Figure 1 The Real2Sim model database 180 (one or more) of the Sim2Real and Real2Sim discriminator models in the Real2Sim model database 180 is used to process the real turn prediction real image 312 to determine whether the real turn prediction simulated image 312 is a real simulated image or a real turn prediction simulated image generated by the Real2Sim generator 421. Based on this determination, the loss engine 440A can generate the Real2Sim adversarial loss 440A1. (For example, refer to Figure 1 and Equation 1), generating the Real2Sim adversarial loss is described in more detail in this paper.

[0098] Additionally, the Sim2Real generator 423 may use (e.g., stored in Figure 1 The (one or more) Sim2Real and Real2Sim model databases 180 of the (one or more) Sim2Real generator models process the real round prediction simulated image 412 to generate the real round prediction real image 413. The Sim2Real discriminator 424 can use (for example, stored in Figure 1 (one or more) Sim2Real and Real2Sim model databases 180) (one or more) Sim2Real discriminator models are used to process the real round predicted real image 413 to determine whether the real round predicted real image 413 is a real image or a real round predicted real image generated by the Sim2Real generator 423. Based on this determination, the loss engine 440B can generate a Sim2Real adversarial loss 440B1. (For example, refer to Figure 1 and Equation 1) The paper also describes the generation of the Sim2Real adversarial loss in more detail.

[0099] In addition, the loss engine 440C can compare the real image 411 generated by the (one or more) visual components 492 with the real round predicted real image 413 generated by the Sim2Real generator 423 to generate a Real2Sim2Real cycle consistency loss 440C1. (For example, refer to Figure 1and Equation 2) The present invention describes generating the Real2Sim2Real cycle consistency loss in more detail. Notably, the real image 411, the real round predicted simulated image 412, and the real round predicted real image 413 form a Real2Sim2Real image triplet, and there should be little or no variance between the images in the Real2Sim2Real image triplet. In other words, the Real2Sim2Real cycle consistency loss 440C1 can be used to update the Sim2Real generator model 423 to ensure that the features of the original real image 411 are preserved when processed by the generators 421 and 423 of the Real2Sim2RealCycleGAN model. (For example, as shown in reference Figure 1 , 3A , 3B and 4B) Real2Sim adversarial loss 440A1, Sim2Real adversarial loss 440B1, Real2Sim2Real cycle consistency loss 440C1 and / or one or more of the additional losses may be used to generate Figure 4A Updates to the Sim2Real model used by the Sim2Real generator 423 of the Real2Sim2Real CycleGAN model. The updates can be back-propagated on the Real2Sim2RealCycleGAN model to train during real rounds of the robotics task Figure 4A The Sim2Real generator of the Real2Sim2RealCycleGAN model 423.

[0100] In some additional and / or alternative versions of these implementations, the Sim2Real model can be trained based on additional or alternative losses. Figure 4B , depicts an example process flow for generating RL scene consistency loss using a Sim2Real2SimCycleGAN model for updating a Sim2Real model. In some versions of these implementations, the Sim2Real model may be trained based on the Real2Sim2Real RL scene consistency loss and may be used in Figure 4A In addition, you can refer to Figure 4A , as described above, adversarial losses (e.g., Real2Sim adversarial loss 440A1 and Sim2Real adversarial loss 440B1) and cycle consistency losses (e.g., Real2Sim2Real cycle consistency loss 440C1) are generated in the same manner. Figure 3A , 3B , 4A, and 3B described any one or any combination of the losses to update the Sim2Real model.

[0101] For example, Figure 4B As shown, the vision-based robot task engine 432 can process the real image 411 and the real robot action 499 to generate a fourth prediction value 464, process the real turn prediction simulated image 412 and the real robot action 499 to generate a fifth prediction value 465, and process the real turn prediction real image 413 and the simulated robot action 499 to generate a sixth prediction value 466. The real robot action 499 can be any vision-based action that can be performed by a real robot during a real robot task (e.g., a real object manipulation task, a real navigation task, a real path planning task, and / or any other real vision-based action). In addition, the vision-based robot task engine 432 can use (e.g., stored in Figure 1 Each of the images 411, 412 and 413 of the Real2Sim2Real image triplet is processed using (one or more) vision-based robotic task models in the vision-based robotic task model database 190.

[0102] In some implementations, the vision-based robotics task engine 432 uses a single vision-based robotics task model to process each of the images 311, 412, and 413 of the Real2Sim2Sim image triplet, while in other implementations, the vision-based robotics task engine 432 uses multiple vision-based robotics task models to process each of the images 411, 412, and 413. As a non-limiting example, a first vision-based robotics task model trained based on a real training round can be used to process the real image 411 and the real-round predicted real image 413, and a second vision-based robotics task model trained based on a simulated training round can be used to process the real-round predicted simulated image 412. (For example, refer to Figure 1 ) The vision-based robotics task model(s) used by the vision-based robotics task engine 432 are described in more detail herein. In some implementations, the vision-based robotics task model(s) used by the vision-based robotics task engine 432 is a Q-network, and the prediction values ​​464, 465, and 466 are corresponding Q-values ​​for each image 411, 412, and 413 of the Real2Sim2Real image triplet. (For example, see Figure 1 )The Q function used to generate the corresponding Q value is described in more detail above.

[0103] The loss engine 440D may process the fourth prediction value 461, the fifth prediction value 465, and the sixth prediction value 466 to generate a Real2Sim2Real RL scene consistency loss 440D1. More specifically, the loss engine 440D may determine a mean square error based on the fourth prediction value 461, the fifth prediction value 465, and the sixth prediction value 466, and generate the Real2Sim2Real RL scene consistency loss 440D1 based on these errors. (For example, refer to Figure 1 and Equation 4) The above describes in more detail the generation of Real2Sim2Real RL scene consistency loss 440D1. In addition, although not in Figure 4B , but (for example, see Figure 1 and Equation 5) as described in more detail above, a temporal difference (“TD”) loss may be generated by processing the Real2Sim2Sim RL scene consistency loss 440D1 and / or the TD loss may be used to generate Figure 4B The updates are back-propagated onto the Real2Sim2Real CycleGAN model of , thereby training the Sim2Real model during real episodes of the robotics task.

[0104] In this way, the Real2Sim2Real cycle consistency loss 440C1 can be used to update the Sim2Real generator model 423 to ensure that the features of the original real image 411 are preserved when processed by the generators 421 and 423 of the Sim2Real2Sim CycleGAN model, and the preserved features are relevant to the simulated task being performed during the simulation round, so that the trained Sim2Real model is task-aware. In addition, the updates used to train the Sim2Real model can be based on the reference Figure 4A and 4B One, a subset, and / or all of the losses described herein. Although each of loss engines 440A, 440B, 440C, and 440D are depicted as different loss engines, it should be understood that this is for purposes of example and loss engines 440A, 440B, 440C, and 440D may be a single loss engine and / or different loss engines.

[0105] also, Figure 3A , 3B , 4A and 4B can be the same model and can be used to use the Sim2Real and Real2Sim models. Figure 3A , 3B, 4A and 4B to update the Sim2Real model. In some implementations, the simulated rounds and real rounds of the robot task can be performed in parallel. For example, as shown in reference Figure 3B As described, three prediction values ​​can be generated during the simulated robot task and as referenced Figure 4B As described, three additional prediction values ​​can be generated during the real robot task. In this example, it can be (as shown in reference Figure 1 4) generates an RL scene consistency loss based on the six prediction values. The Sim2Real model can be updated based on the RL scene consistency loss and optionally one or more additional losses described herein. In some additional and / or alternative implementations, the Sim2Real model can be trained primarily using simulated rounds or real rounds of the robotic task. In implementations where the Sim2Real model is trained primarily (or even exclusively) on simulated rounds of the robotic task, the RL scene consistency loss can be the Sim2Real2Sim RL consistency loss, as described in reference Figure 3B In implementations where the Sim2Real model is trained primarily (or even exclusively) on real episodes of the robotics task, the RL scenario consistency loss can be the Real2Sim2Real RL consistency loss, which generates updates to the Sim2Real model, as described in reference Figure 4B described.

[0106] Now go to Figure 5 , Figure 5 is a flowchart illustrating an example method 500 for training a Sim2Real model according to various implementations disclosed herein. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include one or more processors, such as Figure 1 Robot 130 and / or Figure 6 the processor(s) of the robot 620, Figure 7 The processor(s) 714 and / or other processor(s) of the computing device of the method 500 may be executed by the method 500. Although the operations of the method 500 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0107] At box 552, the system processes the simulated image using the simulated-to-real generator model to generate a simulated-episode predicted-real image. The simulated image can be generated by a robot simulator during a simulated episode in which a simulated robot attempts to perform a robot task. In addition, the simulated-episode predicted-real image corresponds to the simulated image. For example, the simulated image can include a simulated robot arm attempting to perform a simulated object manipulation task (such as a simulated grasping task). As another example, the simulated image can include a simulated environment navigated by a simulated robot during a simulated navigation task. In addition, in some implementations, the simulated-episode predicted-real image can be evaluated by a simulated-to-real discriminator model that predicts whether the simulated-episode predicted-real image is actually a real image or a simulated-episode predicted-real image generated by the simulated-to-real generator model. A simulated-to-real adversarial loss can be generated based on the prediction by the simulated-to-real discriminator model.

[0108] At box 554, the system processes the simulated-episode-predicted real image using the real-to-simulation generator model to generate a simulated-episode-predicted simulated image. The simulated-episode-predicted simulated image corresponds to the simulated-episode-predicted real image, and therefore also corresponds to the simulated image. In addition, in some implementations, the simulated-episode-predicted simulated image can be evaluated by a real-to-simulator discriminator model, which predicts whether the simulated-episode-predicted simulated image is actually a simulated image or a simulated-episode-predicted simulated image generated by the real-to-simulation generator model. A real-to-simulation adversarial loss can be generated based on the prediction by the real-to-simulation discriminator model. In addition, a cycle loss can be determined based on comparing the simulated image initially processed by the simulation-to-real generator model with the simulated-episode-predicted simulated image. The result of the processing in boxes 552 and 554 is an image triplet of [simulated image, simulated-episode-predicted real image, simulated-episode-predicted simulated image].

[0109] At block 556, the system processes the simulated image and the simulated robot action using the task machine learning model trained for the robot task to generate a first prediction value. In some implementations, the first prediction value is generated by applying the simulated image and the simulated robot action as input across a value network. In some other implementations, the first prediction value is a first Q-value generated by applying the simulated image and the simulated robot action as input across a Q-network.

[0110] At box 558, the system uses the task machine learning model to process the simulated round predicted real image and simulated robot action to generate a second prediction value. In some implementations, the second prediction value is generated by applying the simulated image and simulated robot action as input across the value network. In some other implementations, the second prediction value is a second Q value generated by applying the simulated image and simulated robot action as input across the Q network.

[0111] At block 560, the system processes the simulated round prediction simulated image and simulated robot action using the task machine learning model to generate a third prediction value. In some implementations, the third prediction value is generated by applying the simulated image and simulated robot action as input across the value network. In some other implementations, the third prediction value is a third Q-value generated by applying the simulated image and simulated robot action as input across the Q network.

[0112] At box 562, the system generates a loss based on a comparison of the first prediction value, the second prediction value, and the third prediction value. These comparisons include three comparisons between unique pairs of the first prediction value, the second prediction value, and the third prediction value (e.g., [first prediction value, second prediction value], [first prediction value, third prediction value], and [second prediction value, third prediction value]). By comparing these unique prediction value pairs, the variance across the first prediction value, the second prediction value, and the third prediction value can be determined. The loss generated at box 562 can be generated based on the variance. Again, because each image in the image triplet of [simulated image, simulated round predicted real image, simulated round predicted simulated image] represents the same scene, ideally there is little variance in these prediction values.

[0113] At block 564, the system updates the simulated-to-real model based on the generated losses. The system can update the simulated-to-real model by back-propagating the generated losses on the simulated-to-real model. In some implementations, one or more additional losses can also be back-propagated on the simulated-to-real model. The one or more additional losses can include the simulated-to-real adversarial loss from block 552, the real-to-simulated adversarial loss from block 554, the loop loss from block 554, and / or other losses. The various losses can be back-propagated separately, or combined into a single loss (e.g., a loss function that is itself based on the various losses).

[0114] At block 566, the system determines whether to perform further training on the simulated-to-real model. If at an iteration of block 566, the system determines that further training is to be performed on the simulated-to-real model, the system returns to block 552 to process additional simulated images. If at an iteration of block 566, the system determines that no further training is to be performed on the simulated-to-real model, the system proceeds to block 568, and training of the simulated-to-real model ends.

[0115] By training a simulation-to-real model based on a generative loss from a task machine learning model, the simulation-to-real model can perform more accurately in a real-world environment and can be customized for a given task of the task machine learning model. A given task of the task machine learning model can be based on simulated robot actions used to generate predictions. Thus, a given simulation-to-real generator model can be customized for a given robotic task, and various semantics (e.g., objects, (one or more) robot poses, and / or other semantics) and / or styles (e.g., lighting, textures, and / or other styles) can be preserved when the policy generated in the simulation is transferred to a real robot operating in a real-world environment.

[0116] Figure 6 An example architecture of a robot 620 is schematically depicted. The robot 620 includes a robot control system 660, one or more operating components 604a-n, and one or more sensors 608a-m. The sensors 608a-m may include, for example, visual sensors (e.g., camera(s), 3D scanners), light sensors, pressure sensors, position sensors, pressure wave sensors (e.g., microphones), proximity sensors, accelerometers, gyroscopes, thermometers, barometers, and the like. Although the sensors 608a-m are depicted as being integrated with the robot 620, this is not meant to be limiting. In some implementations, the sensors 608a-m may be located external to the robot 620, such as as a standalone unit.

[0117] The operating components 604a-n may include, for example, one or more end effectors (e.g., gripping end effectors) and / or one or more servo motors or other actuators to achieve movement of one or more components of the robot. For example, the robot 620 may have multiple degrees of freedom, and each actuator may control the actuation of the robot 620 within one or more degrees of freedom in response to control commands (e.g., torque and / or other commands generated based on a control strategy) provided by the robot control system 660. As used herein, the term actuator also encompasses a mechanical or electrical device (e.g., a motor) that produces motion, in addition to any driver (s) that may be associated with the actuator and translates a received control command into one or more signals for driving the actuator. Thus, providing a control command to an actuator may include providing a control command to a driver that translates the control command into an appropriate signal for driving the electrical or mechanical device to produce the desired motion.

[0118] The robot control system 660 may be implemented in one or more processors (such as a CPU, GPU, and / or other controller(s)) of the robot 620. In some implementations, the robot 620 may include a "brainbox" that may include all or some aspects of the control system 660. For example, the brainbox may provide real-time bursts of data to the operating components 604a-n, where each real-time burst includes a set of one or more control commands that specify, among other things, one or more parameters of the motion (if any) of each of one or more of the operating components 604a-n. In various implementations, the control commands may be selectively generated by at least the control system 660 based at least in part on the selected robot action and / or other determination(s) made using a machine learning model stored locally on the robot 620 and trained according to implementations described herein.

[0119] Although in Figure 6 60 is shown as an integral part of the robot 620, but in some implementations, all or some aspects of the control system 660 may be implemented in a component separate from the robot 620 but in communication with it. For example, all or some aspects of the control system 660 may be implemented in one or more computing devices (such as a computer) that communicate with the robot 620 in wired and / or wireless communication. Figure 7 The system is implemented on a computing device 710).

[0120] Figure 7 7 is a block diagram of an example computing device 710 that may optionally be used to perform one or more aspects of the techniques described herein. The computing device 710 typically includes at least one processor 714 that communicates with a number of peripheral devices via a bus subsystem 712. These peripheral devices may include a storage subsystem 724, including, for example, a memory subsystem 725 and a file storage subsystem 726, a user interface output device 720, a user interface input device 722, and a network interface subsystem 716. The input and output devices allow a user to interact with the computing device 710. The network interface subsystem 716 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0121] The user interface input device 722 may include a keyboard, a pointing device (such as a mouse, trackball, touch pad or graphic tablet), a scanner, a touch screen incorporated into a display, an audio input device (such as a voice recognition system, a microphone), and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways to input information into the computing device 710 or a communication network.

[0122] User interface output device 720 may include a display subsystem, a printer, a fax machine, or a non-visual display (such as an audio output device). The display subsystem may include a cathode ray tube (CRT), a flat panel device (such as a liquid crystal display (LCD)), a projection device, or some other mechanism for creating a visual image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and methods for outputting information from computing device 710 to a user or another machine or computing device.

[0123] Storage subsystem 724 stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, storage subsystem 724 may include logic to perform selected aspects of one or more methods described herein.

[0124] These software modules are generally executed by processor 714 alone or in combination with other processors. The memory 725 used in the storage subsystem 724 may include multiple memories, including a main random access memory (RAM) 730 for storing instructions and data during program execution and a read-only memory (ROM) 732 for storing fixed instructions. The file storage subsystem 726 may provide persistent storage for program and data files, and may include a hard drive, a floppy disk drive, and associated removable media, a CD-ROM drive, an optical drive, or a removable media box. Modules that implement the functions of certain implementations may be stored in the storage subsystem 724 by the file storage subsystem 726, or in other machines accessible to (one or more) processors 714.

[0125] The bus subsystem 712 provides a mechanism for the various components and subsystems of the computing device 710 to communicate with each other as intended. Although the bus subsystem 712 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0126] The computing device 710 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, for the purpose of illustrating some implementations, the Figure 7 The description of computing device 710 depicted in FIG. 7 is intended only as a specific example. Many other configurations of computing device 710 may have more Figure 7 The computing devices depicted may have greater or fewer components.

Claims

1. A method implemented by one or more processors, the method comprising: Process the simulated images using the simulated-to-real generator model to generate simulated round-predicted real images, wherein the simulated image is generated by a robot simulator during a simulation round in which a simulated robot attempts to perform a robot task; processing the simulated round predicted real image using a real-to-simulated generator model to generate a simulated round predicted simulated image; Processing the simulated images and simulated robot actions using a task machine learning model trained for use in the robot task to generate a first prediction value; Processing the simulated round prediction real image and the simulated robot action using the task machine learning model to generate a second prediction value; Processing the simulated round prediction simulation image and the simulated robot action using the task machine learning model to generate a third prediction value; generating a loss based on a comparison of the first predicted value, the second predicted value, and the third predicted value; and The simulation-to-real generator model is updated based on the generated loss.

2. The method according to claim 1, further comprising: processing the real image using the real-to-simulated generator model to generate a real turn-predicted simulated image, wherein the real image is captured by a real camera associated with the real robot during a real round of the real robot attempting to perform the robot task; processing the real turn prediction simulated image using the simulated-to-real generator model to generate a real turn prediction real image; Processing the real image and the real robot action using the task machine learning model or an additional task machine learning model trained for use in the robot task to generate a fourth prediction value; Processing the real round prediction simulation image and the real robot action using the task machine learning model or the additional task machine learning model to generate a fifth prediction value; and using the task machine learning model or the additional task machine learning model to process the real round prediction real image and the real robot action to generate a sixth prediction value, Wherein, generating the loss is further based on an additional comparison of the fourth prediction value, the fifth prediction value and the sixth prediction value.

3. The method according to claim 1, wherein: The comparison of the first predicted value, the second predicted value, and the third predicted value includes three comparisons, each of the three comparisons being between a unique pair of the first predicted value, the second predicted value, and the third predicted value.

4. The method according to claim 2, wherein: The additional comparisons of the fourth predicted value, the fifth predicted value, and the sixth predicted value include three additional comparisons, each of the three additional comparisons being between a unique pair of the fourth predicted value, the fifth predicted value, and the sixth predicted value.

5. The method according to claim 2, wherein: Generating the loss is further based on an adversarial loss and / or a cycle consistency loss, wherein the adversarial loss and the cycle consistency loss are generated independently of any output generated using the task machine learning model or the additional task machine learning model.

6. The method according to claim 5, in, The adversarial loss is generated based on whether the simulated-to-real discriminator model predicts whether the predicted real image is an actual real image or a predicted real image generated by the simulated-to-real generator, and Wherein, the cycle consistency loss is generated based on the comparison between the simulated image and the simulated round predicted simulated image.

7. The method according to claim 5, wherein: Generating the loss is further based on both the adversarial loss and the cycle consistency loss.

8. The method according to claim 1, wherein: The task machine learning model represents a Q function, wherein the task machine learning model is trained during reinforcement learning based on the simulation episodes and additional simulation episodes, and wherein the first predicted value is a first Q value, the second predicted value is a second Q value, and the third predicted value is a Q value.

9. The method according to claim 2, further comprising: Generating a task machine learning model loss based on the second prediction value; as well as The task machine learning model is updated based on the task machine learning model loss.

10. The method according to claim 9, wherein: Generating the task machine learning model loss is independent of the first prediction value, the third prediction value, the fifth prediction value, and the sixth prediction value.

11. The method according to claim 10, wherein: Generating the task machine learning model loss is further based on the fourth prediction value.

12. The method according to any one of claims 1 to 11, wherein: The simulation round is an offline round.

13. The method according to any one of claims 1 to 11, wherein: The simulated round is an online round.

14. The method according to any one of claims 1 to 11, wherein: The robotic task is an object manipulation task or a navigation task.

15. The method according to claim 14, wherein: The robotic task is the object manipulation task, and wherein the object manipulation task is a grasping task.

16. A method implemented by one or more processors, comprising: Training a simulation-to-reality generator model, wherein training the simulation-to-reality generator model comprises: iteratively updating the simulation-to-real generator model according to the method of any one of claims 1 to 15; and After training the simulated-to-real generator model: generating a robot control policy for a real robot based on simulated training data generated using the simulated-to-real generator model; and The real robot is controlled based on the generated robot control strategy.

17. The method according to claim 16, in, A plurality of prediction values ​​generated using the task machine learning model are generated based on processing simulated-to-real-to-simulated image triplets using the task machine learning or the additional task machine learning model; as well as Among them, the reinforcement learning scenario consistency loss generated based on the multiple prediction values ​​is generated based on the comparison of the multiple prediction values.

18. A system comprising: A plurality of processors, the plurality of processors being configured to: Performing a method according to any one of claims 1 to 17.

19. A robot comprising: Vision sensors that observe the environment; A trained machine learning model stored in one or more non-transitory computer-readable media, the trained machine learning model trained according to any one of claims 1 to 17; at least one processor, the at least one processor being configured to: processing images captured by the vision sensor and candidate robot actions and using the trained machine learning model to generate predictions; and A determination is made whether to implement the candidate robot action based on the predicted value.

Images