Mixed reality robotic training system and method

Abstract

A robotic training system includes a first computing device and a mixed reality device. The first computing device includes a first program for controlling a mixed reality device. The first program also includes a simulation robot operating in the simulation environment and instructions of the mixed reality robot for display by the mixed reality device. The first computing device may receive input and determine instructions for controlling the simulation robot, and determine instructions for controlling the mixed reality robot based on the input. A simulation robot may be viewed on a first display of a first computing device, and a mixed reality robot may be viewed on a mixed reality device. Further, the first computing device may be connected to the second computing device to train a user of the first computing device to operate the industrial robot. The second computing device may be connected to the physical robot.

Description

Technical Field

[0001] The present disclosure relates to the field of robotic systems, and more particularly to an industrial robot training system using mixed reality. Background Art

[0002] Robotic systems have wide applications in various industries including automotive, aerospace, electronics, medical and other fields, which use robotic systems to improve efficiency and streamline operations. In order to operate these robotic systems, programmers and operators require in-depth training. Typically, the training includes learning how to control a physical robot by observing the robot's behavior based on input to the robot controller. These systems may also include virtual simulator programs that operate in a simulated environment in a computer to simulate the robot's operations or program the operations of a physical robot based on the simulated robot.

[0003] The ability to observe how a robot operates based on user input, to observe how a robot will operate in a specific physical space, and to be able to train on multiple robot models can be a useful tool for the user. However, coordinating training on these systems can be expensive and logistically difficult. The user must go to a training facility, or the robot must be shipped to the customer location. If multiple robots are used, each additional robot further increases the cost. Furthermore, these robots take up a large amount of physical space.

[0004] The global pandemic has further altered the above considerations. Many employers have adopted remote work policies due to shifting social distancing requirements and other reasons. As a result, users may not be located at one central location to be trained on a robotic system. This unprecedented new paradigm creates additional obstacles in training users on robotic systems. Therefore, there is a need for improved robotic training systems. Summary of the invention

[0005] In some embodiments, a method for operating a mixed reality robot training system includes a first computing device and a mixed reality device, the method including: obtaining a first computing device, the first computing device including a processor, a non-transitory computer-readable medium having instructions executable by the processor stored thereon, a first program, a touch screen display, and a user input interface; obtaining a mixed reality device including a camera and a second display; displaying a scene of the mixed reality robot on the second display based on a simulated robot by the mixed reality device; receiving a first input on the user input interface by the first computing device to control the operation of the mixed reality robot at the touch screen display; determining a first instruction for controlling the mixed reality robot based on the first input by the first computing device; transmitting the first instruction to the mixed reality device by the first computing device to control the operation of the mixed reality robot in the scene; and displaying the operation of the mixed reality robot on the second display by the mixed reality device based on the first indication in the scene.

[0006] In some embodiments, the method also includes: displaying, by the first computing device, a first image identifying the first computing device on a first display; capturing, by the mixed reality device using a camera, the first image; and connecting the mixed reality device to the first computing device.

[0007] In some embodiments, the method further includes: displaying, by the mixed reality device, the command log of the first computing device in the scene of the mixed reality robot.

[0008] In some embodiments, controlling the movement of the mixed reality robot based on the first input further includes: controlling the operation of the simulated robot based on the first instruction by the first computing device; and determining the operation of the mixed reality robot based on the first instruction by the first computing device.

[0009] In some embodiments, the robot training system also includes a second computing device, wherein the second computing device includes a processor, a non-transitory computer-readable medium having instructions executable by the processor stored thereon, and a second program, and wherein the method also includes: acquiring the second computing device, connecting the first computing device to the second computing device; receiving the first instruction from the first computing device by the second computing device; and displaying the operation of the simulated robot in the second program based on the first instruction by the second computing device.

[0010] In some embodiments, the robot training system further includes an industrial robot, and the method further includes: connecting the industrial robot to a second computing device; and controlling, by the second computing device, the operation of the industrial robot based on the first instruction from the first computing device.

[0011] In some embodiments, the method also includes: receiving a second input at the second computing device by the second computing device; determining, by the second computing device, a second instruction for controlling the operation of the simulated robot; transmitting, by the second computing device, the second instruction to the first computing device to control the operation of the simulated robot and the mixed reality robot; and controlling, by the first computing device, the operation of the simulated robot and the mixed reality robot based on the second instruction.

[0012] In some embodiments, the method also includes: capturing, by the mixed reality device, multiple images of objects in the scene of the mixed reality robot, and transmitting the multiple images to the first computing device; determining, by the first computing device, the interaction between the object and the mixed reality robot based on the multiple images; determining, by the first computing device, a third instruction based on the interaction between the object and the mixed reality robot; controlling, by the first computing device, the operation of the simulation robot and the operation of the mixed reality robot based on the third instruction; and displaying, by the mixed reality device, the operation of the mixed reality robot based on the third instruction.

[0013] In some embodiments, the method further includes: determining, by the first computing device, a boundary associated with the mixed reality robot, and displaying, by the mixed reality device, the boundary when the object approaches the boundary; wherein determining the interaction between the object and the mixed reality robot includes determining the interaction between the object and the boundary.

[0014] In some embodiments, the object comprises a user's hand; and wherein the interaction comprises a gesture.

[0015] In some embodiments, a method for remotely operating a mixed reality robot training system, the robot training system includes a first computing device and a mixed reality device, the first computing device includes a processor, a non-transitory computer-readable medium having instructions executable by the processor stored thereon, a first program and a first display, and wherein the mixed reality device includes a camera and a second display, the first computing device and the mixed reality device are at a first position, the method including: receiving input by the first computing device to command a simulated robot of the first program; determining by the first computing device a first instruction for controlling the operation of the simulated robot and a second instruction for controlling the operation of the mixed reality robot based on the input; controlling by the first computing device the operation of the simulated robot based on the first instruction and controlling the operation of the mixed reality robot based on the second instruction; transmitting by the first computing device the second instruction to the mixed reality device to display the operation of the mixed reality robot in the scene of the second display.

[0016] In some embodiments, the method further includes: displaying, by the mixed reality device, a log of the instructions of the first program in the scene of the second display.

[0017] In some embodiments, the robot training system also includes a second computing device, wherein the second computing device includes a processor, a non-transitory computer-readable medium having instructions executable by the processor stored thereon, a second program, and a third display, and wherein the method also includes: connecting the first computing device to the second computing device; receiving a second input by the second computing device to control the simulated robot and the mixed reality robot; determining by the second computing device a third instruction for controlling the operation of the simulated robot based on the input and determining by the second computing device a fourth instruction for controlling the operation of the mixed reality robot; transmitting the third instruction and the fourth instruction to the first computing device by the second computing device; and controlling the operation of the simulated robot and the mixed reality robot by the first computing device based on the third instruction and the fourth instruction.

[0018] In some embodiments, the robot training system also includes an industrial robot, and wherein the method further includes: connecting to the industrial robot via a second computing device; wherein transmitting the third instruction to the first computing device also includes transmitting the third instruction to the industrial robot to control the industrial robot.

[0019] In some embodiments, receiving input by the first computing device to command a simulated robot of a first program also includes: determining, by the first computing device, a boundary associated with the mixed reality robot; receiving, by the first computing device, multiple images of an object and the mixed reality robot captured by a camera; and determining, by the first computing device, an interaction between the object and the boundary.

[0020] In some embodiments, the object comprises a user's hand, and wherein the interaction comprises a gesture.

[0021] In some embodiments, a system includes: a first computing device, the first computing device including: a processor, a non-transitory computer-readable medium having instructions executable by the processor stored thereon, a first program, a touch screen display, the first program including a user input interface, a simulated robot and a mixed reality robot; and a mixed reality device, the mixed reality device including a camera, a second display; wherein the first computing device determines a first instruction for controlling the simulated robot based on at least one input at the user input interface; wherein the first computing device determines a second instruction for controlling the mixed reality robot based on at least one input; wherein the first computing device transmits the second instruction to the mixed reality device to display the mixed reality robot on the second display.

[0022] In some embodiments, the system also includes: an industrial robot; and a second computing device; wherein the second computing device includes: a processor, a non-transitory computer-readable medium having instructions executable by the processor stored thereon, and a second program, wherein the second program includes a second user input interface; wherein the second computing device includes being communicatively connected to the first computing device; wherein the first computing device also includes determining a third instruction based on at least one input, and wherein the first computing device transmits the third instruction to the second computing device; wherein the third instruction controls the operation of the industrial robot.

[0023] In some embodiments, the system also includes: a first calibration marker, which is set at a first position in the user's scene; and a second calibration marker, which is set at a second position in the user's scene; wherein the first position and the second position include being positioned a fixed distance apart, and the fixed distance is determined by the first computing device; wherein at least one input includes initiating calibration of the simulated robot and the mixed reality robot; wherein the mixed reality robot moves from the first position to the second position, and the first computing device calibrates the simulated robot and the mixed reality robot based on the movement of the mixed reality robot from the first position to the second position.

[0024] In some embodiments, the mixed reality robot comprises an object defined by a boundary; and wherein the interaction between the object and the boundary comprises at least one input. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Some embodiments of the present disclosure are described herein by way of example only with reference to the accompanying drawings. With specific reference now to the accompanying drawings in detail, it is emphasized that the embodiments shown are provided as examples and for the purpose of illustrative discussion of the embodiments of the present disclosure. In this regard, the description in conjunction with the accompanying drawings makes it clear to those skilled in the art how to practice the embodiments of the present disclosure.

[0026] Figure 1 is a schematic diagram of a system according to some embodiments.

[0027] Figure 2 is a schematic diagram of a system according to some embodiments.

[0028] Figure 3 is a diagram of a simulation program according to some embodiments.

[0029] Figure 4 is a perspective view of a scene of a mixed reality device according to some embodiments.

[0030] Figure 5 is a perspective view of a mixed reality robot of a system according to some embodiments.

[0031] Figure 6 is a perspective view of a mixed reality robot of a system according to some embodiments.

[0032] Figure 7 is a flow chart of a method according to some embodiments.

[0033] Figure 8 is a flow chart of a method according to some embodiments.

[0034] Fig. 9 is a flow chart of a method according to some embodiments.

[0035] Fig.10 is a flow chart of a method according to some embodiments.

[0036] Fig.11 is a flowchart of a method according to some embodiments DETAILED DESCRIPTION

[0037] Among the benefits and improvements that have been disclosed, other objects and advantages of the present disclosure will become apparent from the following description in conjunction with the accompanying drawings. Detailed embodiments of the present disclosure are disclosed herein; however, it should be understood that the disclosed embodiments are merely illustrative of the present disclosure that can be implemented in various forms. In addition, each example given with respect to the various embodiments of the present disclosure is intended to be illustrative rather than restrictive.

[0038] All prior patents and publications cited herein are incorporated by reference in their entirety.

[0039] Throughout the specification and claims, unless the context clearly dictates otherwise, the following terms adopt the meanings explicitly associated herein. The phrases "in one embodiment," "in an embodiment," and "in some embodiments" as used herein do not necessarily refer to the same embodiment, although it may refer to the same embodiment. In addition, the phrases "in another embodiment" and "in some other embodiments" as used herein do not necessarily refer to different embodiments, but they may also be. All embodiments of the present disclosure are intended to be combinable without departing from the scope or spirit of the present disclosure.

[0040] As used herein, the term "based on" is not exclusive and allows for being based on additional factors that are not described unless the context clearly indicates otherwise. In addition, throughout the specification, the meanings of "a", "an", and "the" include plural references. The meaning of "in..." includes "in..." and "on...".

[0041] As used herein, the term "between" does not necessarily require being directly next to other elements. Generally, the term refers to a structure in which something is sandwiched between two or more other objects. At the same time, the term "between" can describe something that is directly adjacent to two opposing things. Therefore, in any one or more embodiments disclosed herein, a specific structural component disposed between two other structural elements may be:

[0042] being disposed directly between two other structural elements so that the particular structural component is in direct contact with the two other structural elements;

[0043] being disposed in close proximity to only one of the two other structural elements such that the particular structural component is in direct contact with only one of the two other structural elements;

[0044] indirectly disposed adjacent to only one of the two other structural elements, such that the particular structural component is not in direct contact with only one of the two other structural elements, and there is another element juxtaposing the particular structural component with the one of the two other structural elements;

[0045] is indirectly arranged between the two other structural elements, so that the particular structural component is not in direct contact with the two other structural elements and other features may be arranged therebetween; or

[0046] Any combination thereof.

[0047] As used herein, "embedded" means that a first material is distributed throughout a second material.

[0048] According to various aspects of the present disclosure, methods and systems for a robotic training system using mixed reality are discussed below. Various embodiments presented herein provide a remote robotic training system without a physical robot at the location where a user operates a mixed reality robot.

[0049] Figure 11 is a schematic diagram of a system 100 according to some embodiments. System 100 may include a first computing device 102 and a mixed reality device 120. In some embodiments, first computing device 102 may include a processor 104, a memory 106, a first program 108, and a first display 112. In some embodiments, mixed reality device 120 may include a camera 122 and a second display 124. In some embodiments, first computing device 102 may be connected to mixed reality device 120 and transmit information between first computing device 102 and mixed reality device 120. First computing device 102 may cause mixed reality device 120 to display mixed reality robot 405 in a real world scene of mixed reality device 120. In addition, mixed reality device 120 may capture one or more images of the scene of mixed reality device 120, transmit the image or images to first computing device 102, and display an image of mixed reality robot 405 on second display 124 based on information received from first computing device 102. In some embodiments, the first computing device 102 may include any of a number of components to operate programs stored in the memory 106, display 112 interface 110, receive input, and connect to the mixed reality device 120 to display the mixed reality robot 405 ( Figure 4 ).

[0050] In some embodiments, the first computing device 102 may include a processor 104. The processor 104 controls the operation of the computer, including executing programs stored in the memory 106. The first processor 104 may execute instructions stored in the memory 106 for effectively controlling the first program 108, including instructions to the mixed reality device 120 for displaying the mixed reality scene. In some embodiments, the processor 104 may include a computer capable of executing the first program 108, connecting to the mixed reality device 120 to send and receive data, displaying the user input interface 110 on the display, receiving input from the user on the user input interface 110, and operating the simulated robot 310 ( Figure 3 ) and mixed reality robot 405( Figure 4 )'s multiple processors 104.

[0051] In some embodiments, the first computing device 102 may include a memory 106. In some embodiments, the memory 106 may include a non-transitory computer-readable medium having instructions executable by the processor 104 stored thereon. In some embodiments, the memory 106 may include a first program 108. In some embodiments, the memory 106 may include any of a plurality of databases stored thereon, the plurality of databases containing any of a plurality of information used by the first computing device 102 to control the first program 108, the mixed reality device 120, the simulation program 116, any other components of the first computing device 102, and combinations thereof. In some embodiments, the memory 106 may include an object database. The object database may include any of a plurality of objects, including simulation objects, mixed reality objects, objects defined by a user, and combinations thereof. In some embodiments, the object database may include pre-loaded objects from any of a plurality of sources including, but not limited to, the mixed reality device 120.

[0052] In some embodiments, the first computing device 102 may include a first display 112. In some embodiments, the first computing device 102 may display an image or multiple images on the first display 112. In some embodiments, the first display 112 may display the first program 108, the user input interface 110, the simulation program 116, the mixed reality robot 405, the command log for the first program 108, the image or multiple images of the mixed reality device 120, other images, and combinations thereof. In some embodiments, the first display 112 may include a touch screen panel to receive input from the user, which is received by the first computing device 102 to control the operation of the first computing device 102. The touch screen display of the first computing device 102 allows the user operating the mixed reality device 120 to input commands to control the mixed reality robot 405 on the first computing device 102 without the need for additional third-party components, such as an external teach pendant controller. In addition, the virtual teach pendant, the simulated robot 310, the command log for the simulated robot 310, and other features displayed by the user input interface 110 on the first display 112 can provide real-time feedback to the user about the operation of the simulated robot 310 and the mixed reality robot 405. Integrating the virtual teach pendant into the first computing device 102 reduces the number of system components, allows the user to control the simulation program 116 and the mixed reality elements, and provides the user with a visual reference that can also be displayed by the second display 124 of the mixed reality device 120. Therefore, in some embodiments, the first computing device 102 can also transmit data related to the user input interface 110 to display the user input interface 110 as a mixed reality element in the real world scene. The user can provide input to the mixed reality user input interface 110 to control the operation of the first program 108, the simulation program 116, the mixed reality robot 405, and combinations thereof.

[0053] In some embodiments, the first computing device 102 may include a first program 108 stored in the memory 106. In some embodiments, the first program 108 of the first computing device 102 may include a user input interface 110 and a simulation program 116. In some embodiments, when executed by the first processor 104, the first program 108 may cause the first computing device 102 to display the user input interface 110 on the first display 112, operate the simulation program 116 including the simulated robot 310 and the mixed reality robot 405, and cause the mixed reality device 120 to display the mixed reality robot 405 and other mixed reality objects on the second display 124 of the mixed reality device 120. In some embodiments, the first computing device 102 may use the first program 108 to send and receive data between the first computing device 102 and the mixed reality device 120. In some embodiments, the first computing device 102 may transmit instructions to the mixed reality device 120 to display a mixed reality image on the display 124. In some embodiments, the mixed reality image may include a mixed reality robot, a mixed reality object, a mixed reality display panel, other markers and objects, and combinations thereof.

[0054] In some embodiments, the first computing device 102 may include a user input interface 110. In some embodiments, the first program 108 may include a user input interface 110. The user input interface 110 may receive one or more inputs from a user to cause the first computing device 102 to perform a function based on the input. For example, a user may provide an input for moving the robot head downward by pulling down a joystick displayed by the user input interface 110, and the first computing device 102 may receive the input and cause the first program 108 to move the simulated robot 310 and the mixed reality robot 405 heads downward. In some embodiments, the user input interface 110 may be displayed on a first display 112 for the user, and may control the operation of the first computing device 102, the first program 108, the simulation program 116, the mixed reality robot 405, other parts of the first program 108, and combinations thereof. In some embodiments, the user input interface 110 may include a teach pendant displayed on the first display 112. In some embodiments, the user input interface 110 may receive user input from an external device that may be connected to the first computing device 102. In some embodiments, the external device connected to the first computing device 102 may include a keyboard, a mouse, a stylus, a stylus pad, a mobile phone, a PDA, a tablet, other devices, and combinations thereof. In addition, in some embodiments, the external device may be connected to the first computing device 102 by any of a variety of methods, including wi-fi, Bluetooth, HDMI, CDMA, LTE, orthogonal frequency division multiplexing (OFDM) including 5G, other connection methods, and combinations thereof. In some embodiments, the user input interface 110 may include an external robot teaching device connected to the first computing device 102.

[0055] In some embodiments, the first computing device 102 may include the simulation program 116. In some embodiments, the first program 108 of the first computing device 102 may include the simulation program 116. In some embodiments, the simulation program 116 may include the simulation environment 305 and the simulated robot 310 operating in the simulation environment 305. In some embodiments, the first computing device 102 may control the operation of the simulation program 116. In some embodiments, the first computing device 102 may receive user input and determine instructions for controlling the operation of the simulation program 116, the simulation environment 305, the simulated robot 310, other components, and combinations thereof based on the user input. In some embodiments, the first computing device 102 that controls the simulated robot 310 by providing instructions to the simulation program 116 may also cause the first computing device 102 to control the mixed reality device 120 to display the mixed reality robot 405 and the operation of the mixed reality robot 405. Therefore, in some embodiments, the first computing device 102 that controls the operation of the simulation program 116 and the simulated robot 310 may also cause the first computing device 102 to display the operation of the mixed reality device 120 on the mixed reality robot 405. In some embodiments, control of the mixed reality robot 405 may be based at least in part on operation of the simulated robot 310 in the simulation program 116. In some embodiments, the simulation program 116 may include mixed reality objects displayed by the mixed reality device 120 including the mixed reality robot 405.

[0056] In some embodiments, the first computing device 102 may receive input to command the operation of the simulated robot 310 and the mixed reality robot 405. In some embodiments, the first computing device 102 may receive user input and determine a first instruction for controlling the operation of the simulated robot 305 based on the input. In some embodiments, the first instruction may control the operation of the simulated robot 310 and the mixed reality robot 405. In some embodiments, the first computing device 102 may also determine a second instruction for controlling the operation of the mixed reality robot 405. In some embodiments, the first instruction may include the second instruction. In other embodiments, the second instruction may be based at least in part on the first instruction. In some embodiments, the second instruction may also be based on feedback from the simulated robot 310 operating in the first program 108.

[0057] In some embodiments, the first computing device 102 may include an input-output interface 114. The input-output interface 114 may include any of a plurality of communication components to allow the first computing device 102 to communicate with other components of the first computing device 102, external devices including the mixed reality device 120, other computers, and combinations thereof. In some embodiments, the input-output interface 114 may connect the first computing device 102 to the mixed reality device 120. Thus, in some embodiments, the first computing device 102 may use the input-output interface 114 to communicate with the mixed reality device 120 to partially display an image or image sequence from the camera 122, display a mixed reality robot, display a mixed reality object, transmit user input, other signals, and combinations thereof. In some embodiments, the input-output interface 114 may connect the first computing device 102 to the network 180. In some embodiments, the first computing device 102 may use the input-output interface 114 to connect to the second computing device 132, as will be discussed further below. In some embodiments, the first computing device 102 may connect to a server or network by using the input-output interface 114 to send and receive information to the server or network.

[0058] In some embodiments, the first computing device 102 may include a communication port or antenna to allow the first computing device 102 to communicate with other devices and / or networks. In some embodiments, the antenna and other communication ports may be located on the input-output interface 114. In some embodiments, the input-output interface 114 may include a WWAN radio using a WWAN protocol selected from the group consisting of the 3rd Generation Partnership Project (3GPP) Long Term Evolution (LTE) standard and the Institute of Electrical and Electronics Engineers (IEEE 802.16 standard. In some embodiments, the input-output interface 114 may include a WLAN radio using a WLAN protocol selected from the group consisting of the IEEE 802.11 standard, the IEEE 802.15 standard, the Bluetooth standard, the Wireless Display Port standard, the Wiggig standard, the Ultra Wideband (UWB) standard, the Wireless HD standard, and the Wireless Home Digital Interface (WHDI) standard.

[0059] In some embodiments, the first computing device 102 and the mixed reality device 120 may be at a first location. In some embodiments, the first computing device 102 may be associated with a first user at the first location, and the mixed reality device 120 may be a wearable device worn by the first user at the first location. In some embodiments, the first computing device 102 may include a mobile device held by the first user, and the mixed reality device 120 may include a wearable device worn by the user.

[0060] In some embodiments, the system 100 may include a mixed reality device 120. In some embodiments, the mixed reality device 120 may include a camera 122. The camera 122 may capture a scene of the environment of the mixed reality device 120 in the field of view of the camera 122. In some embodiments, the camera 122 may be a first camera 122. In some embodiments, the mixed reality device 120 may include multiple cameras to capture multiple images of the mixed reality device 120 and / or a scene of a first user wearing the mixed reality device 120. In some embodiments, the image captured by the camera 122 may be a single image. In other embodiments, the image captured by the camera 122 may include multiple images, videos, and combinations thereof. In some embodiments, the system 100 may include a third camera connected to the mixed reality device 120, the first computing device 102, and combinations thereof. In some embodiments, the mixed reality device 120 may also include a processor, a memory, a GPU, other components, and combinations thereof to capture images of the mixed reality device 120, process one or more images captured by the camera 122, and display mixed reality elements on a second display 124.

[0061] In some embodiments, the mixed reality device 120 may transmit an image to the first computing device 102. In some embodiments, the mixed reality device 120 may receive an image captured by the camera 122, convert the image into data, and transmit the data to the first computing device 102. The first computing device 102 may receive data from the mixed reality device 120. In some embodiments, the mixed reality device 120 may also determine other information based on one or more images, such as position data, reference position, depth, scale, other factors, and combinations thereof, and transmit the information to the first computing device 102. In some embodiments, the first computing device 102 may receive image data and determine such information based on the image data. In some embodiments, the camera 122 may capture an image of a physical object in the user's scene. In some embodiments, the first computing device 102 may receive an image and identify the presence of an object in the scene. In some embodiments, the first computing device 102 may receive an image captured by the camera 122 to determine the interaction between the object of the scene and the mixed reality scene. In some embodiments, the object may include a physical object such as a calibration point. In some embodiments, the object may include an object of the user, including but not limited to the user's hand. In some embodiments, the camera 122 may capture an image of the object. First computing device 102 may receive an image of an object and may determine that the object may be interacting with objects of a mixed reality scene including a mixed reality robot. Thus, in some embodiments, first computing device 102 may detect object interactions and may control the operation of mixed reality scene or simulation program 116 based on the object interactions.

[0062] In some embodiments, the camera 122 of the mixed reality device 120 may capture one or more images of the scene of the mixed reality device 120. The first computing device 102 may receive information from the mixed reality device 120 and may determine instructions for displaying the mixed reality robot based on the image. The first computing device 102 may transmit instructions related to the display of the mixed reality robot 405, other mixed reality objects, and combinations thereof to the mixed reality device 120. For example, the mixed reality device 120 may capture an image of a room and transmit (multiple) images of the room to the first computing device 102. The first computing device 102 receives the image and determines that the mixed reality robot 405 can be positioned at one of multiple locations in the room based on the image and the size of the mixed reality robot 405 model. The first computing device 102 may transmit instructions to the mixed reality device 120 to display the mixed reality robot at one of the determined locations. The mixed reality device 120 may receive the instructions and display the mixed reality robot at the location. The user may view the mixed reality robot at the location. In some embodiments, the mixed reality robot 405 displayed by the mixed reality device 120 and the first computing device 102 may be scaled based on the real world scene. For example, if the default size of the mixed reality robot 405 is not appropriate for the size of the real-world scene, the first computing device 102 can scale the display of the mixed reality robot 405 based on user input of the desired scale, or the first computing device 102 can determine an adjusted scale to display the mixed reality robot 405.

[0063] In some embodiments, the second display 124 may include a transparent display. In some embodiments, the second display 124 may include a translucent display. In some embodiments, the second display 124 may be a non-transparent display. In some embodiments, the second display 124 may display an image of a mixed reality scene on a real world scene, both of which are viewable by the user. In some embodiments, the camera 122 may capture an image of the scene of the first user and display the captured one or more images together with the image of the mixed reality scene on the second display 124. The transparent display may allow the mixed reality device 120 to display the mixed reality scene on a real world scene viewable by the first user.

[0064] In some embodiments, the mixed reality device 120 may include, but is not limited to, a processor, a non-transitory computer-readable medium having instructions executable by the processor stored thereon, buttons, speakers, sensors, antennas, other components, and combinations thereof. In some embodiments, the mixed reality device 120 may include any device capable of connecting to the first computing device 102 and displaying a mixed reality scene in a real-world scene. In some embodiments, the mixed reality device 120 may be a wearable device that can be worn by the first user. For example, the first user may wear the wearable device on the head of the first user. The mixed reality device 120 may include, for example, Microsoft's holographic lens, Magic Leap's One, Google Glasses, or any other mixed reality device including a wearable device with a transparent screen.

[0065] Figure 2 1 is a schematic diagram of a system 100 according to some embodiments. In some embodiments, the system 100 may further include a second computing device 132. The second computing device 132 may include a second processor 134, a second memory 136, a second program 138, and a third display 142. In some embodiments, the second program 138 may include a second user input interface 140 and a second simulation program 146. In some embodiments, the second computing device 132 may include any of a plurality of components to allow the second computing device 132 to connect to the first computing device 102, the network 180, the industrial robot 150, and combinations thereof, and to control the first computing device 102, the second computing device 132, the industrial robot 150, and combinations thereof.

[0066] In some embodiments, the second computing device 132 may include a second processor 134 and a second memory 136. The second processor 134 may be configured to execute instructions stored in the second memory 136. In some embodiments, the second memory 136 may include a non-transitory computer-readable medium having instructions executable by the processor stored thereon. In some embodiments, the second memory 136 may include a second program 138 stored thereon.

[0067] Second computing device 132 may include third display 142. In some embodiments, third display 142 may include a touch screen display to display images from second computing device 132 and receive input from a user. Thus, in some embodiments, second computing device 132 may be connected to an input device capable of receiving input from a user to control second computing device 132. In some embodiments, the input device may include a keyboard, a touch pad, a laptop, a tablet, a mobile device, other devices capable of receiving and sending input, and combinations thereof.

[0068] In some embodiments, second computing device 132 may display one or more images showing second program 138, second user input interface 140, second simulation program 146, simulation program 116, a mixed reality robot of first computing device 102, other images, and combinations thereof. For example, second computing device 132 may be connected to first computing device 102, and third display 142 may display an image being displayed by first computing device 102 on first display 112. Alternatively, second computing device 132 may display an image captured by camera 122 of mixed reality device 120 and show a virtual robot operating in the scene of mixed reality device 120 as seen by the user.

[0069] In some embodiments, second computing device 132 may include second program 138. In some embodiments, second program 138 may be the same program as first program 108. In some embodiments, second program 138 may be a different program than first program 108. In some embodiments, each of first program 108 and / or second program 138 may include advanced permissions accessible by a user with credentials for advanced permissions. For example, second computing device 132 may be a computing device of a second user at a second location, and the second user may use second computing device 132 and / or second program 138 to connect to first computing device 102 and / or first program 108 to control first computing device 102 including simulated environment 305, simulated robot 310, and mixed reality robot 405.

[0070] In some embodiments, the second program 138 may include a second user input interface 140 and a second simulation program 146. In some embodiments, the second user input interface 140 may include a virtual teach pendant displayed on the third display 142. In some embodiments, the user may operate the second user input interface 140 to control and operate the simulated robot of the second computing device 132, the simulated robot of the first computing device 102, the mixed reality robot of the first computing device 102, and a combination thereof. In some embodiments, the second user input interface 140 may include an external input device connectable to the second computing device 132. The external input device may receive user input and transmit the input to the second computing device 132. The second computing device 132 may receive the input and determine any one of a plurality of actions based on the input, which are used to control the operation of the second computing device 132, the second program 138, the second simulation program 146, the first computing device 102, other devices, and a combination thereof.

[0071] In some embodiments, second computing device 132 may be connected to first computing device 102. In some embodiments, second computing device 132 may be connected to a server or network and connected to first computing device 102 through the server or network. In some embodiments, second computing device 132 may be connected to first computing device 102 and send and receive information between first computing device 102 and second computing device 132. Thus, in some embodiments, second computing device 132 may control the operation of first computing device 102, which includes, but is not limited to, first program 108, simulation program 116, mixed reality robot, and combinations thereof.

[0072] In some embodiments, the second program 138 may include a second simulation program 146. In some embodiments, the second simulation program 146 may include a second simulation environment 305, a second simulation robot 310, and a combination thereof. In some embodiments, the second computing device 132 may control a second simulation program 146 including a second simulation environment 305 and a second simulation robot 310. In some embodiments, the simulation program 116 may be a program that is the same or similar to the second simulation program 146. In some embodiments, the simulation program 116 and the second simulation program 146 may be different programs with different coding to perform different functions. In some embodiments, the simulation program 116 and the second simulation program 146 may include multiple user profiles. Each of the multiple user profiles may include different privileges to allow each user profile to perform different operations. Therefore, in some embodiments, a first user of the first computing device 102 may have a first user profile, and a second user of the second computing device 132 may have a second user profile. The first user profile may allow the first user to connect to the first computing device 102 and the mixed reality device 120. The second user profile may allow the second user to connect to the second computing device 132, the first computing device 102, the mixed reality device 120, the industrial robot 150, and combinations thereof. Additionally, in some embodiments, the second user profile may allow the first computing device 102 to connect and control the operation of the industrial robot 150.

[0073] In some embodiments, the instructions for controlling the second simulation program 146 by the second computing device 132 may also transmit instructions to the first computing device 102 to control the operation of the first simulated robot 310, the mixed reality robot 405, and combinations thereof. In addition, in other embodiments, the instructions for controlling the operation of the mixed reality robot 405, the first simulated robot 310, and combinations thereof from the first computing device 102 may be sent by the first computing device 102 to the second computing device 132 to control the operation of the second computing device 132, which includes the second simulated robot 310, the mixed reality robot 405, the industrial robot 150, and combinations thereof.

[0074] In some embodiments, the second computing device 132 may include a second input-output interface 144. In some embodiments, the second computing device 132 may be connected to a server, a network 180, the first computing device 102, the industrial robot 150, and combinations thereof. In some embodiments, the second computing device 132 may be connected to the first computing device 102 and transmit commands, images, or image sequences between the second computing device 132 and the first computing device 102. In some embodiments, the second computing device 132 may be connected to the first computing device 102 via the network 180 using the second input-output interface 144. In some embodiments, the input-output interface 114 may include a communication port or antenna that allows the first computing device 102 to communicate with other devices and / or networks. In some embodiments, the input-output interface 114 may include a WWAN radio using a WWAN protocol selected from the group consisting of the 3rd Generation Partnership Project (3GPP) Long Term Evolution (LTE) standard and the Institute of Electrical and Electronics Engineers (IEEE) 802.16 standard. In some embodiments, the input-output interface 114 may include a WLAN radio using a WLAN protocol selected from the group consisting of IEEE 802.11 standard, IEEE 802.15 standard, Bluetooth standard, Wireless DisplayPort standard, Wigg standard, Ultra Wideband (UWB) standard, Wireless HD standard, and Wireless Home Digital Interface (WHDI) standard.

[0075] In some embodiments, the second computing device 132 may be at the first location. In some embodiments, the second computing device 132 may be at the second location. In some embodiments, the second location may be remote from the first location. For example, the first location may be the home of the first user, and the second location may be the office of the industrial robot trainer.

[0076] In some embodiments, the system 100 may include an industrial robot 150. In some embodiments, the industrial robot 150 may be connected to the second computing device 132. In some embodiments, the industrial robot 150 may be connected to the first computing device 102 through the second computing device 132, the network 180, and a combination thereof. In some embodiments, the second computing device 132 that controls the operation of the second simulation program 146 may also control the operation of the industrial robot 150 based on the operation of the second simulated robot of the second simulation program 146. In addition, in some embodiments, receiving instructions for controlling the second simulation program 146 from the first computing device 102 may include instructions for controlling the operation of the industrial robot 150.

[0077] In some embodiments, the second computing device 132 may control the operation of the industrial robot 150 connected to the second computing device 132. In addition, in some embodiments, the second computing device 132 may control the operation of the simulation program 116 of the first computing device 102, the second simulation program 146, the mixed reality robot of the first computing device 102, the industrial robot 150, and combinations thereof. In other embodiments, the first computing device 102 may control the operation of the mixed reality robot, the simulation program 116, the second simulation program 146, the industrial robot 150, and combinations thereof. In some embodiments, one of the first computing device 102 and the second computing device 132 may communicate with the other of the first computing device 102 and the second computing device 132 to control the operation of the simulation program 116, the second simulation program 146, the mixed reality robot, the industrial robot 150, other objects, and combinations thereof.

[0078] In some embodiments, the industrial robot 150 may be at the location of the first computing device 102. In some embodiments, the industrial robot 150 may be at the location of the second computing device 132. In some embodiments, the industrial robot 150 may be at a third location remote from the first location and the second location. In some embodiments, the second computing device 132 may include a controller for the industrial robot 150 to control the operation of the industrial robot 150.

[0079] Figure 3 is a diagram of a simulation program 116 according to some embodiments. Figure 4 is a perspective view of a mixed reality robot 405 according to some embodiments. Unless specific reference is made to a particular figure, the following will be discussed together. Figure 3 -4.

[0080] refer to Figure 3 In some embodiments, the simulation program 116 may include a first simulated environment 305 and a first simulated robot 310. In some embodiments, the first computing device 102 may control the simulation program 116 to display the simulated environment 305 and the simulated robot 310 operating in the simulated environment 305. In some embodiments, the first computing device 102 may receive input from a user, determine instructions for controlling the operation of the robot based at least in part on the input, and transmit the instructions to the simulation program 116 to control the operation of the simulated environment 305, the simulated robot 310, and a combination thereof. In some embodiments, commands to the simulation program 116 may also be sent to the mixed reality device 120 to control the operation of the mixed reality robot. In some embodiments, the input may be received by the mixed reality device 120 and transmitted to the first computing device 102 to command the operation of the simulated robot, the mixed reality robot, the industrial robot 150, and a combination thereof.

[0081] In some embodiments, the simulation program 116 may also include a simulation object 315 in the simulation environment 305. In some embodiments, the simulation object 315 may be associated with a physical object of a real-world scene. Therefore, in some embodiments, the user may define the simulation object 315 in the first program 108 for use in the simulation program 116. In some embodiments, the simulation object 315 may be created by designing the simulation object 315 using a third program, and data representing the simulation object 315 may be saved to the memory 106 of the first computing device 102 and used by the first program 108. In some embodiments, the user may create the simulation object 315 by using the first program 108.

[0082] In some embodiments, the first computing device 102 can calibrate the simulated robot 310, the mixed reality robot 405, and a combination thereof. In some embodiments, the simulated robot 310 can be calibrated to the mixed reality robot 405. Calibration ensures proper coordination and precise control of the mixed reality robot 405 and the simulated robot 310. Therefore, proper calibration provides instructions for controlling the operation of the simulated robot 310 and the mixed reality robot 405, which will result in substantially similar operations between the simulated robot 310 and the mixed reality robot 405.

[0083] In some embodiments, the first computing device 102 may include calibration of the mixed reality robot 405 and the simulated robot 310 upon launching the first program 108. In some embodiments, the user may initiate calibration. In some embodiments, the first computing device 102 may initiate calibration if an error is detected between the operation of the simulated robot 310 and the mixed reality robot 405. In some embodiments, the first computing device 102 may determine that calibration is needed based at least in part on the operation of the simulated robot 310 in the simulation program 116 and the movement of the mixed reality robot 405 in the scene of the mixed reality device 120 visible on the second display 124.

[0084] In some embodiments, the simulation program 116 may include calibration points 320. In some embodiments, the first computing device 102 may calibrate the simulated robot 310 and the mixed reality robot 405 by determining calibration points for the simulated robot 310 and the mixed reality robot 405. In some embodiments, the first computing device 102 may use two calibration points. In some embodiments, the first computing device 102 may use three calibration points. In some embodiments, the first computing device 102 may use more than three calibration points. In some embodiments, the first computing device 102 may determine the number of calibration points based on the axis being calibrated. The first computing device 102 determines the calibration position for each calibration point. In some embodiments, the first computing device 102 may display the calibration points for the simulated robot 310, including calibration point positions, distances, angles, other information, and combinations thereof. Therefore, in some embodiments, the user can locate calibration markers in the real world scene based on the calibration points shown in the simulation program 116 and by the first computing device 102. For example, the first computing device 102 may provide instructions for positioning calibration marks on the surface shown on the first display 112, the first calibration mark and the second calibration mark being a first distance apart on the first axis, and the third calibration mark being a second distance apart on the second axis, perpendicular to the first axis, and from the first mark and the second mark. The first computing device 102 may then calibrate the simulated robot 310 and the mixed reality robot 405 based on the known positions of the first calibration mark, the second calibration mark, and the third calibration mark to align the simulated robot 310 with the mixed reality robot 405.

[0085] refer to Figure 4 , the mixed reality device 120 may receive instructions from the first computing device 102 to cause the second display 124 to display the mixed reality robot 405, the calibration point 420, other mixed reality objects, and combinations thereof. In some embodiments, the second display 124 may display the movement and operation of the mixed reality robot 405 based on instructions from the first computing device 102, user input to the first computing device 102, user input based on interaction with the mixed reality environment including the mixed reality robot 405, and combinations thereof.

[0086] In some embodiments, the first computing device 102 may send instructions to control the movement of the mixed reality robot 405, and the second display 124 may display the mixed reality robot 405 in the scene of the first user. Thus, in some embodiments, based on the instructions from the first computing device 102 for moving the mixed reality robot 405 from the first position 425 to the second position 430, the mixed reality device 120 may receive the instructions and cause the second display 124 to display the movement of the mixed reality robot 405. In some embodiments, the movement of the mixed reality robot 405 may include displaying the mixed reality robot 405 at the first position 425, and displaying the robot or a portion of the robot in the second position 430, the second position 430 being the end position of the mixed reality robot 405 based on the instructions received by the first computing device 102. Thus, the user may observe and anticipate the movement of the mixed reality robot 405. In some embodiments, the first position may be represented by the current position of the mixed reality robot 405. In some embodiments, the second position may include the end position of the mixed reality robot 405 based on the instructions received by the first computing device 102.

[0087] In some embodiments, the mixed reality robot 405 can be calibrated by the first computing device 102. In some embodiments, the mixed reality device 120 can capture an image of the calibration mark 410. The first computing device 102 can receive the image from the mixed reality device 120, and can then determine the calibration point 420 based on the image of the calibration mark 410. In some embodiments, the mixed reality robot 405 can be calibrated by positioning multiple calibration marks 410 in the real world scene. In some embodiments, each of the multiple calibration marks can be associated with a calibration point 320 in the simulated environment 305. In some embodiments, the first program 108 can generate an image of the calibration mark 410 and display the calibration mark 410 on the first display 112 or some other display connected to the first computing device 102. In some embodiments, the image of the calibration mark 410 can be printed and used in the real world scene. In some embodiments, the calibration mark 410 can include an image including a calibration point for the mixed reality robot 405. In some embodiments, the calibration mark 410 can be a scannable QR code. In some embodiments, the calibration mark 410 can include other images, including but not limited to barcodes, text, other coded images including readable instructions, and combinations thereof.

[0088] Figure 5 is a view of display 124 on mixed reality device 120 according to some embodiments. Figure 6 is a diagram of a display 124 on a mixed reality robot 120 according to some embodiments. Unless specific reference is made to a particular figure, the following will be discussed together. Figure 5 -6.

[0089] refer to Figure 5 In some embodiments, each mixed reality object may include a boundary 505 defined by the first computing device 102. The boundary 505 generally defines the shape of the mixed reality object viewed through the second display 124. In some embodiments, the boundary 505 may include the outer shape of the mixed reality object, including the mixed reality robot 405. In some embodiments, each mixed reality object displayed by the mixed reality device 120 may include a boundary 505 that may define the shape of each object. In some embodiments, interaction with the boundary 505 may provide input to the first computing device 102 to control the mixed reality object. In some embodiments, the boundary 505 may be associated with the shape of the mixed reality object rather than defining the shape of the mixed reality object.

[0090] In some embodiments, the first computing device 102 may receive an image captured by the mixed reality device 120 of the object 510 interacting with the boundary 505 of the mixed reality object. In some embodiments, the first computing device 102, the mixed reality device 120, and a combination thereof may determine an interaction between a mixed reality object (e.g., the mixed reality robot 405) and the object 510 based in part on the mixed reality device 120 capturing the image of the object 510 interacting with the boundary 505 of the mixed reality robot 405. For example, based on the object 510 pressing the boundary 505 of the mixed reality robot 405 in the scene captured by the mixed reality device 120, the object 510 may push the mixed reality robot 405 in a particular direction. In some embodiments, the user input to the first computing device 102 may be based on the interaction between the boundary 505 of the mixed reality object and the object 510. Therefore, in some embodiments, similar to receiving input at the user input interface 110, the first computing device 102 may determine instructions for controlling the operation of the simulated robot 310, the mixed reality robot 405, and a combination thereof based on the interaction of the object 510 with the boundary 505 of the mixed reality object.

[0091] refer to Figure 6, in some embodiments, the boundary 505 may be the first boundary 505, and the first computing device 102 may determine a second boundary 605 associated with the mixed reality object. In some embodiments, the second boundary 605 may extend beyond the dimensions of the mixed reality object and the first boundary 505. The second boundary 605 may be represented by any of a plurality of shapes, lines, graphics, other graphical representations, and combinations thereof. In some embodiments, the boundary 605 may be an outline shape around the boundary 505 of the mixed reality object. In some embodiments, the second boundary 605 may not be displayed by the mixed reality device 120 unless the first computing device 102 determines an interaction between the object 510 and the second boundary 605. In some embodiments, the interaction between the object 510 and the second boundary 605 may cause the first computing device 102 to transmit an instruction to display the second boundary 605. In some embodiments, the first computing device 102 may determine the interaction between the object 510 and the second boundary 605 as an input command to the first computing device 102. Thus, the first computing device 102 can receive input from the mixed reality device 120 based on the interaction of the object 510 and the second boundary 605, determine instructions based on the input, and transmit the instructions to the simulation program 116, the simulated robot 310, the mixed reality robot 405, and a combination thereof to control the operation of the simulated robot 310 and the mixed reality robot 405.

[0092] In some embodiments, object 510 may include any of a number of objects that interact with mixed reality robot 405. In some embodiments, object 510 may include mixed reality objects, calibration markers, hands, other body parts, other objects, and combinations thereof.

[0093] Figure 7 is a flow chart of method 700 according to some embodiments. In some embodiments, method 700 involves operating a mixed reality robotics training system, which may include first computing device 102 and mixed reality device 120.

[0094] At 705, method 700 may include acquiring a first computing device 102. In some embodiments, first computing device 102 may be at a first location of a first user. In some embodiments, first computing device 102 may include processor 104, memory 106, first program 108, first display 112, user input interface 110, and simulation program 116.

[0095] In some embodiments, the first program 108 may include a simulation program 116 in which the simulated robot 310 operates in the simulated environment 305. In some embodiments, the first program 108 may also control the operation of the mixed reality robot 405 and other mixed reality objects displayed by the mixed reality device 120.

[0096] In some embodiments, the first display 112 may include a touch screen display. The touch screen display may display images and receive input from a user at the first computing device 102. In some embodiments, the first program 108 may include a user input interface 110 to receive input to the first computing device 102 at the first display 112 or some other input device. In some embodiments, the first computing device 102 may display the user input interface 110 on the first display 112. In some embodiments, the first computing device 102 may display other images on the first display 112. In some embodiments, the first computing device 102 may receive an image from the camera 122 and overlay a mixed reality image onto the image and display the image on the first display 112.

[0097] At 710, method 710 may include acquiring a mixed reality device 120. In some embodiments, the mixed reality device 120 may be at a first location. In some embodiments, each of the first computing device 102 and the mixed reality device 120 may be at a location of a first user. In some embodiments, the mixed reality device 120 may include a camera 122 and a second display 124. In some embodiments, the mixed reality device 120 may be connectable to the first computing device 102. In some embodiments, the mixed reality device 120 may be hardwired into the first computing device 102. In some embodiments, the mixed reality device 120 may be connected to the first computing device 102 via Wi-Fi, Bluetooth, other wireless connection schemes, and combinations thereof.

[0098] At 715, the method 700 can receive, by the first computing device 102, a first input on the user input interface 110 to control the operation of the mixed reality robot 405. In some embodiments, the first input can be received at the first display 112, and the first display 112 includes a touch screen display. In some embodiments, the first input can be received on an external input device. In some embodiments, the first input can be received based on an interaction with the mixed reality robot 405, which will be further described below.

[0099] At 720, the method 700 may include determining, by the first computing device 102, a first instruction for controlling the mixed reality robot 405 based on the first input. In some embodiments, the first computing device 102 may determine instructions for controlling the simulated robot 310 and the mixed reality robot 405 based on the first input. In some embodiments, the first computing device 102 may determine a first instruction for controlling the simulated robot 310 and a second instruction for controlling the mixed reality robot 405. In some embodiments, the first instruction and the second instruction may include the same instruction. In some embodiments, the first instruction may control each of the simulated robot 310 and the mixed reality robot 405. In some embodiments, the first instruction may include instructions for the mixed reality device 120 to display the mixed reality robot 405 in a scene of the second display 124. The image on the mixed reality device 120 may be viewed by a user wearing the mixed reality device 120. In some embodiments, the first program 108 determines instructions for controlling each of the simulated robot 310 and the mixed reality robot 405 based on the first input.

[0100] In some embodiments, the input may include user input at the user input interface 110. In some embodiments, the input may be on an external input device. In some embodiments, the input may come from an interaction of the object 510 with the boundary 505 of the mixed reality robot 405. In some embodiments, the input may be an interaction of the object 510 with the second boundary 605 of the mixed reality robot 405. In some embodiments, the input may come from the simulation program 116. In some embodiments, the first input may include at least one input at the first computing device 102. In some embodiments, the first input may include multiple inputs.

[0101] At 725, the method 700 may include transmitting the first instructions to the mixed reality device 120. In some embodiments, the first instructions may also include instructions for controlling the operation of the simulated robot 310. In some embodiments, the first instructions determined by the first computing device 102 may be transmitted to each of the simulated robot 310 and the mixed reality robot 405 to control each of the simulated robot 310 and the mixed reality robot 405.

[0102] At 730, the method 700 may include displaying, by the mixed reality device 120, the operation of the mixed reality robot 405 based on the first instruction. In some embodiments, the operation of the mixed reality robot 405 may include movement of the mixed reality robot 405 in the real world scene on the second display 124. In some embodiments, the operation of the mixed reality robot 405 may include calibration of the mixed reality robot 405 to the simulated robot 310. In some embodiments, the operation of the mixed reality robot 405 may include other operational functions of the mixed reality robot 405, including movement caused by user input at the user input interface 110, detected by the mixed reality device 120, or caused by some other device.

[0103] Figure 8 is a flow diagram of method 700 according to some embodiments. At 805, method 800 may include displaying, by first computing device 102, on first display 112, a first image identifying first computing device 102. In some embodiments, the first image may include data representing an identification of first computing device 102. In some embodiments, the first image may include an IP address of first computing device 102. In some embodiments, the first image may include other identification information of first computing device 102. In some embodiments, mixed reality device 120 may pair with first computing device 102 by detecting the first image using camera 122. In some embodiments, the first image may include an encoded image containing information from first computing device 102. In some embodiments, the first image may include a QR code or other scannable image.

[0104] At 810 , method 700 may include capturing, by mixed reality device 120 , a first image using camera 122 . In some embodiments, mixed reality device 120 may capture an image, and instructions encoded into the image may cause mixed reality device 120 to connect to first computing device 102 .

[0105] At 815, method 700 may include connecting, by the first computing device 102, the mixed reality device 120 to the first computing device 102. The first computing device 102 is connected to the mixed reality device 120 to communicate with each of the first computing device 102 and the mixed reality device 120, including receiving data related to images captured by the mixed reality device 120 and data related to displaying the operation of the mixed reality robot 405.

[0106] Fig. 98 is a flow diagram of method 800 according to some embodiments. At 905, method 700 may include displaying, by the mixed reality device 120, a log of commands of the first computing device 102 for the mixed reality robot 405 on the second display 124. In some embodiments, the log may be a mixed reality object displayed on the second display 124. In some embodiments, the log may include commands to the simulated robot 310 and the mixed reality robot 405. In some embodiments, the log of commands may be displayed on the first display 112 and as an object on the second display 124.

[0107] Fig.10 700 according to some embodiments. In some embodiments, at 1005, method 700 may include acquiring a second computing device 132. In some embodiments, second computing device 132 may be at a first location. In some embodiments, second computing device 132 may be at a second location remote from first computing device 102.

[0108] In some embodiments, at 1010, method 700 may include connecting first computing device 102 to second computing device 132. In some embodiments, first computing device 102 is connected to second computing device 132. In some embodiments, second computing device 132 may include second program 138 that communicates with first computing device 102. In some embodiments, second computing device 132 may include industrial robot 150. In some embodiments, industrial robot 150 may be connected to second computing device 132.

[0109] In some embodiments, at 1015, method 700 may include receiving, by second computing device 132, first instructions from first computing device 102. In some embodiments, first computing device 102 may transmit the first instructions to second computing device 132. In some embodiments, the first instructions received by second computing device 132 may include instructions regarding the operation of a simulated robot to be displayed on third display 142 of second computing device 132. In some embodiments, the first instructions may control the operation of industrial robot 150 connected to second computing device 132.

[0110] In some embodiments, at 1020 , the method 700 may include displaying, by the second computing device 132 , in the second program 138 based on the first instructions, operations of the simulated robot based on the first instructions.

[0111] Fig.111 is a flow chart of method 700 according to some embodiments. In some embodiments, at 1105, method 700 may include capturing, by mixed reality device 120, a plurality of images of object 510 in a scene of a mixed reality robot, and transmitting the plurality of images to first computing device 102. In some embodiments, object 510 may include any one of a plurality of objects. In some embodiments, object 510 may include a body part of a user. In some embodiments, object 510 may include a hand of a user.

[0112] In some embodiments, at 1110, method 700 may include determining, by first computing device 102, an interaction between object 510 and mixed reality robot 405 based on the plurality of images. In some embodiments, the interaction may include object 510 approaching boundary 505 of mixed reality robot 405. In some embodiments, the interaction may include object 510 exerting a force on boundary 505, as captured by the plurality of images of mixed reality device 120.

[0113] In some embodiments, at 1115 , method 700 may include determining, by first computing device 102 , a third instruction based on the interaction.

[0114] In some embodiments, at 1120, the method 700 may include controlling, by the first computing device 102, the operation of the simulation robot 310 and the operation of the mixed reality robot 405 based on the third instructions. In some embodiments, the operation may include moving a portion of the mixed reality robot 405, calibrating the mixed reality robot 405, repositioning the mixed reality robot 405, other operations, and combinations thereof.

[0115] In some embodiments, at 1125, the method 700 may include displaying, by the mixed reality device 120, the operation of the mixed reality robot 405 based on the third instruction. The mixed reality device 120 may receive the third instruction from the first computing device 102, and the mixed reality device 120 displays the movement of the mixed reality robot 405 based at least in part on the third instruction.

[0116] In some embodiments, at 1130, method 700 may include displaying, by mixed reality device 120, boundary 505 when object 510 approaches the boundary 505. In some embodiments, boundary 505 may be displayed where object 510 contacts boundary 505. In some embodiments, boundary 505 may be displayed when object 510 contacts boundary 505 for a predetermined period of time.

[0117] aspect

[0118] Various aspects are described below. It should be understood that any one or more features described in the following aspects may be combined with any one or more other aspects.

[0119] In some aspects, the technology described herein relates to a method of operating a mixed reality robot training system, which includes a first computing device and a mixed reality device, the method including: obtaining the first computing device, wherein the first computing device includes a processor, a non-transitory computer-readable medium having instructions executable by the processor stored thereon, a first program, a first display, and a user input interface; obtaining the mixed reality device, wherein the mixed reality device includes a camera and a second display; receiving, by the first computing device, a first input on the user input interface to control the operation of the mixed reality robot; determining, by the first computing device, a first instruction for controlling the mixed reality robot based on the first input; transmitting, by the first computing device, the first instruction to the mixed reality device; and displaying, through the mixed reality device, the operation of the mixed reality robot based on the first instruction.

[0120] In some aspects, the technology herein relates to a method that also includes: displaying, by a first computing device, a first image identifying the first computing device on a first display; capturing, by a mixed reality device using a camera, the first image; and connecting the mixed reality device to the first computing device.

[0121] In some aspects, the technology herein relates to a method, further comprising: displaying, by the mixed reality device, a command log of the first computing device in a scene of the mixed reality robot.

[0122] In some aspects, the technology described herein relates to a method, wherein determining a first instruction for controlling a mixed reality robot based on a first input also includes: determining, by a first computing device, a first instruction for controlling the operation of a simulated robot; and determining, by the first computing device, a second instruction for controlling the operation of the mixed reality robot based on the first instruction.

[0123] In some aspects, the technology described herein relates to a method, wherein the robot training system also includes a second computing device, wherein the second computing device includes a processor, a non-transitory computer-readable medium having instructions executable by the processor stored thereon, a second program, and wherein the method further includes: acquiring the second computing device, connecting the first computing device to the second computing device; receiving, by the second computing device, the first instruction from the first computing device; and displaying, by the second computing device, the operation of the simulated robot in the second program based on the first instruction.

[0124] In some aspects, the technology described herein relates to a method, wherein the robot training system also includes an industrial robot, and the method further includes: connecting the industrial robot to a second computing device; and controlling the operation of the industrial robot by the second computing device based on a first instruction from the first computing device.

[0125] In some aspects, the technology herein relates to a method, wherein the method further includes: receiving, by a second computing device, a second input at the second computing device; determining, by the second computing device, a second instruction for controlling the operation of a simulated robot; transmitting, by the second computing device, the second instruction to a first computing device to control the operation of a simulated robot and a mixed reality robot; and controlling, by the first computing device, the operation of the simulated robot and the mixed reality robot based on the second instruction.

[0126] In some aspects, the technology of this document relates to a method, further comprising: capturing, by a mixed reality device, multiple images of an object in a scene of a mixed reality robot, and transmitting the multiple images to a first computing device; determining, by the first computing device, an interaction between the object and the mixed reality robot based on the multiple images; determining, by the first computing device, a third instruction based on the interaction; controlling, by the first computing device, an operation of a simulation robot and an operation of the mixed reality robot based on the third instruction; and displaying, by the mixed reality device, the operation of the mixed reality robot based on the third instruction.

[0127] In some aspects, the technology herein relates to a method, further comprising: determining, by a first computing device, a boundary associated with a mixed reality robot, and displaying, by the mixed reality device, the boundary when an object approaches the boundary; wherein determining an interaction between the object and the mixed reality robot comprises determining an interaction between the object and the boundary.

[0128] In some aspects, the technology herein relates to a method, further comprising: wherein the object comprises a user's hand; and wherein the interaction comprises a gesture.

[0129] In some aspects, the technology described herein relates to a method for remotely operating a mixed reality robot training system, the robot training system comprising a first computing device and a mixed reality device, the first computing device comprising a processor, a non-transitory computer-readable medium having instructions executable by the processor stored thereon, a first program, and a first display, and wherein the mixed reality device comprises a camera and a second display, the first computing device and the mixed reality device being at a first location, the method comprising: receiving input by the first computing device to command a simulated robot of the first program; determining by the first computing device a first instruction for controlling the operation of the simulated robot and a second instruction for controlling the operation of the mixed reality robot based on the input; controlling by the first computing device the operation of the simulated robot based on the first instruction and controlling the operation of the mixed reality robot based on the second instruction; transmitting by the first computing device the second instruction to the mixed reality device to display the operation of the mixed reality robot in a scene of the second display.

[0130] In some aspects, the technology herein relates to a method, further comprising: displaying, by the mixed reality device, a log of instructions of a first program in a scene of a second display.

[0131] In some aspects, the technology described herein relates to a method, wherein the robot training system also includes a second computing device, wherein the second computing device includes a processor, a non-transitory computer-readable medium having instructions executable by the processor stored thereon, a second program, and a third display, and wherein the method also includes: connecting the first computing device to the second computing device; receiving a second input by the second computing device to control a simulated robot and a mixed reality robot; determining by the second computing device a third instruction for controlling the operation of the simulated robot based on the input and determining by the second computing device a fourth instruction for controlling the operation of the mixed reality robot; transmitting the third instruction and the fourth instruction to the first computing device by the second computing device; and controlling the operation of the simulated robot and the mixed reality robot by the first computing device based on the third instruction and the fourth instruction.

[0132] In some aspects, the technology described herein relates to a method, wherein the robot training system also includes an industrial robot, and wherein the method further includes: connecting to the industrial robot via a second computing device; wherein transmitting a third instruction to the first computing device also includes transmitting the third instruction to the industrial robot to control the industrial robot.

[0133] In some aspects, the technology herein relates to a method in which receiving input by a first computing device to command a simulated robot of a first program also includes: determining, by the first computing device, a boundary associated with a mixed reality robot; receiving, by the first computing device, a plurality of images of an object and the mixed reality robot captured by a camera; and determining, by the first computing device, an interaction between the object and the boundary.

[0134] In some aspects, the techniques described herein relate to a method in which the object comprises a user's hand and in which the interaction comprises a gesture.

[0135] In some aspects, the technology herein relates to a system comprising: a first computing device comprising a processor, a non-transitory computer-readable medium having stored thereon instructions executable by the processor, a first program, a touch screen display, the first program comprising a user input interface, a simulated robot, and a mixed reality robot; and a mixed reality device comprising: a camera, a second display; wherein the first computing device determines a first instruction for controlling the simulated robot based on at least one input at the user input interface; wherein the first computing device determines a second instruction for controlling the mixed reality robot based on the at least one input; wherein the first computing device transmits the second instruction to the mixed reality device to display the mixed reality robot on the second display.

[0136] In some aspects, the technology herein relates to a system, wherein the system further includes: an industrial robot; and a second computing device; wherein the second computing device includes: a processor, a non-transitory computer-readable medium having instructions executable by the processor stored thereon, a second program, wherein the second program includes a second user input interface; wherein the second computing device includes being communicatively connected to the first computing device; wherein the first computing device also includes determining a third instruction based on at least one input, and wherein the first computing device transmits the third instruction to the second computing device; wherein the third instruction controls the operation of the industrial robot.

[0137] In some aspects, the technology herein relates to a system, further comprising: a first calibration marker, disposed at a first position in a scene of a user; and a second calibration marker, disposed at a second position in the scene of the user; wherein the first position and the second position comprise being positioned a fixed distance apart, the fixed distance being determined by a first computing device; wherein at least one input comprises initiating calibration of a simulated robot and a mixed reality robot; wherein the mixed reality robot moves from the first position to the second position, and the first computing device calibrates the simulated robot and the mixed reality robot based on the movement of the mixed reality robot from the first position to the second position.

[0138] In some aspects, the technology herein relates to a system, further comprising: wherein the mixed reality robot comprises an object defined by a boundary; and wherein the interaction between the object and the boundary comprises at least one input.

[0139] It should be understood that changes may be made in detail, particularly in the materials of construction employed and the shapes, sizes and arrangements of parts without departing from the scope of the present disclosure. The specification and described embodiments are examples, with the true scope and spirit of the disclosure being indicated by the following claims.

Claims

1. A method of operating a mixed reality robot training system, the mixed reality robot training system comprising a first computing device and a mixed reality device, the method comprising: acquiring the first computing device, wherein the first computing device includes a processor, a non-transitory computer-readable medium having stored thereon instructions executable by the processor, a first program, a first display, and a user input interface; acquiring the mixed reality device, wherein the mixed reality device includes a camera and a second display; receiving, by the first computing device, a first input on the user input interface to control the operation of the mixed reality robot; Determining, by the first computing device, a first instruction for controlling the mixed reality robot based on the first input; Transmitting the first instruction to the mixed reality device by the first computing device; as well as The operation of the mixed reality robot is displayed through the mixed reality device based on the first instruction.

2. The method according to claim 0, further comprising: displaying, by the first computing device on the first display, a first image identifying the first computing device; capturing, by the mixed reality device, the first image using the camera; as well as Connecting the mixed reality device to the first computing device.

3. The method according to claim 1, further comprising: The command log of the first computing device is displayed by the mixed reality device in the scene of the mixed reality robot.

4. The method according to claim 0, wherein determining a first instruction for controlling the mixed reality robot based on the first input further comprises: determining, by the first computing device, first instructions for controlling the operation of the simulated robot; A second instruction for controlling the operation of the mixed reality robot is determined by the first computing device based on the first instruction.

5. The method of claim 4, wherein the robotic training system further comprises a second computing device, wherein the second computing device comprises a processor, a non-transitory computer readable medium having stored thereon instructions executable by the processor, a second program, and wherein the method further comprises: acquiring the second computing device, connecting the first computing device to the second computing device; receiving, by the second computing device, the first instruction from the first computing device; as well as The operation of the simulated robot is displayed in the second program by the second computing device based on the first instruction.

6. The method of claim 5, wherein the robot training system further comprises an industrial robot, and the method further comprises: connecting the industrial robot to the second computing device; The operation of the industrial robot is controlled by the second computing device based on the first instruction from the first computing device.

7. The method according to claim 5, wherein the method further comprises: receiving, by the second computing device at the second computing device, a second input; determining, by the second computing device, second instructions for controlling the operation of the simulated robot; transmitting, by the second computing device, the second instruction to the first computing device to control the operations of the simulation robot and the mixed reality robot; as well as The operations of the simulation robot and the mixed reality robot are controlled by the first computing device based on the second instructions.

8. The method according to claim 1, further comprising: capturing, by the mixed reality device, a plurality of images of objects in a scene of the mixed reality robot, and transmitting the plurality of images to the first computing device; determining, by the first computing device, an interaction between the object and the mixed reality robot based on the plurality of images; determining, by the first computing device, a third instruction based on the interaction; controlling, by the first computing device, the operation of the simulation robot and the operation of the mixed reality robot based on the third instruction; The operation of the mixed reality robot is displayed by the mixed reality device based on the third instruction.

9. The method according to claim 8, further comprising: determining, by the first computing device, a boundary associated with the mixed reality robot, When the object approaches the boundary, the mixed reality device displays the boundary; Wherein determining the interaction between the object and the mixed reality robot includes determining the interaction of the object with the boundary.

10. The method according to claim 8, further comprising: wherein the object comprises a user's hand; as well as The interaction includes a gesture.

11. A method for remotely operating a mixed reality robotic training system, the robotic training system comprising a first computing device and a mixed reality device, the first computing device comprising a processor, a non-transitory computer-readable medium having instructions executable by the processor stored thereon, a first program, and a first display, and wherein the mixed reality device comprises a camera and a second display, the first computing device and the mixed reality device being at a first location, the method comprising: receiving, by the first computing device, input to command the simulated robot of the first program; determining, by the first computing device based on the input, a first instruction for controlling the operation of the simulated robot and a second instruction for controlling the operation of the mixed reality robot; controlling, by the first computing device, the operation of the simulated robot based on the first instruction and controlling the operation of the mixed reality robot based on the second instruction; The second instruction is transmitted by the first computing device to the mixed reality device to display the operation of the mixed reality robot in a scene of a second display.

12. The method according to claim 11, further comprising: A log of the instructions of the first program is displayed by the mixed reality device in a scene of the second display.

13. The method of claim 11, wherein the robotic training system further comprises a second computing device, wherein the second computing device comprises a processor, a non-transitory computer readable medium having stored thereon instructions executable by the processor, a second program, and a third display, and wherein the method further comprises: connecting the first computing device to the second computing device; receiving, by the second computing device, a second input to control the simulated robot and the mixed reality robot; determining, by the second computing device, third instructions for controlling the operation of the simulated robot based on the input, and determining, by the second computing device, fourth instructions for controlling the operation of the mixed reality robot; transmitting the third instruction and the fourth instruction to the first computing device by the second computing device; The operations of the simulation robot and the mixed reality robot are controlled by the first computing device based on the third instruction and the fourth instruction.

14. The method of claim 13, wherein the robot training system further comprises an industrial robot, and wherein the method further comprises: connecting to the industrial robot via the second computing device; Wherein transmitting the third instruction to the first computing device also includes transmitting the third instruction to the industrial robot to control the industrial robot.

15. The method of claim 11, wherein receiving, by the first computing device, the input to command the simulated robot of the first program further comprises: determining, by the first computing device, a boundary associated with the mixed reality robot; receiving, by the first computing device, a plurality of images of an object and the mixed reality robot captured by the camera; An interaction between the object and the boundary is determined by the first computing device.

16. The method of claim 15, wherein the object comprises a user's hand, and The interaction includes a gesture.

17. A system comprising: A first computing device, the first computing device comprising: processor, a non-transitory computer-readable medium having stored thereon instructions executable by the processor, a first program, the first program comprising a user input interface, a simulation robot, and a mixed reality robot; a touch screen display; and A mixed reality device, the mixed reality device comprising: camera, Second display; wherein the first computing device determines first instructions for controlling the simulated robot based on at least one input at the user input interface; wherein the first computing device determines second instructions for controlling the mixed reality robot based on the at least one input; The first computing device transmits the second instruction to the mixed reality device to display the mixed reality robot on the second display.

18. The system of claim 17, wherein the system further comprises: Industrial robots; as well as a second computing device; The second computing device comprises: processor, a non-transitory computer-readable medium having stored thereon instructions executable by the processor, Second procedure, wherein the second program includes a second user input interface; wherein the second computing device comprises being communicatively connected to the first computing device; wherein the first computing device further comprises determining a third instruction based on the at least one input, and wherein the first computing device transmits the third instruction to the second computing device; The third instruction controls the operation of the industrial robot.

19. The system of claim 17, further comprising: a first calibration mark disposed at a first position in the user's scene; as well as a second calibration mark disposed at a second location in the user's scene; wherein the first position and the second position include being located a fixed distance apart, the fixed distance being determined by the first computing device; wherein the at least one input comprises initiating calibration of the simulation robot and the mixed reality robot; The mixed reality robot moves from the first position to the second position, and the first computing device calibrates the simulation robot and the mixed reality robot based on the movement of the mixed reality robot from the first position to the second position.

20. The system of claim 19, further comprising: wherein the mixed reality robot comprises a boundary defined by; and Wherein the interaction between the object and the boundary comprises the at least one input.

Images