Method and system for programming robot

Abstract

To provide a method and a system for programming a robot.SOLUTION: A method for programming a robot includes: identifying a robot device and a calibration fixture in the vicinity of the robot device; normalizing the calibration fixture as a base of the robot device so as to determine a first pose of the robot device; receiving a 3D image of an environment in which the 3D image includes the calibration fixture; determining a second pose of the calibration fixture with respect to a sensor; determining a third pose of the robot device with respect to the senor on the basis of the first pose and the second pose; receiving a plurality of trajectory points; determining a plurality of virtual trajectory points corresponding to the plurality of trajectory points on the basis of the 3D image and the third pose; providing the display of the plurality of virtual trajectory points; and providing an interface for operating the virtual trajectory points.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a system and method for programming a robot to perform a sequence of trajectory points, and more particularly to the manner in which a robot is visually programmed in an operating environment. [Background technology]

[0002] Robots are often used to manipulate the surfaces of objects. The existing practice for programming robots involves teaching them a sequence of trajectory points. During programming, the robot is taught how to perform a task by being guided through various trajectory points along a desired motion path. Alternatively, if a three-dimensional (3D) CAD model of the object exists, a person with a robotics background can teach the trajectory points to a robot simulation system, so-called offline programming. Summary of the Invention [Means for solving the problem]

[0003] The present invention relates to a computer-implemented method that includes identifying a robotic device and a calibration fixture in a vicinity of the robotic device within an environment; referencing the calibration fixture to a predetermined portion of the robotic device to determine a first pose of the robotic device relative to the calibration fixture; receiving a 3D image of the environment from a sensor, the 3D image including the calibration fixture; determining a second pose of the calibration fixture relative to the sensor based on the 3D image; determining a third pose of the robotic device relative to the sensor based on the first pose and the second pose; receiving a plurality of trajectory points from a display interface or a device interface; and determining a plurality of virtual trajectory points corresponding to the plurality of trajectory points based on the 3D image and the third pose.

[0004] The present invention further relates to a system for programming a robotic device, comprising: a sensor; and a computing system communicatively coupled to the sensor and configured to perform a method according to any one of the embodiments disclosed herein.

[0005] The present invention further relates to a non-transitory computer-readable medium having stored therein instructions that, when executed by a computing system, cause the computing system to perform a method according to any one of the embodiments disclosed herein.

[0006] The illustrative examples described herein are not meant to be limiting, as it will be readily understood that certain aspects of the disclosed methods and systems can be arranged and combined in a wide variety of different configurations, all of which are contemplated herein.

[0007] Furthermore, the particular arrangements shown in the figures should not be viewed as limiting. It should be understood that other embodiments may include more or fewer elements than each element shown in a given figure. Furthermore, some of the illustrated elements may be combined or omitted. Nevertheless, example embodiments may include the elements illustrated in the figures.

[0008] Exemplary systems and methods are provided for acquiring a 3D image of an environment using a sensor that may be coupled to a display interface, where the 3D image includes a robot, an object, and a calibration fixture. The 3D image is used to create, manipulate, and / or manage trajectory points for the robot. The image of the environment refers to a 2D image on the display interface. The display interface coupled with the sensor is used to visualize and manage trajectory points for the robot within an actual image of the robot's environment.

[0009] According to one embodiment, a method is provided that includes identifying a robotic device and a calibration fixture in a vicinity of the robotic device within an environment. The robotic device may also be referred to as a robot. For example, if the robotic device is an industrial robot, the robotic device may have an end effector including a tool for performing a process, such as a welding tool or a painting tool. Identifying the robotic device refers to identifying the presence of the robotic device. Identifying may further include identifying additional information referring to the robotic device, such as the robotic device's identification number, make, model, location, orientation, and / or status. Identifying may include providing the robotic device. Alternatively, or in addition, identifying may refer to identifying the location of the robotic device in sensor data, such as image data, magnetic field data, electric field data, signals transmitted via a cable, signals transmitted wirelessly, or any other data that can be used to identify the robotic device. The calibration fixture may be visually distinctive, for example, marked with a color marking, a bar code, a two-dimensional code, or a unique coating.

[0010] The method further includes referencing the calibration fixture to a predetermined portion (e.g., a base) of the robotic device to determine a first pose of the robotic device relative to the calibration fixture. The referencing can be based on a referencing signal. For example, the referencing can include using a referencing rail, a distancing device (e.g., a laser-based distancing device), an image sensor, or another signal that can be used to refer the calibration fixture to the base of the robotic device.

[0011] The method further includes receiving a 3D image of the environment from a sensor, where the 3D image includes a calibration fixture. The 3D image can represent the robot, the object, and the calibration fixture. The 3D image is used to create, manipulate, and / or manage the trajectory points. The 3D image can include a series of infrared images, a series of structured light images, a series of still images, a series of dynamic range images, a series of shot noise images, a series of red noise images, a series of dark noise images, and / or a video stream. The sensor capturing the 3D image can be a depth sensor and / or a 3D sensor. The sensor can be coupled to a display interface for presenting the 3D image. The calibration fixture can be a three-dimensional (3D) object used to calculate and detect the pose and, generally, the movement (translation and rotation) of the robot in its environment using the sensor.

[0012] Because the method does not rely on using a completely virtual 3D model (e.g., a computer-aided design, or "CAD," model), an inaccurate 3D model can lead to a situation where a trajectory is commanded that causes the robot to unexpectedly collide with an object or the environment. Thus, the method is safer.

[0013] The method further includes determining a second pose of the calibration fixture relative to the sensor based on the 3D image. In that regard, the determining can include identifying one or more fiducial markers in estimating the position and / or orientation of the calibration fixture and / or the position and / or orientation of the robotic device. A fiducial marker, which may also be referred to as a fiducial, is an object placed in the field of view of the sensor that appears in the generated image for use as a reference or point of measure. For example, the fiducial marker can be a color dot, a barcode, or a recognizable object. For example, the fiducial marker can be painted on, attached to, or placed on the calibration fixture and / or the robotic device. Additionally or alternatively, the determining can be based on 3D shape recognition of the calibration marker. In that regard, the calibration marker can be a 3D structure or object attached to or forming part of the calibration fixture. Alternatively, the calibration fixture can be the calibration marker as a whole. Additionally, the calibration marker may be a 3D structure attached to or forming part of the robotic device. Alternatively, the robotic device may be the calibration marker as a whole. In that regard, 3D shape recognition may include pattern recognition, pattern matching, machine learning, or any other suitable shape recognition technique.

[0014] Since the method includes determining a second pose of the calibration fixture relative to the sensor based on the 3D image, the second pose can be determined quickly without requiring additional equipment. When the second pose is determined based on 3D shape recognition, the accuracy of the determination is even further improved.

[0015] Further, the method includes determining a third pose of the robotic device relative to the sensor based on the first pose. Alternatively, determining the third pose can be based on the first pose and the second pose.

[0016] The method further includes receiving a plurality of trajectory points. The trajectory points, which may also be called waypoints, define a path the robot is to follow while processing the object. The trajectory points include three-dimensional position information and three-dimensional orientation information. These trajectory points are stored as instructions in a memory in or coupled to a robot control unit or computing system of the robot. During operation of the robot, the program instructions are executed, thereby causing the robot to move as desired. The trajectory points may be provided in the form of user entry via a display interface or in the form of a robot script via a device interface.

[0017] The method further includes determining, based on the 3D image and the third pose, a plurality of virtual trajectory points corresponding to the plurality of trajectory points, which may be pre-programmed by a user as three-dimensional points relative to a predetermined portion of the robot (e.g., a base) without knowledge of the robot's current environment.

[0018] Because the method involves determining a plurality of virtual trajectory points corresponding to a plurality of trajectory points based on the 3D image and the third pose, the method does not require the user or operator programming the robot to have knowledge of computer science. Thus, the operator programming the robot only needs to have sufficient implicit knowledge of the movements and processes to be performed by the robot. Therefore, the method is less costly, does not require a 3D CAD model of the object, and is intuitive to use.

[0019] Further, optionally, the method includes receiving a 2D image of the environment. In that regard, the 2D image of the environment may be received from a camera included in or coupled to the display device. Alternatively, the 2D image of the environment may be received from a camera included in or coupled to a sensor.

[0020] Additionally, optionally, the method includes providing a display of a 2D image of the environment overlaid with the plurality of virtual trajectory points.

[0021] Because the method includes providing a display of a 2D image of the environment overlaid with multiple virtual trajectory points, if the 3D model of the environment is inaccurate or the environment changes, trajectories that may unexpectedly cause the robot to collide with objects in the environment or produce other unexpected results can be easily determined. Therefore, the method is safer and more predictable. Furthermore, because the method includes providing a display of a 2D image of the environment overlaid with multiple virtual trajectory points, an operator can determine whether they have missed any part of the process, for example, any part of the surface in a painting operation, without having to run the program for testing. Therefore, a single iteration may be sufficient to achieve programming of satisfactory quality.

[0022] Additionally, optionally, the method includes providing an interface for manipulating the virtual trajectory points through the display interface.

[0023] Because the method involves providing an interface for manipulating virtual trajectories through a display interface, the method does not require physical movement of the robot. Therefore, the method is less time-consuming, less tedious, and less prone to error. Furthermore, an acceptable program can be achieved in only one or a few iterations.

[0024] Thus, this method provides an intuitive visualization of the robot trajectory points and simplifies robot programming.

[0025] The method may also include providing a display of a 3D image that may or may not be superimposed on a 2D image of the environment.

[0026] The pose (position and orientation) of the robot relative to the sensor may be determined to determine locations for projecting the robot's trajectory points into an image on the display interface. The robot's pose may be determined by processing a 3D image of the calibration fixture. Once the robot's pose relative to the sensor is determined, the pose of one or more trajectory points may be projected onto the display interface or optionally into an augmented reality presentation on the display interface, allowing a user to visualize the robot's trajectory within its given environment.

[0027] A calibration fixture is an object with 3D features that are used to determine the pose of the robot relative to the sensor. The calibration fixture may be placed in the robot's environment with a relative pose determined between the robot and the calibration fixture, so that when the sensor acquires a 3D image of the environment that includes a representation of the calibration fixture, the pose of the robot relative to the sensor can be determined.

[0028] In another embodiment, the method may further include receiving, from the device interface, input data indicating one or more adjustments to one or more properties of one or more of the virtual trajectory points. In this regard, the properties may be one or more selected from the group consisting of trajectory position, trajectory heading, end effector state, trajectory velocity, electronic signal input, and electronic signal output. The device interface may be a display interface of a display device or an input interface of a mobile, laptop, notebook, or desktop computer. The input data indicates the adjustments, where each adjustment relates to one or more properties of one or more of the virtual trajectory points. For example, the input data may include data indicating an adjustment to one property of one trajectory point, such as the trajectory heading of one trajectory point. As another example, the input data may include data indicating an adjustment to the trajectory heading of one trajectory point and data indicating an adjustment to the trajectory velocity of another trajectory point. Furthermore, the method according to this embodiment may include receiving, from the device interface, input data for creating, duplicating, or deleting one or more of the virtual trajectory points. Further, the method may include determining, based on input data received on the display interface, one or more adjusted properties of one or more of the trajectory points corresponding to one or more of the plurality of virtual trajectory points.

[0029] In another embodiment, the method may further include receiving input data from an interface of the device including the sensor, the input data providing instructions for the robotic device to move according to one or more of the trajectory points or virtual trajectory points and their respective properties, and transmitting the instructions to the robotic device.

[0030] The received trajectory points may include references to or be based on one of various coordinates. For example, trajectory points that may be received from user input via a display interface may be based on virtual coordinates and thus may be referred to as virtual trajectory points. In another example, trajectory points received as a robot script may be based on physical or robot coordinates and thus may be referred to as physical trajectory points. Because virtual coordinates may be separate from the physical or robot coordinates used by the robotic device, if the received trajectory points are virtual trajectory points, they may require conversion or translation into physical trajectory points, which are then sent to the robotic device for execution. Accordingly, sending the above-described instructions to the robotic device may include converting the virtual trajectory points into physical trajectory points and sending the physical trajectory points to the robotic device.

[0031] In another embodiment, the method may further include providing a display interface for manipulating the virtual trajectory points and receiving, from the display interface, input data indicating creation of a trajectory pattern including a plurality of virtual trajectory points arranged in a specified pattern and / or adjustments to the trajectory pattern. Furthermore, the method may include converting the two-dimensional trajectory pattern into a three-dimensional trajectory pattern on the display interface based on the created or adjusted trajectory pattern. Furthermore, the method may include receiving, from the display interface, input data for translating and / or rotating the three-dimensional trajectory pattern on the display interface. Furthermore, the method may include receiving, from the display interface, input data for projecting the three-dimensional trajectory pattern onto a portion of the 3D image. In that regard, in one embodiment, the 3D image may be superimposed on the 2D image on the display interface. Alternatively, the 3D image may be displayed without the 2D image.

[0032] According to another embodiment, there is provided a system for programming a robotic device, the system including a sensor and a computing system communicatively coupled to the sensor and configured to perform a method according to an embodiment.

[0033] Additionally, the system may be coupled or connected to a memory of a robotic device to store the trajectory points in the memory, or alternatively, the system may be coupled or connected to a control of the robotic device such that the system can operate the robotic device according to the trajectory points.

[0034] According to another embodiment, a non-transitory computer-readable medium is provided having stored thereon instructions that, when executed by a computing system, cause the computing system to perform functions according to an embodiment of the method described herein.

[0035] According to another embodiment, a method is provided that includes identifying a robotic device and a calibration fixture in a vicinity of the robotic device within an environment, referencing the calibration fixture to a predetermined portion (e.g., a base) of the robotic device to determine a first pose of the robotic device relative to the calibration fixture, receiving a 3D image of the environment from a sensor, the 3D image including the calibration fixture, determining a second pose of the calibration fixture relative to the sensor based on the 3D image, determining a third pose of the robotic device relative to the sensor based on the first pose and the second pose, providing an interface for creating virtual trajectory points, and determining trajectory points corresponding to the virtual trajectory points based on the 3D image and the third pose.

[0036] In embodiments including the method, system, and / or non-transitory computer-readable medium according to any one of the embodiments disclosed herein, the plurality of trajectory points is no more than one trajectory point, and the plurality of virtual trajectory points is no more than one virtual trajectory point.

[0037] For a better understanding of the embodiments, together with other and further features and advantages thereof, reference is made to the following description taken in conjunction with the accompanying drawings, the scope of which will be pointed out in the appended claims. [Brief explanation of the drawings]

[0038] [Figure 1] 1 is a flow chart illustrating a method according to one embodiment. [Figure 2] FIG. 1 illustrates a system according to one embodiment. [Figure 3] FIG. 1 illustrates a 2D image of a display interface. [Figure 4] 10A-10C illustrate an image of a trajectory pattern being projected onto a 3D image of an object using a method according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0039] Referring to the figures, FIG. 1 illustrates a flow chart showing a method 100 that can consider using 3D images acquired by a sensor to determine a third pose (the relative pose of the robot and the sensor) and / or to create or manage robot trajectory points. Method 100 can be performed using a display device including a camera, such as a tablet device, a smartphone, a head-mounted display (HMD), or a mobile, laptop, notebook, or desktop computer. Sensors that are separate from the display device may be used; however, the sensors may be coupled to the display device through a physical fixture or adhesive medium. The sensors may also, or alternatively, be communicatively coupled to the display device through a cable (wired) or wireless connection.

[0040] As indicated by block 102 of FIG. 1 , method 100 includes receiving a 3D image of an environment from a sensor. The sensor may be a depth sensor and / or a 3D sensor. The 3D image may be a series of infrared images, a series of structured light images, a series of still images, and / or a composite of a video stream. The 3D image may be a single still infrared image and / or structured light image. The environment may include at least one robotic device and a calibration fixture.

[0041] Referring to FIG. 1 , method 100 further includes determining the pose of the robotic device relative to the sensor, as illustrated by block 104. The 3D image of the environment may be processed to determine where the robotic device is located in the environment relative to the sensor. Determining the pose of the robotic device includes determining the position (X, Y, Z) and orientation (roll, pitch, yaw) of the robotic device relative to the sensor. The pose of the robotic device relative to the sensor may be determined by processing a 3D image that includes a calibration fixture. By recognizing three-dimensional features on the calibration fixture in the 3D image, the translation and rotation of the robotic device from the sensor can be determined. The system may have prior knowledge of the relative pose of the robotic device and the calibration fixture, thereby enabling the pose of the robotic device relative to the sensor to be determined.

[0042] Method 100 optionally further includes determining the pose of the trajectory points for the robotic device relative to the sensor, as shown in block 106, if an augmented reality presentation is desired. When the pose of the robotic device relative to the sensor is determined, the robot may be used as a reference point to determine where to virtually overlay the trajectory points as part of the optional augmented reality presentation. Because the sensor is coupled to the display device at a known relative displacement, the relative pose of the robot and the visual camera can be derived from the relative pose of the visual camera and the sensor through pose compensation based on the known relative displacement between the sensor and the visual camera on the display device. In that regard, the pose of the trajectory points relative to the visual camera of the display device can be determined based on the relative pose of the robotic device and the sensor. These locations, e.g., the relative displacement between the sensor and the visual camera on the display device, may be used to virtually overlay the trajectory points within a 2D image captured by the visual camera of the display device.

[0043] In a step not shown in FIG. 1 for simplicity, method 100 optionally includes receiving a 2D image of the environment, in which the 2D image is received from a sensor. In an alternative embodiment, the 2D image may be received by an additional camera, preferably located close to the sensor and whose orientation is the same as that of the sensor.

[0044] As indicated at block 108, method 100 includes providing a display of the 3D image received at block 102. Optionally, the 3D image may be overlaid on a 2D image of the environment.

[0045] In a step not shown in FIG. 1 for simplicity, method 100 includes receiving a plurality of trajectory points. Method 100 further includes determining a plurality of virtual trajectory points corresponding to the plurality of trajectory points based on the 3D image and the position of the robotic device relative to the sensor determined in block 106. At that point, each virtual trajectory point is positioned in the 3D image in a manner that superimposes the exact position of the corresponding trajectory point in the corresponding 3D image. That is, for example, if a trajectory point is positioned at a lower joint of the robotic device, the corresponding virtual trajectory point is positioned on a 3D representation of the lower joint of the robotic device in the 3D image. Optionally, each virtual trajectory point is positioned in the 2D image in a manner that superimposes the exact position of the corresponding trajectory point in the corresponding 3D image.

[0046] As indicated at block 110, method 100 further includes providing a display of a 3D image overlaid with the determined plurality of virtual trajectory points. Optionally, method 100 further includes providing a display of a 2D image of the environment overlaid with the determined plurality of virtual trajectory points.

[0047] As indicated by block 112 of FIG. 1 , method 100 may include overlaying the virtual trajectory points on a 3D image presented on the display interface. Optionally, block 112 of method 100 may include overlaying the virtual trajectory points on a 2D image of the environment presented on the display interface. The image of the environment may be a 2D image acquired by a visual camera on the display device. The image of the environment may include the entire robotic device, a portion of the robotic device, or none of the robotic device. Additionally, the image of the environment may include the entire calibration fixture, a portion of the calibration fixture, or none of the calibration fixture. Additionally, the image of the environment may be a single still 2D image, a series of still 2D images, and / or a video stream.

[0048] As indicated by block 112 of FIG. 1 , method 100 may further include creating and / or managing virtual trajectory points through the display interface. In other words, a trajectory or trajectory pattern to be executed or performed by the robotic device may be generated. A user interface associated with the display interface may receive input data indicating one or more changes to the virtual trajectory of the robotic device. For example, the display interface may include a touch-based interface adjusted to a 3D image and / or optionally to a 2D image. Virtual trajectory points may be created, deleted, and / or duplicated through the input data on the display interface. Properties of each virtual trajectory point, such as position, orientation, end effector state, trajectory velocity, electronic signal input, and electronic signal output, may be adjusted through the input data on the display interface. The trajectory points of the robotic device correspond to the virtual trajectory points on the display interface. The robotic device may be commanded to execute a trajectory, where the trajectory includes at least some sequence of multiple trajectory points. The robotic device may be commanded to execute properties of each trajectory point along the trajectory. For example, an end effector may be commanded to be activated mid-trajectory as commanded through input data on a display interface.

[0049] The functionality described in connection with the flowcharts described herein may be implemented as hardware modules of specific function and / or configured general function, as portions of program code for implementing the specific logical functions, decisions, and / or steps described in connection with the flowcharts shown in Figure 1. When used, the program code may be stored on any type of computer-readable medium, such as a storage device including, for example, a disk or hard drive.

[0050] The functions in the flowchart shown in FIG. 1 may be performed out of order from that shown or discussed, including parallel execution or even reverse order of separately described functions, so long as the overall functionality of the described method is maintained. The functions in the flowchart shown in FIG. 1 may be performed selectively. For example, one embodiment may perform calibration of a robot relative sensor, another embodiment may perform visualization of virtual trajectory points for the robot through an augmented reality presentation in addition to calibration, another embodiment may perform visualization of virtual trajectory points for the robot using 3D images in addition to calibration, and another embodiment may perform creation or management of robot trajectory points using 3D images acquired by a sensor in addition to calibration and visualization in augmented reality and / or using 3D images. Other combinations may also be possible.

[0051] FIG. 2 illustrates a system consisting of a robotic device 202, a calibration fixture 228, a display device 222, and a sensor 224.

[0052] The robotic device 202 includes a predetermined portion, e.g., a base 204, which may be a stationary base or a mobile base. The robotic device may be controlled to operate and move along a trajectory 220 including trajectory points 210-218. Additionally, the robotic device may include an end effector 226, which may take the form of a gripper, such as a finger gripper or a different type of gripper, such as a suction gripper. The end effector may take the form of a tool, such as a drill, a brush, or a paint gun. The end effector may include sensors, such as a force sensor, a proximity sensor, or a camera. Other examples may also be possible.

[0053] Display device 222 may be a device that includes an interface and, optionally, a visual camera that captures 2D images of the environment. For example, the display device may be part of a tablet computer, a handheld smartphone, or a mobile, laptop, notebook, or desktop computer.

[0054] The sensor 224 may be a depth sensor and / or a 3D sensor that acquires a 3D image of the environment. The 3D image may be a composite of a series of infrared images, a series of structured light images, a series of still images, and / or a video stream. The 3D image may be a single still infrared image and / or structured light image. The sensor 224 may be physically fixed to the display device 222 through a fixture or adhesive medium. The fixture for coupling the sensor to the display device may have a removable or non-removable mechanism. The sensor 224 may be connected to the display device 222 through a cable (wired) or wireless connection so that the display device 222 receives the 3D image from the sensor 224.

[0055] Calibration fixture 228 is an object with three-dimensional features that is placed in the environment of the robotic device. According to the exemplary embodiment of FIG. 2 , calibration fixture 228 can include base 206 and asymmetric geometric feature 208 attached to the base. Base 206 can be used to place or adhere calibration fixture 228 on a surface in the environment, such as a table, a wall, or an object. In the exemplary embodiment, asymmetric geometric feature 208 can include two spherical objects having different sizes and / or different positions. However, the exemplary embodiment described herein is not meant to be limiting. The asymmetric geometric feature can be realized by configurations not shown in the exemplary embodiment. For example, the asymmetric geometric feature on the calibration fixture can consist of additional shapes or features that can be hemispherical, cylindrical, conical, rectangular, triangular, trapezoidal, elliptical, sinusoidal, concave, convex, or combinations or variations thereof.

[0056] By recognizing the asymmetric geometric feature 208 on the calibration fixture 228 in the 3D image acquired by the sensor 224, the pose of the calibration fixture 228 relative to the sensor 224 may be determined. Because the pose of the robotic device 202 relative to the calibration fixture 228 may be known, the pose of the robotic device 202 relative to the sensor 224 may thereby be determined by recognizing the calibration fixture 228 in the 3D image.

[0057] In an alternative embodiment, the calibration fixture can have a base and a 3D symmetric geometric feature attached to the base. Examples of symmetric geometric features can include the shapes or features described above that are applicable to asymmetric geometric features. In an alternative embodiment, the calibration fixture can have a base and a 3D non-geometric or irregularly shaped feature attached to the base. Such non-geometric features can be symmetric or asymmetric. In an alternative embodiment, the total number of 3D features can be one, two, or more.

[0058] 3 illustrates a 2D image on display interface 330, where virtual trajectory points 310-328 are positioned over object 332. Display interface 330 may be an interface on display device 222. The 2D image on display interface 330 may be a single still image or a video stream continuously received from a visual camera on display device 222. As the orientation of display device 222 changes, the 2D image on display interface 330 may be updated to display a portion of the environment from a corresponding perspective.

[0059] Referring to FIG. 3 , virtual trajectory points 310-328 may be projected onto the display interface 330, or optionally in an augmented reality presentation on the display interface 330. The virtual trajectory points correspond to trajectory points of the robotic device 302. The pose of the trajectory points 310-328 for the robotic device 302 relative to the sensors is determined based on block 106 of FIG. 1 . The virtual trajectory points are overlaid on the display interface 330 based on block 112 of FIG. 1 . Depending on the orientation of the display device 222, only a subset of the trajectory points projected by the virtual trajectory points 310-328 may be projected onto the display interface 330. The robotic device may be commanded to move through the sequence of trajectory points as projected by the virtual trajectory points 310-328 shown on the display interface 330.

[0060] Referring to block 108 of FIG. 1 , a 3D image acquired by the sensor 224 may be overlaid on the display interface 330, or optionally on a 2D image of the environment on the display interface 330. While overlaid on the 2D image of the environment on the display interface 330, the 3D image may be visible or invisible. Block 112 of FIG. 1 includes creating and / or managing virtual trajectory points on the display interface. The creation and / or management of virtual trajectory points may be performed on the 3D image, which may or may not be overlaid on the 2D image of the environment on the display interface 330. For example, the position of an input on the display interface 330 may be projected onto the 3D image, and then a corresponding virtual trajectory point may be created where a ray cast from the input position on the display interface 330 intersects with the 3D image. In this manner, the virtual trajectory points, and thus the trajectory points, may be created and / or managed accurately with respect to the environment and the robotic device 302. As shown in FIG. 3, the virtual trajectory points 312-328 are placed on a 3D image of the object 332, which may or may not be superimposed on a 2D image of the object 332 on the device interface 330, where the virtual trajectory points 312-328 may appear to be positioned on the surface of the object 332 on the device interface 330.

[0061] 3, calibration fixture 334 may be used to determine the pose of robotic device 302 relative to sensor 224. Calibration fixture 334 may not need to be constantly captured by sensor 224 and visual camera of display device 222 for proper overlay of virtual trajectory points on display interface 330. Calibration fixture 334 may consist of asymmetric geometric feature 308 and base 306.

[0062] FIG. 4 illustrates an image of a trajectory pattern being projected onto a 3D image of an object 412. The display interface 430 may include a subset of the user interface 402 that may enable a user to define the trajectory pattern 404. For example, the user interface 402 may enable a user to define properties of the trajectory pattern. The properties may include the dimensions (vertical and horizontal lengths) of the trajectory pattern 404, the number and / or density of trajectory points along the trajectory pattern 404, and the frequency of stepovers. The user interface 402 may provide a preview of the trajectory pattern 404 defined by the user. The trajectory pattern may not be limited to the raster pattern illustrated in the exemplary embodiment; the trajectory pattern may be other patterns, including zigzag, spiral, contour, and fishtail.

[0063] The trajectory pattern 404 may be subjected to a process 406 that converts the two-dimensional trajectory pattern 404 on the user interface 402 into a three-dimensional trajectory pattern 408. The three-dimensional trajectory pattern 408 may be translated and rotated along its three-dimensional coordinate frame when input data is received on the display interface 430. The three-dimensional trajectory pattern 408 may be projected onto a 3D image of an object 412 to result in a projected trajectory pattern 414 that matches the surface of the object 412. Each virtual trajectory point on the three-dimensional pattern 408 may be projected onto the surface of the object 412 via a respective projection path 410, where the projection paths 410 include origins at their respective locations on the three-dimensional trajectory pattern 408 and projection orientations that may be orthogonal to the plane of the three-dimensional trajectory pattern 408. The projection of the three-dimensional trajectory pattern 408 may not be limited to the object 412 shown in the exemplary embodiment. For example, the three-dimensional trajectory pattern 408 may be projected onto other objects not shown in the illustrative embodiment, such as a turbine blade, an airfoil, a metal sheet, or other manufactured part. The three-dimensional trajectory pattern 408 may also be projected onto an environment, where the environment does not include the object 412 or robotic device, such as a table, floor, wall, fixture, or conveyor system.

[0064] The present disclosure will not be limited with respect to the particular examples described in this application, which are intended as illustrations of its various aspects. As will be apparent to those skilled in the art, many modifications and variations can be made without departing from its spirit and scope. Functionally equivalent methods and apparatuses within the scope of the present disclosure, in addition to those set forth herein, will be apparent to those skilled in the art from the foregoing description. Such modifications and variations are intended to fall within the scope of the appended claims.

[0065] The above detailed description describes various features and functions of the disclosed systems and methods with reference to the accompanying figures. In the figures, like symbols generally identify like parts unless context dictates otherwise. The illustrative embodiments described in the specification and figures are not meant to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that aspects of the present disclosure, as broadly described herein and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a variety of different configurations, all of which are expressly contemplated herein.

[0066] Blocks representing information processing, such as the method blocks described above, may correspond to circuitry that can be configured to perform specific logical functions of the methods or techniques described herein. Alternatively, or in addition, blocks representing information processing may correspond to modules, segments, or portions of program code (including associated data). The program code may include one or more instructions executable by a processor for implementing specific logical functions or actions of the method or technique. The program code and / or associated data may be stored on any type of computer-readable medium, such as a storage device, including a disk or hard drive, or other storage medium.

[0067] Blocks representing one or more information transfers may correspond to information transfers between software and / or hardware modules in the same physical device, however, other information transfers may be between software and / or hardware modules in different physical devices.

[0068] Computer-readable media may also include non-transitory computer-readable media, such as computer-readable media that store data for short periods of time, such as register memory, processor cache, and random-access memory (RAM). Computer-readable media may also include non-transitory computer-readable media that store program code and / or data for longer periods of time, such as secondary or permanent long-term storage, such as read-only memory (ROM), optical or magnetic disks, and compact-disc read-only memory (CD-ROM). Computer-readable media may be considered computer-readable storage media or tangible storage devices.

[0069] The particular arrangement shown in the figures should not be seen as limiting. It should be understood that other embodiments may include more or fewer elements than each element shown in a given figure. Some of the illustrated elements may be combined or omitted.

[0070] The articles "a," "an," and "the," when used in reference to features or elements, should be understood to include a reference to one or more of the features or elements. The term "and / or" includes any and all combinations of one or more of the associated features or elements. The terms "comprises," "includes," "involves," and "having" are intended to be open-ended, meaning that additional features or elements may be present other than those listed. Identifiers such as "first," "second," and "third" are used merely as labels and are not intended to impose quantitative requirements on those objects, nor should they be interpreted as imposing any relative position or time sequence among them. The term "coupled" can refer to being physically coupled, electrically coupled, and / or communicatively coupled. When applied to two objects, the term "coupled" can refer to the two objects being coupled directly or indirectly through a third object.

[0071] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those of ordinary skill in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope being indicated by the following claims. [Explanation of symbols]

[0072] 100 ways 102 blocks 104 blocks 106 blocks 108 blocks 110 blocks 112 blocks 202 Robot Devices 204 Base 206 Base 208 Asymmetric Geometric Features 210~218 Orbital points 220 orbit 222 Display Devices 224 Sensors 226 End Effector 228 Fixtures 302 Robot Devices 306 Base 308 Asymmetric Geometric Features 310~328 Virtual orbit points 330 Display Interface 332 objects 334 Calibration Fixture 402 User Interface 404 Orbital Pattern 406 Process 408 Orbital Pattern 410 Projection Path 412 objects 414 Orbital Pattern 430 Display Interface

Claims

1. identifying a robotic device and a calibration fixture in a vicinity of the robotic device within the environment; referencing the calibration fixture relative to a predetermined portion of the robotic device to determine a first pose of the robotic device relative to the calibration fixture; receiving a 3D image of the environment from a sensor, the 3D image including the calibration fixture; determining a second pose of the calibration fixture relative to the sensor based on the 3D image; determining a third pose of the robotic device relative to the sensor based on the first pose and the second pose; receiving a plurality of trajectory points from a display interface or a device interface; determining a plurality of virtual trajectory points corresponding to the plurality of trajectory points based on the 3D image and the third pose; receiving a 2D image of the environment; providing a display of the 2D image of the environment overlaid with the virtual trajectory points; providing a display of the 2D image of the environment overlaid with the 3D image received from the sensor, wherein the 3D image overlaid on the 2D image of the environment can appear visibly or invisibly on the display interface; updating the display in response to a change in orientation of a display device to show at least a portion of the 2D image superimposed with a 3D image from a corresponding viewpoint; receiving input data from the display interface or the device interface providing instructions for the robotic device to move according to one or more of the trajectory points or the virtual trajectory points and their respective properties; transmitting the command to the robotic device, including converting the virtual trajectory points into physical trajectory points and sending the physical trajectory points to the robotic device.

10. A computer-implemented method comprising:

2. The method of claim 1 , wherein the determining a second pose of the calibration fixture relative to the sensor is based on recognizing three-dimensional features of the calibration fixture in the 3D image.

3. providing a display of an overlaid virtual representation of a trajectory for the robotic device, the trajectory including a sequence of at least some of the plurality of trajectory points; The method of claim 1 or 2, further comprising:

4. generating and displaying a graphical representation of the trajectory orientation at one or more of the virtual trajectory points; generating and displaying a graphical representation of a tool performing a process along said trajectory; 4. The method of claim 1, further comprising:

5. receiving, from the display interface, input data indicative of one or more adjustments to one or more properties of one or more of the plurality of the virtual trajectory points, the properties being one or more selected from the group consisting of trajectory position, trajectory heading, end effector state, trajectory velocity, electronic signal input, and electronic signal output; receiving input data from the display interface to create, duplicate, or delete one or more of the virtual trajectory points; determining one or more adjusted properties of one or more of the trajectory points corresponding to the one or more of the plurality of virtual trajectory points based on input data received on the display interface; 5. The method of claim 1, further comprising:

6. receiving input data from the display interface or the device interface providing instructions for the robotic device to move according to one or more of the trajectory points or the virtual trajectory points and their respective properties; transmitting said command to said robotic device; 6. The method of any one of claims 1 to 5, further comprising:

7. transmitting the command to the robotic device; converting the virtual trajectory points into physical trajectory points and sending the physical trajectory points to the robotic device; The method of claim 6, comprising:

8. providing the display interface for manipulating the virtual trajectory points; receiving input data from the display interface indicating creation of and / or adjustments to a trajectory pattern including the plurality of virtual trajectory points arranged in a specified pattern; converting the two-dimensional trajectory pattern into a three-dimensional trajectory pattern on the display interface based on the created or adjusted trajectory pattern; receiving input data from the display interface for translating and / or rotating the three-dimensional trajectory pattern on the display interface; receiving input data from the display interface for projecting the three-dimensional trajectory pattern onto a portion of the 3D image shown on the display interface; 8. The method of any one of claims 1 to 7, further comprising:

9. 9. The method of claim 1, wherein the trajectory points are in the form of user entries via the display interface or in the form of robot scripts via the device interface.

10. Interpolating or extrapolating one or more trajectory points along the surface of the object and the environment based on the 3D image.

10. The method of any one of claims 1 to 9, further comprising:

11. transmitting robot information, such as robot joint angles, robot status, and end effector state, from the robotic device to a mobile device, laptop, or desktop computer; generating and displaying, on the display interface, the received robot information about the virtual robot overlaid on a representation of the real robot device; 11. The method of any one of claims 1 to 10, further comprising:

12. A sensor, a computing system communicatively coupled to the sensor; 1. A system for programming a robotic device, comprising: the computing system, identifying a robotic device and a calibration fixture in a vicinity of the robotic device within the environment; referencing the calibration fixture relative to a predetermined portion of the robotic device to determine a first pose of the robotic device relative to the calibration fixture; receiving a 3D image of the environment from the sensor, the 3D image including the calibration fixture; determining a second pose of the calibration fixture relative to the sensor based on the 3D image; determining a third pose of the robotic device relative to the sensor based on the first pose and the second pose; receiving a plurality of trajectory points from a display interface or a device interface; determining a plurality of virtual trajectory points corresponding to the plurality of trajectory points based on the 3D image and the third pose; receiving a 2D image of the environment; providing a display of the 2D image of the environment overlaid with the virtual trajectory points; providing a display of the 2D image of the environment overlaid with the 3D image received from the sensor, wherein the 3D image overlaid on the 2D image of the environment can appear visibly or invisibly on the display interface; updating the display in response to a change in orientation of a display device to show at least a portion of the 2D image superimposed with a 3D image from a corresponding viewpoint; receiving input data from the display interface or the device interface providing instructions for the robotic device to move according to one or more of the trajectory points or the virtual trajectory points and their respective properties; transmitting the command to the robotic device, including converting the virtual trajectory points into physical trajectory points and sending the physical trajectory points to the robotic device. A system configured to:

13. the computing system, determining the second pose of the calibration fixture relative to the sensor based on recognizing 3D features of the calibration fixture in the 3D image; The system of claim 12 further configured to:

14. the computing system, providing the display interface for adjusting one or more properties of one or more of the plurality of virtual trajectory points, wherein the properties are one or more selected from the group consisting of trajectory position, trajectory orientation, end effector state, trajectory velocity, electronic signal input, and electronic signal output; providing the display interface for creating, duplicating, or deleting the virtual trajectory points; determining one or more adjusted properties of one or more of the trajectory points corresponding to the one or more of the plurality of virtual trajectory points; 14. The system of claim 12 or 13, further configured to:

15. the computing system, generating and displaying a graphical representation of the trajectory orientation at one or more of the virtual trajectory points; Generating and displaying a graphical representation of the tool performing the process along the trajectory; 15. The system of any one of claims 12 to 14, further configured to:

16. the computing system, providing the display interface for manipulating the virtual trajectory points; receiving input data from the display interface indicating creation of and / or adjustments to an orbit pattern including the plurality of virtual orbit points arranged in a specified pattern; converting the two-dimensional trajectory pattern into a three-dimensional trajectory pattern on the display interface based on the created or adjusted trajectory pattern; receiving input data from the display interface for translating and / or rotating the three-dimensional trajectory pattern on the display interface; receiving input data from the display interface for projecting the three-dimensional trajectory pattern onto a portion of the 3D image shown on the display interface; 16. The system of any one of claims 12 to 15, further configured to:

17. A non-transitory computer-readable medium having stored therein instructions that, when executed by a computing system, cause the computing system to perform the method of any one of claims 1 to 11.

18. Identifying a robotic device and a calibration fixture in a vicinity of the robotic device within an environment; referencing the calibration fixture relative to a predetermined portion of the robotic device to determine a first pose of the robotic device relative to the calibration fixture; receiving a 3D image of the environment from a sensor, the 3D image including the calibration fixture; determining a second pose of the calibration fixture relative to the sensor based on the 3D image; determining a third pose of the robotic device relative to the sensor based on the first pose and the second pose; providing a display interface for creating virtual trajectory points; determining a trajectory point corresponding to the virtual trajectory point based on the 3D image and the third pose; receiving a 2D image of the environment; providing a display of the 2D image of the environment overlaid with the virtual trajectory points; providing a display of the 2D image of the environment overlaid with the 3D image received from the sensor, wherein the 3D image overlaid on the 2D image of the environment can appear visibly or invisibly on the display interface; updating the display in response to a change in orientation of a display device to show at least a portion of the 2D image superimposed with a 3D image from a corresponding viewpoint; receiving input data from the display interface or the device interface that provides instructions for the robotic device to move according to one or more of the trajectory points or the virtual trajectory points and their respective properties; transmitting the command to the robotic device, including converting the virtual trajectory points into physical trajectory points and sending the physical trajectory points to the robotic device; A method comprising:

Images