Method and system for programming a robot

Abstract

The method comprises the steps of: identifying a robot device and a calibration fixture near the robot device; referencing the calibration fixture with respect to the base of the robot device to determine a first pose of the robot device; receiving a 3D image of the environment—the 3D image includes the calibration fixture—; determining a second pose of the calibration fixture with respect to a sensor; determining a third pose of the robot device with respect to the sensor based on the first pose and the second pose; receiving a plurality of trajectory points; determining a plurality of virtual trajectory points corresponding to the plurality of trajectory points based on the 3D image and the third pose; providing a display of the plurality of virtual trajectory points; and providing an interface for manipulating the virtual trajectory points.

Description

Technology Field

[0001] The present invention relates to a system and method for programming a robot to perform a sequence of trajectory points. More specifically, the present invention relates to ways in which a robot is visually programmed in an operating environment. Background Technology

[0002] Robots are often used to process the surfaces of objects. Conventional practices for programming robots involve teaching the robot a sequence of trajectory points. During programming, the robot is taught how to perform a task by being guided through various trajectory points along a desired motion path. Alternatively, if a three-dimensional (3D) CAD model of the object exists, a person with a robotics background teaches the trajectory points in a robot simulation system, so-called offline programming.

[0003] The present invention relates to a computer-implemented method comprising the steps of: identifying a robot device and a calibration fixture near the robot device within an environment; referencing the calibration fixture to a predetermined part of the robot device to determine a first pose of the robot device with respect to the calibration fixture; receiving a 3D image of the environment from a sensor—the 3D image includes the calibration fixture—; determining a second pose of the calibration fixture with respect to the sensor based on the 3D image; determining a third pose of the robot device with respect to the sensor based on the first pose and the second pose; receiving a plurality of trajectory points from a display interface or a device interface; and determining a plurality of virtual trajectory points corresponding to the plurality of trajectory points based on the 3D image and the third pose.

[0004] The present invention also relates to a system for programming a robot device. The system includes a sensor and a computing system coupled to communicate therewith and configured to perform a method according to any one of the embodiments disclosed herein.

[0005] The present invention also relates to a non-transient computer-readable medium storing instructions that, when executed by a computing system, cause the computing system to perform a method according to any one of the embodiments disclosed herein.

[0006] The exemplary embodiments described herein are not intended to be limiting. It will be readily understood that specific aspects of the disclosed method and system may be arranged and combined in a wide variety of different configurations, all of which are taken into account in this specification.

[0007] Furthermore, the specific arrangements depicted in the drawings should not be considered limiting. It should be understood that other embodiments may include more or fewer of each element depicted in the given drawings. Additionally, some of the illustrated elements may be combined or omitted. Furthermore, exemplary embodiments may include the elements illustrated in the drawings.

[0008] An exemplary system and method are provided for using a sensor that can be coupled to a display interface to acquire a 3D image of an environment, wherein the 3D image includes a robot, an object, and a calibration fixture. The 3D image is used to generate, manipulate, and / or manage trajectory points of the robot. The image of the environment refers to a 2D image on the display interface. The display interface coupled with the sensor is used to visualize and manage trajectory points for the robot within the actual image of the robot's environment.

[0009] According to one embodiment, a method is provided, the method comprising the step of identifying a robot device and a calibration fixture near the robot device within an environment. The robot device is also referred to as a robot. For example, if the robot device is an industrial robot, the robot device has an end-effector comprising a tool for performing processing, for example, a welding tool or a painting tool. The step of identifying the robot device refers to the step of identifying the presence of the robot device. The step of identifying may further include the step of identifying additional information identifying the robot device, for example, the identification number, manufacturer, model, location, orientation, and / or status of the robot device. The step of identifying may include the step of providing the robot device. Alternatively or additionally, the step of identifying may refer to the step of identifying the location of the robot device in sensor data, for example, image data, magnetic field data, electric field data, signals transmitted via cable, signals transmitted wirelessly, or any other data that can be used to identify the robot device. The calibration fixture may be marked so as to be visually distinguishable, for example, by color marking, a barcode, a two-dimensional code, or a distinguishing coating.

[0010] Additionally, the method includes the step of referencing a calibration fixture to a predetermined part (e.g., base) of the robot device to determine a first pose of the robot device relative to the calibration fixture. The referencing step may be based on a reference signal. For example, the referencing step may include using a reference rail, a distance measuring device (e.g., a laser-based distance measuring device), an image sensor, or other signal that can be used to refer the calibration fixture to the base of the robot device.

[0011] Additionally, the method includes the step of receiving a 3D image of the environment from a sensor, and the 3D image may include a calibration fixture. The 3D image may show a robot, an object, and a calibration fixture. The 3D image is used to generate, manipulate, and / or manage trajectory points. The 3D image may include a series of infrared images, a series of structured light images, a series of still images, dynamic range images, shot noise images, red noise images, dark noise images, and / or video streams. The sensor capturing the 3D image may be a depth sensor and / or a 3D sensor. The sensor may be coupled to a display interface to display the 3D image. The calibration fixture may be a three-dimensional (3D) object used to calculate and detect the pose and generally movement (translation and rotation) of the robot in its environment using the sensor.

[0012] Because the method does not rely on using a fully virtual 3D model (e.g., computer-aided design or "CAD" model), an inaccurate 3D model can command a trajectory that causes the robot to unexpectedly collide with an object or environment. Therefore, the method is safer.

[0013] Additionally, the method includes a step of determining a second pose of the calibration fixture relative to the sensor based on a 3D image. Here, the determining step may include a step of identifying one or more reference markers when estimating the position and / or orientation of the calibration fixture and / or the position and / or orientation of the robotic device. A reference marker, which may also be called a reference, is an object placed within the field of view of the sensor appearing in the generated image to be used as a point of reference or scale. For example, the reference marker may be a color dot, a barcode, or a recognizable object. For example, the reference marker may be painted on, attached to, or placed on the calibration fixture and / or robotic device. Additionally or alternatively, the determining step may be based on the recognition of the 3D shape of the calibration marker. Here, the calibration marker may be a 3D structure or object attached to or forming a part of the calibration fixture. Alternatively, the entire calibration fixture may be the calibration marker. Additionally, the calibration marker may be a 3D structure attached to or forming a part of the robotic device. Alternatively, the entire robotic device may be the calibration marker. Here, 3D shape recognition may include pattern recognition, pattern matching, machine learning, or any other suitable shape recognition technique.

[0014] The method includes the step of determining a second pose of the calibration fixture for the sensor based on a 3D image, so the second pose can be determined quickly without requiring additional equipment. When the second pose is determined based on 3D shape recognition, the accuracy of the determination is significantly improved.

[0015] Additionally, the method includes the step of determining a third pose of the robot device relative to the sensor based on the first pose. Alternatively, the determination of the third pose may be based on the first pose and the second pose.

[0016] Additionally, the method includes the step of receiving a plurality of trajectory points. Trajectory points, which may also be called waypoints, define the path that the robot must follow during the processing of an object. Trajectory points include three-dimensional position and three-dimensional orientation information. These trajectory points are stored as instructions in memory within or coupled to the robot control unit or the robot's computing system. During the robot's operation, program instructions are executed to cause the robot to operate as desired. Trajectory points may be provided in the form of user input through a display interface or in the form of robot scripts through a device interface.

[0017] Additionally, the method includes the step of determining a plurality of virtual trajectory points corresponding to a plurality of trajectory points based on a 3D image and a third pose. The trajectory points may have been previously programmed by a user as 3D points for a predetermined part of the robot (e.g., base) without knowledge of the robot's current environment.

[0018] Since the method includes the step of determining a plurality of virtual trajectory points corresponding to a plurality of trajectory points based on a 3D image and a third pose, the method does not require the user or operator programming the robot to have knowledge of computer science. Therefore, the operator programming the robot may be a person with good implicit knowledge of the movements and processes performed by the robot. Thus, the method is less costly, does not require a 3D CAD model of the object, and is intuitive to use.

[0019] Additionally, optionally, the method includes the step of receiving a 2D image of the environment. Here, the 2D image of the environment may be received from a camera included in or coupled to a display device. Alternatively, the 2D image of the environment may be received from a camera included in or coupled to a sensor.

[0020] Additionally, optionally, the method includes the step of providing a 2D image of an environment overlaid with a plurality of virtual trajectory points.

[0021] Since the method includes the step of providing a display of a 2D image of the environment overlaid with multiple virtual trajectory points, a trajectory that causes the robot to unexpectedly collide with an object in the environment or generates other unexpected results can be easily determined if the 3D model of the environment is inaccurate or if the environment changes. Therefore, the method is safer and more predictable. Additionally, since the method includes the step of providing a display of a 2D image of the environment overlaid with multiple virtual trajectory points, the operator can determine whether a specific part of the process, for example, a specific part of the surface in a painting motion, has been omitted without running the program for testing. Thus, a single iteration may be sufficient to achieve programming of satisfactory quality.

[0022] Additionally, optionally, the method includes the step of providing an interface for manipulating virtual trajectory points through a display interface.

[0023] Since the method includes the step of providing an interface for manipulating virtual trajectory points through a display interface, the method does not require physical movement of the robot. Therefore, the method is less time-consuming, less problematic, and less prone to errors. Additionally, an acceptable program can be achieved in a single or only a few iterations.

[0024] Therefore, this method provides intuitive visualization of robot trajectory points and simplifies robot programming.

[0025] The method may also include the step of providing a display of a 3D image that may or may not be superimposed on a 2D image of the environment.

[0026] To determine where to project the robot's trajectory points onto an image on a display interface, the robot's pose (position and orientation) relative to the sensor may be determined. The robot's pose may be determined by processing a 3D image of the calibration fixture. Once the robot's pose relative to the sensor is determined, the pose of one or more trajectory points may be projected onto the display interface or optionally onto an augmented reality presentation on the display interface, allowing the user to visualize the robot's trajectory within its given environment.

[0027] A calibration fixture is an object having 3D features used to determine the robot's pose relative to a sensor. The calibration fixture can be placed within the robot's environment at a determined relative pose between the robot and the calibration fixture. When the sensor acquires a 3D image of the environment—the 3D image includes a representation of the calibration fixture—the robot's pose relative to the sensor can be determined accordingly.

[0028] In another embodiment, the method may further include the step of receiving input data from a device interface representing one or more adjustments to one or more characteristics of one or more virtual trajectory points among a plurality of virtual trajectory points. Here, the characteristics are one or more selected from the group consisting of trajectory position, trajectory orientation, end effector state, trajectory velocity, electronic signal input, and electronic signal output. The device interface may be a display interface of a display device, or an input interface of a mobile, laptop, notebook, or desktop computer. The input data represents adjustments, and each adjustment relates to one or more characteristics of one or more virtual trajectory points among the virtual trajectory points. For example, the input data may include data representing an adjustment to one characteristic of one trajectory point, such as the trajectory orientation of one trajectory point. As another example, the input data may include data representing an adjustment to the trajectory orientation of one trajectory point, and data representing an adjustment to the trajectory velocity of another trajectory point. Additionally, the method according to this embodiment may include the step of receiving input data from a device interface for creating, duplicating, or deleting one or more of virtual trajectory points. Additionally, the method may include the step of determining one or more adjusted characteristics of one or more trajectory points among trajectory points corresponding to one or more of the virtual trajectory points among the plurality of virtual trajectory points, based on the received input data on a display interface.

[0029] In another embodiment, the method may further include the steps of receiving input data from an interface of a device including a sensor, providing commands for moving a robot device according to one or more of trajectory points or virtual trajectory points and their respective characteristics, and transmitting the said commands to the robot device.

[0030] The aforementioned received trajectory points may include a reference to one of various coordinates or are based on one of various coordinates. For example, trajectory points that may be received from user input through a display interface may be based on virtual coordinates and thus may be referred to as virtual trajectory points. In another example, trajectory points received as robot scripts may be based on physical or robot coordinates and thus may be referred to as physical trajectory points. Because virtual coordinates may differ from the physical or robot coordinates used by the robot device, if the received trajectory points are virtual trajectory points, they will require transformation or conversion into physical trajectory points that are later transmitted to the robot device for execution. Accordingly, the step of transmitting the aforementioned commands to the robot device may include the step of converting virtual trajectory points into physical trajectory points and transmitting the physical trajectory points to the robot device.

[0031] In another embodiment, the method may further include the step of providing a display interface for manipulating virtual trajectory points, and the step of receiving input data from the display interface indicating the creation and / or adjustment thereof of a trajectory pattern comprising a plurality of virtual trajectory points arranged in a specified pattern. Additionally, the method may include the step of converting a two-dimensional trajectory pattern into a three-dimensional trajectory pattern on the display interface based on the created or adjusted trajectory pattern. Additionally, the method may include the step of receiving input data from the display interface for translating and / or rotating the three-dimensional trajectory pattern on the display interface. Additionally, the method may include the step of receiving input data from the display interface for projecting the three-dimensional trajectory pattern onto a portion of a 3D image. Here, in one embodiment, the 3D image may be superimposed on a 2D image on the display interface. Alternatively, the 3D image may be displayed without the 2D image.

[0032] According to another embodiment, a system for programming a robot device is provided. The system includes a sensor and a computing system coupled to communicate therewith and configured to perform a method according to one embodiment.

[0033] Additionally, the system may be coupled to or connected to the memory of the robot device to store trajectory points in memory. Alternatively, the system may be coupled to or connected to the control of the robot device, so that the system can operate the robot device according to the trajectory points.

[0034] According to another embodiment, a non-transient computer-readable medium is provided. The computer-readable medium stores instructions that, when executed by a computing system, cause the computing system to perform functions according to one embodiment of the method described herein.

[0035] According to another embodiment, a method is provided. The method comprises the steps of: identifying a robot device and a calibration fixture near the robot device within an environment; referencing the calibration fixture to a predetermined part of the robot device (e.g., a base) to determine a first pose of the robot device relative to the calibration fixture; receiving a 3D image of the environment from a sensor—the 3D image includes the calibration fixture—determining a second pose of the calibration fixture relative to the sensor based on the 3D image; determining a third pose of the robot device relative to the sensor based on the first pose and the second pose; providing an interface for generating a virtual trajectory point; and determining a trajectory point corresponding to the virtual trajectory point based on the 3D image and the third pose.

[0036] In embodiments comprising a method, system, and / or a non-transient computer-readable medium according to any one of the embodiments disclosed in this specification, a plurality of trajectory points is only one trajectory point, and a plurality of virtual trajectory points is only one virtual trajectory point. Brief explanation of the drawing

[0037] For a better understanding of the embodiments, the following description taken together with the accompanying drawings is referenced, along with other and additional features and advantages of the embodiments. The scope of the invention will be indicated in the appended claims. FIG. 1 illustrates a flowchart illustrating a method according to one embodiment. FIG. 2 illustrates a system according to one embodiment. Figure 3 illustrates a 2D image of a display interface. FIG. 4 illustrates an image of a trajectory pattern projected onto a 3D image of an object using a method according to one embodiment. Specific details for implementing the invention

[0038] Referring to the drawings, FIG. 1 illustrates a flowchart illustrating a method (100) that may allow for the determination of a third pose (relative pose between the robot and the sensor) and / or the generation or management of robot trajectory points using a 3D image acquired by the sensor. The method (100) may be performed using a display device including a camera, such as a tablet device, a smartphone, or a head-mounted display (HMD), or using a mobile, laptop, notebook, or desktop computer. A sensor separate from the display device may be used, but the sensor may be coupled to the display device via a physical fixture or an adhesive medium. The sensor may also be coupled to the display device communicably via a cable (wired) or wireless connection, or alternatively.

[0039] As illustrated by block (102) of FIG. 1, the method (100) includes the step of receiving a 3D image of an environment from a sensor. The sensor may be a depth sensor and / or a 3D sensor. The 3D image may be a composite of a series of infrared images, a series of structured light images, a series of still images, and / or a video stream. The 3D image may be a single still infrared and / or structured light image. The environment may include at least one robotic device and a calibration fixture.

[0040] Referring to FIG. 1, the method (100) further includes the step of determining the pose of a robot device relative to a sensor, as exemplified by block (104). A 3D image of the environment may be processed to determine where the robot device is located within the environment relative to the sensor. The step of determining the pose of the robot device includes determining the position (X, Y, Z) and orientation (roll, pitch, yaw) of the robot device relative to the sensor. The pose of the robot device relative to the sensor may be determined by processing a 3D image including a calibration fixture. By recognizing three-dimensional features on the calibration fixture in the 3D image, the translation and rotation of the robot device relative to the sensor may be determined. The system may have prior knowledge of the relative pose between the robot device and the calibration fixture, thereby enabling the pose of the robot device relative to the sensor to be determined.

[0041] The method (100) optionally further includes the step of determining the poses of the trajectory points of the robot device relative to the sensor when an augmented reality presentation is required, as illustrated in block (106). When the pose of the robot device relative to the sensor is determined, the robot may be used as a reference point to determine where to virtually overlay the trajectory points as part of an optional augmented reality presentation. When the sensor is coupled to the display device with a known relative displacement, the relative pose between the robot and the visual camera may be derived from the relative pose between the visual camera and the sensor through pose compensation based on the known relative displacement between the sensor and the visual camera on the display device. Here, the pose of the trajectory points relative to the visual camera on the display device may be determined based on the relative pose between the robot device and the sensor. These locations, e.g., the relative displacement between the sensor and the visual camera on the display device, may be used to virtually overlay the trajectory points onto a 2D image captured by the visual camera on the display device.

[0042] For simplification, in a step not illustrated in FIG. 1, the method (100) optionally includes the step of receiving a 2D image of the environment. Here, the 2D image is received from a sensor. In an alternative embodiment, the 2D image may be received by an additional camera, preferably located close to the sensor and having the same orientation as the orientation of the sensor.

[0043] As illustrated in block (108), the method (100) includes the step of providing a display of a 3D image received in block (102). Optionally, the 3D image may be superimposed on a 2D image of the environment.

[0044] For simplification, in a step not illustrated in FIG. 1, the method (100) includes the step of receiving a plurality of trajectory points. Furthermore, the method (100) includes the step of determining a plurality of virtual trajectory points corresponding to the plurality of trajectory points based on the position of the robot device relative to the sensor determined in the 3D image and block (106). Here, each virtual trajectory point is located in the 3D image in a manner that overlaps with the exact position of its corresponding trajectory point within the corresponding 3D image. That is, for example, if a trajectory point is located at the lower joint of the robot device, its corresponding virtual trajectory point is located in the 3D representation of the lower joint of the robot device in the 3D image. Optionally, each virtual trajectory point is located in the 2D image in a manner that overlaps with the exact position of its corresponding trajectory point within the corresponding 3D image.

[0045] As illustrated in block (110), the method (100) further includes the step of providing a display of a 3D image overlaid with a plurality of determined virtual trajectory points. Optionally, the method (100) further includes the step of providing a display of a 2D image of an environment overlaid with a plurality of determined virtual trajectory points.

[0046] As illustrated by block (112) of FIG. 1, the method (100) may include an overlay of virtual trajectory points on a 3D image presented on a display interface. Optionally, block (112) of the method (100) may include an overlay of virtual trajectory points on a 2D image of an environment presented on a display interface. The image of the environment may be a 2D image acquired by a visual camera on a display device. The image of the environment may include the entire robot device or a part of the robot device, or may not include any of the robot devices. Additionally, the image of the environment may include the entire calibration fixture or a part of the calibration fixture, or may not include any of the calibration fixtures. Additionally, the image of the environment may be a single still 2D image, a series of still 2D images, and / or a video stream.

[0047] As illustrated by block (112) of FIG. 1, the method (100) may further include the step of creating and / or managing virtual trajectory points through a display interface. That is, a trajectory or trajectory pattern to be executed or performed by a robot device may be created. A user interface associated with the display interface may receive input data representing one or more changes to the virtual trajectory of the robot device. For example, the display interface may include a touch-based interface aligned with 3D images and / or optionally 2D images. Virtual trajectory points may be created, deleted, and / or duplicated through input data on the display interface. Attributes of each virtual trajectory point, such as position, orientation, end effector state, trajectory velocity, electronic signal input, and electronic signal output, may be adjusted through input data on the display interface. The trajectory points of the robot device correspond to the virtual trajectory points on the display interface. The robot device may be commanded to execute a trajectory, and the trajectory comprises a sequence of at least some of a plurality of trajectory points. The robot device can be commanded to execute the attributes of each trajectory point along the trajectory. For example, an end effector can be commanded to be activated midway along the trajectory as indicated by input data on the display interface.

[0048] The functions described in relation to the flowcharts described herein may be implemented as special function and / or configured general function hardware modules, specific logical functions described in relation to the flowchart illustrated in FIG. 1, and parts of program code for achieving decisions and / or steps. When used, the program code may be stored on any type of computer-readable medium, such as a storage device including, for example, a disk or a hard drive.

[0049] The functions in the flowchart illustrated in FIG. 1 may be executed in a different order from or even in reverse order than illustrated or discussed, including the parallel execution of individually described functions, as long as the overall function of the described method is maintained. The functions in the flowchart illustrated in FIG. 1 may be executed optionally. For example, one embodiment may perform calibration of the robot relative sensor; another embodiment may, in addition to calibration, perform visualization of virtual trajectory points for the robot through an augmented reality presentation; another embodiment may, in addition to calibration, perform visualization of virtual trajectory points for the robot using 3D images; and another embodiment may, in addition to calibration and said visualization, perform generation or management of robot trajectory points using 3D images acquired by the sensor in augmented reality and / or using 3D images. Other combinations may also be possible.

[0050] FIG. 2 illustrates a system including a robot device (202), a calibration fixture (228), a display device (222), and a sensor (224).

[0051] The robot device (202) includes a base (204) which may be a predetermined portion, for example, a fixed base or a movable base. The robot device may be controlled to operate and move along a trajectory (220) that includes trajectory points (210-218). Additionally, the robot device may include an end effector (226) which may take the form of a gripper such as a finger gripper or a different type of gripper such as a suction gripper. The end effector may take the form of a tool such as a drill, a brush, or a paint gun. The end effector may include sensors such as force sensors, proximity sensors, or a camera. Other examples may also be possible.

[0052] The display device (222) may be a device including an interface and optionally a visual camera that captures a 2D image of the environment. For example, the display device may be part of a tablet computer, a handheld smartphone, or a mobile, laptop, notebook, or desktop computer.

[0053] The sensor (224) may be a depth sensor and / or a 3D sensor that acquires a 3D image of the environment. The 3D image may be a composite of a series of infrared images, a series of structured light images, a series of still images, and / or a video stream. The 3D image may be a single still infrared and / or structured light image. The sensor (224) may be physically fixed to the display device (222) via a fixture or an adhesive medium. The fixture connecting the sensor to the display device may have a detachable mechanism or a non-detachable mechanism. For the display device (222) to receive the 3D image from the sensor (224), the sensor (224) may be connected to the display device (222) via a cable (wired) or wireless connection.

[0054] A calibration fixture (228) is an object having three-dimensional features that is placed in the environment of a robot device. According to an exemplary embodiment of FIG. 2, the calibration fixture (228) may have a base (206) and asymmetric geometric features (208) attached to the base. The base (206) may be used to position or attach the calibration fixture (228) on a surface within the environment, such as a table, a wall, or an object. In an exemplary embodiment, the asymmetric geometric features (208) may include two spherical objects having different sizes and / or different positions. However, the exemplary embodiments described herein are not intended to be limiting. The asymmetric geometric features may be achieved by configurations not shown in the exemplary embodiments. For example, the asymmetric geometric features on the calibration fixture may include additional shapes or features that may be hemispherical, cylindrical, conical, rectangular, triangular, trapezoidal, elliptical, sine, concave, or combinations or variations thereof.

[0055] By recognizing asymmetric geometric features (208) on the correction fixture (228) in a 3D image acquired by the sensor (224), the pose of the correction fixture (228) with respect to the sensor (224) can be determined. Since the pose of the robot device (202) with respect to the correction fixture (228) can be known, the pose of the robot device (202) with respect to the sensor (224) can be determined accordingly by recognizing the correction fixture (228) in the 3D image.

[0056] In an alternative embodiment, the orthodontic fixture may have a base and 3D symmetric geometric features attached to the base. Examples of symmetric geometric features may include the aforementioned shapes or features applicable to asymmetric geometric features. In an alternative embodiment, the orthodontic fixture may have a base and 3D non-geometric or irregular features attached to the base. Such non-geometric features may be symmetric or asymmetric. In an alternative embodiment, the count of 3D features may be one or more than one.

[0057] FIG. 3 illustrates a 2D image of a display interface (330), wherein virtual trajectory points (310-328) are placed on an object (332). The display interface (330) may be an interface of a display device (222). The 2D image of the display interface (330) may be a single still image, or a video stream continuously received from a viewing camera of the display device (222). As the orientation of the display device (222) changes, the 2D image on the display interface (330) may be updated to display parts of the environment from a corresponding viewpoint.

[0058] Referring to FIG. 3, virtual trajectory points (310-328) can be projected onto the display interface (330) or optionally onto an augmented reality presentation on the display interface (330). The virtual trajectory points correspond to the trajectory points of the robot device (302). The poses of the trajectory points (310-328) of the robot device (302) relative to the sensor are determined based on block (106) of FIG. 1. The virtual trajectory points are overlaid on the display interface (330) based on block (112) of FIG. 1. Only a subset of the trajectory points reflected by the virtual trajectory points (310-328) can be projected onto the display interface (330) according to the orientation of the display device (222). The robot device can be commanded to move through a sequence of trajectory points as reflected by virtual trajectory points (310-328) shown on the display interface (330).

[0059] Referring to block (108) of FIG. 1, a 3D image acquired by the sensor (224) may be superimposed on the display interface (330) or optionally on a 2D image of the environment on the display interface (330). The 3D image may be visible or invisible while superimposed on the 2D image of the environment on the display interface (330). Block (112) of FIG. 1 includes generating and / or managing virtual trajectory points on the display interface. The generation and / or management of virtual trajectory points may be performed on a 3D image that may or may not be superimposed on the 2D image of the environment on the display interface (330). For example, the location of an input on the display interface (330) is projected onto the 3D image, and subsequently, a virtual trajectory point is generated corresponding to the location where a ray projection from the input location on the display interface (330) intersects the 3D image. In this way, virtual trajectory points, and thus trajectory points, can be accurately generated and / or managed for the environment and the robot device (302). As illustrated in FIG. 3, virtual trajectory points (312-328) may be placed on a 3D image of an object (332) that may or may not be superimposed on a 2D image of the object (332) on the device interface (330), and the virtual trajectory points (312-328) may appear to be located on the surface of the object (332) on the device interface (330).

[0060] Referring to FIG. 3, the calibration fixture (334) may be used to determine the pose of the robot device (302) relative to the sensor (224). The calibration fixture (334) may not need to be continuously captured by the sensor (224) and the visual camera of the display device (222) for proper overlay of virtual trajectory points on the display interface (330). The calibration fixture may include asymmetric geometric features (308) and a base (306).

[0061] FIG. 4 illustrates an image of a trajectory pattern projected onto a 3D image of an object (412). The display interface (430) may include a subset of a user interface (402) that allows a user to define a trajectory pattern (404). For example, the user interface (402) may allow a user to define attributes of the trajectory pattern. The aforementioned attributes may include the dimensions of the trajectory pattern (404) (vertical and horizontal lengths), the number and / or density of trajectory points along the trajectory pattern (404), and the frequency of stepovers. The user interface (402) may provide a preview of the trajectory pattern (404) defined by the user. The trajectory pattern may not be limited to the raster pattern exemplified in the exemplary embodiment, and the trajectory pattern may be other patterns including zigzag, spiral, contour, and fishtail.

[0062] The trajectory pattern (404) may undergo a process (406) of converting the two-dimensional trajectory pattern (404) on the user interface (402) into a three-dimensional trajectory pattern (408). The three-dimensional trajectory pattern (408) may be translated and rotated along its three-dimensional coordinate frame when input data is received on the display interface (430). The three-dimensional trajectory pattern (408) may be projected onto a 3D image of an object (412) to produce a projected trajectory pattern (414) that fits the surface of the object (412). Each virtual trajectory point on the three-dimensional pattern (408) may be projected onto the surface of an object (412) through a respective projection path (410), and the projection path (410) includes an origin at each of their respective locations on the three-dimensional trajectory pattern (408) and a projection orientation that may be orthogonal to the plane of the three-dimensional trajectory pattern (408). The projection of the three-dimensional trajectory pattern (408) may not be limited to the object (412) shown in the exemplary embodiment. For example, the three-dimensional trajectory pattern (408) may be projected onto other objects not shown in the exemplary embodiment, such as turbine blades, airfoils, metal sheets, or other manufacturing components. The three-dimensional trajectory pattern (408) may be projected onto an environment, and the environment does not include the object (412) or robot devices, such as a table, floor, wall, fixture, or conveyor system.

[0063] The present disclosure is not to be limited to the specific embodiments described in this application, which are intended as examples of various aspects. Many modifications and variations may be made without departing from the spirit and scope thereof, as will be apparent to those skilled in the art. In addition to those listed herein, functionally equivalent methods and apparatuses within the scope of the present disclosure will be apparent to those skilled in the art from the foregoing descriptions. Such modifications and variations are intended to be within the scope of the appended claims.

[0064] The above detailed description describes various features and functions of the disclosed system and method with reference to the accompanying drawings. In the drawings, similar symbols typically identify similar components unless the context otherwise indicates. The exemplary embodiments described in this specification and drawings are not intended to be limiting. Other embodiments may be used and other modifications made without departing from the spirit or scope of the subject matter presented in this specification. It will be readily understood that the aspects of the disclosure as generally described in this specification and illustrated in the drawings may be arranged, substituted, combined, separated, and designed into various different configurations, all of which are explicitly taken into account in this specification.

[0065] A block representing the processing of information, such as a block of the method described above, may correspond to a circuit that can be configured to perform specific logical functions of the method or technique described herein. Alternatively or additionally, a block representing the processing of information may correspond to a module, segment, or part of program code (including related data). Program code may include one or more instructions executable by a processor to implement specific logical functions or actions in the method or technique. Program code and / or related data may be stored on any type of computer-readable medium, such as a storage device including a disk, a hard drive, or other storage media.

[0066] A block representing one or more information transmissions may correspond to information transmissions between software and / or hardware modules within the same physical device. However, other information transmissions may exist between software modules and / or hardware modules within different physical devices.

[0067] Computer-readable media may also include non-transient computer-readable media that store data for a short period, such as register memory, processor cache, and random access memory (RAM). Computer-readable media may also include non-transient computer-readable media that store program code and / or data for a long period, such as auxiliary or permanent long-term storage, such as read-only memory (ROM), optical or magnetic disks, and compact disc read-only memory (CD-ROM). Computer-readable media may be considered as computer-readable storage media or tangible storage devices.

[0068] The specific arrangements depicted in the drawings should not be considered limiting. It should be understood that other embodiments may include more or fewer of each element depicted in the given drawings. Some of the exemplified elements may be combined or omitted.

[0069] It should be understood that articles as used in relation to features or elements (“a,” “an,” and “the”) contain a reference to one or more of the features or elements. The term “and / or” includes any and all combinations of one or more of the associated features or elements. The terms “comprising,” “including,” “involving,” and “having” are intended to be open-ended and imply that there may be additional features or elements other than those listed. Identifiers such as “first,” “second,” and “third” are used merely as labels and are not intended to impose numerical requirements on their objects, nor are they interpreted to impose any relative position or time sequence between the limitations. The term “combination” may refer to a physical combination, an electrical combination, and / or a communicable combination. The term "combination" when applied to two objects may refer to the two objects being combined directly or indirectly through a third object.

[0070] Although various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for illustrative purposes only and are not intended to be limiting, and the true scope is indicated by the following claims. Explanation of the symbols

[0071] 100 Ways 102 blocks 104 blocks 106 blocks 108 blocks 110 blocks 112 blocks 202 Robot Devices 204 base 206 base 208 Asymmetric Geometric Features 210-218 Trajectory Points 220 Trajectory 222 Display Devices 224 sensor 226 End Effector 228 Fixture 302 Robot Device 306 base 308 Asymmetric Geometric Features 310-328 Virtual Trajectory Points 330 display interface 332 objects 334 Orthodontic Fixture 402 User Interface 404 Trajectory Pattern 406 process 408 Trajectory Pattern 410 Projection Path 412 object 414 Trajectory Pattern 430 Display Interface

Claims

Claim 1 A computer-implemented method comprising: identifying, in an environment, a robot device and a calibration fixture near said robot device; referencing said calibration fixture with respect to a predetermined portion of said robot device to determine a first pose of said robot device with respect to said calibration fixture; receiving a 3D image of said environment from a sensor, wherein the 3D image includes said calibration fixture; determining a second pose of said calibration fixture with respect to said sensor based on said 3D image; determining a third pose of said robot device with respect to said sensor based on said first pose and said second pose; and receiving a plurality of trajectory points from a display interface or a device interface. A computer-implemented method comprising the step of determining a plurality of virtual trajectory points corresponding to the plurality of trajectory points based on the 3D image and the third pose, wherein the method further comprises: receiving a 2D image of the environment; providing a display of the 2D image of the environment overlaid with the virtual trajectory points on the display interface; providing a display of the 2D image of the environment superimposed with the 3D image received from the sensor on the display interface, wherein the 3D image superimposed on the 2D image of the environment may appear visibly or invisibly on the display interface; and updating the display to show at least a portion of the 2D image superimposed with the 3D image from a corresponding point in time in response to changes in the orientation of the display device providing the display interface. Claim 2 A computer-implemented method according to claim 1, wherein the determination of the second pose of the calibration fixture with respect to the sensor is based on recognizing three-dimensional features of the calibration fixture in the 3D image. Claim 3 A computer-implemented method according to claim 1 or 2, further comprising the step of providing a display of an overlaid virtual representation of a trajectory for the robot device, wherein the trajectory comprises a sequence of at least some of the plurality of trajectory points. Claim 4 A computer-implemented method according to claim 1 or 2, further comprising the steps of: generating and displaying a graphic representation of a trajectory orientation at a single virtual trajectory point or at a plurality of virtual trajectory points; and generating and displaying a graphic representation of a tool that performs a process along the trajectory. Claim 5 A computer-implemented method according to claim 1 or 2, further comprising: receiving input data from the display interface representing one or more adjustments to one or more attributes of one or more virtual trajectory points among the plurality of virtual trajectory points—wherein one or more attributes are selected from a group consisting of trajectory position, trajectory orientation, end-effector state, trajectory velocity, electronic signal input, and electronic signal output—; receiving input data from the display interface for creating, duplicating, or deleting one or more of the virtual trajectory points; and determining one or more adjusted attributes of one or more trajectory points among the trajectory points corresponding to one or more virtual trajectory points among the plurality of virtual trajectory points based on the received input data on the display interface. Claim 6 A computer-implemented method according to claim 1 or 2, further comprising the steps of: receiving input data from the display interface or the device interface that provides commands for moving the robot device according to one or more of the trajectory points or virtual trajectory points and each of the attributes thereof; and transmitting the aforementioned commands to the robot device. Claim 7 A computer-implemented method according to claim 6, wherein the step of transmitting the aforementioned commands to the robot device comprises: converting the virtual trajectory points into physical trajectory points and sending the physical trajectory points to the robot device. Claim 8 A computer-implemented method according to claim 1 or 2, further comprising: providing a display interface for manipulating virtual trajectory points; receiving input data from the display interface representing at least one of generating a two-dimensional trajectory pattern including a plurality of virtual trajectory points arranged in a specified pattern and adjusting the two-dimensional trajectory pattern; converting the two-dimensional trajectory pattern into a three-dimensional trajectory pattern on the display interface based on the generated or adjusted trajectory pattern; receiving input data from the display interface for translating or rotating the three-dimensional trajectory pattern on the display interface; and receiving input data from the display interface for projecting the three-dimensional trajectory pattern onto a part of the 3D image shown on the display interface. Claim 9 A computer implementation method according to claim 1 or 2, wherein the trajectory points are in the form of user input through the display interface or in the form of robot scripts through the device interface. Claim 10 A computer-implemented method according to claim 1 or 2, further comprising the step of interpolating or extrapolating one trajectory point or a plurality of trajectory points along the surface of an object and an environment based on the 3D image. Claim 11 A computer-implemented method according to claim 1 or 2, further comprising: a step of transmitting robot information, such as robot joint angles, robot state, and end effector state, from the robot device to a mobile device, laptop, or desktop computer; and a step of generating and displaying the received robot information on a virtual robot that is overlaid on a representation of the actual robot device on the display interface. Claim 12 delete Claim 13 delete Claim 14 A system for programming a robot device comprises: a sensor; and a computing system communically coupled to the sensor, wherein the computing system: identifies, in an environment, a robot device and a calibration fixture near the robot device; references the calibration fixture to a predetermined portion of the robot device to determine a first pose of the robot device relative to the calibration fixture; receives a 3D image of the environment from the sensor, wherein the 3D image includes the calibration fixture; determines a second pose of the calibration fixture relative to the sensor based on the 3D image; determines a third pose of the robot device relative to the sensor based on the first pose and the second pose; receives a plurality of trajectory points from a display interface or a device interface; and determines a plurality of virtual trajectory points corresponding to the plurality of trajectory points based on the 3D image and the third pose, wherein the computing system also receives a 2D image of the environment; and provides a display of the 2D image of the environment overlaid with the virtual trajectory points; A system for programming a robot device, comprising: providing a display of a 2D image of the environment superimposed on a 3D image received from the sensor, wherein the 3D image superimposed on the 2D image of the environment may appear visibly or invisibly on the display interface; and updating the display to show at least a portion of the 2D image superimposed on the 3D image from a corresponding point in time in response to changes in the orientation of the display device providing the display interface. Claim 15 In claim 14, the computing system is also: a system for programming a robot device configured to determine the second pose of the calibration fixture relative to the sensor based on the recognition of 3D features of the calibration fixture within the 3D image. Claim 16 A system for programming a robot device, wherein, in claim 14 or 15, the computing system also: provides the display interface for adjusting one or more attributes of one or more virtual trajectory points among the plurality of virtual trajectory points, wherein the attributes are selected from one or more of the group consisting of trajectory position, trajectory orientation, end effector state, trajectory velocity, electronic signal input, and electronic signal output; provides the display interface for creating, duplicating, or deleting the virtual trajectory points; and is configured to determine one or more adjusted attributes of one or more trajectory points among the trajectory points corresponding to one or more virtual trajectory points among the plurality of virtual trajectory points. Claim 17 A system for programming a robot device, wherein, in claim 14 or 15, the computing system also: generates and displays a graphic representation of a trajectory orientation at one virtual trajectory point or at a plurality of virtual trajectory points; and generates and displays a graphic representation of a tool that performs a process along a trajectory including a sequence of at least some of the plurality of trajectory points. Claim 18 A system for programming a robot device, wherein, in claim 14 or 15, the computing system also: provides the display interface for manipulating the virtual trajectory points; receives from the display interface input data representing at least one of generating a two-dimensional trajectory pattern comprising the plurality of virtual trajectory points arranged in a specified pattern and adjusting the two-dimensional trajectory pattern; converts the two-dimensional trajectory pattern into a three-dimensional trajectory pattern on the display interface based on the generated or adjusted trajectory pattern; receives from the display interface input data for translating or rotating the three-dimensional trajectory pattern on the display interface; and receives from the display interface input data for projecting the three-dimensional trajectory pattern onto a portion of the 3D image shown on the display interface. Claim 19 A system for programming a robot device according to claim 14 or 15, wherein the computing system also: receives input data from the display interface or the device interface that provides commands for moving the robot device according to one or more of the trajectory points or virtual trajectory points and the attributes of each thereof; and is configured to transmit the aforementioned commands to the robot device, including converting the virtual trajectory points into physical trajectory points and sending the physical trajectory points to the robot device. Claim 20 A non-transient computer-readable medium having instructions stored therein, wherein the instructions, when executed by a computing system, cause the computing system to perform the method of claim 1 or 2. Claim 21 delete

Images